tokio.connectors.mmperfmon module¶
Connectors for the GPFS mmperfmon query usage
and
mmperfmon query gpfsNumberOperations
.
The typical output of mmperfmon query usage
may look something like:
Legend:
1: xxxxxxxx.nersc.gov|CPU|cpu_user
2: xxxxxxxx.nersc.gov|CPU|cpu_sys
3: xxxxxxxx.nersc.gov|Memory|mem_total
4: xxxxxxxx.nersc.gov|Memory|mem_free
5: xxxxxxxx.nersc.gov|Network|lo|net_r
6: xxxxxxxx.nersc.gov|Network|lo|net_s
Row Timestamp cpu_user cpu_sys mem_total mem_free net_r net_s
1 2019-01-11-10:00:00 0.2 0.56 31371.0 MB 18786.5 MB 1.7 kB 1.7 kB
2 2019-01-11-10:01:00 0.22 0.57 31371.0 MB 18785.6 MB 1.7 kB 1.7 kB
3 2019-01-11-10:02:00 0.14 0.55 31371.0 MB 18785.1 MB 1.7 kB 1.7 kB
Whereas the typical output of mmperfmon query gpfsnsdds
is:
Legend:
1: xxxxxxxx.nersc.gov|GPFSNSDDisk|na07md01|gpfs_nsdds_bytes_read
2: xxxxxxxx.nersc.gov|GPFSNSDDisk|na07md02|gpfs_nsdds_bytes_read
3: xxxxxxxx.nersc.gov|GPFSNSDDisk|na07md03|gpfs_nsdds_bytes_read
Row Timestamp gpfs_nsdds_bytes_read gpfs_nsdds_bytes_read gpfs_nsdds_bytes_read
1 2019-03-04-16:01:00 203539391 0 0
2 2019-03-04-16:02:00 175109739 0 0
3 2019-03-04-16:03:00 57053762 0 0
In general, each Legend: entry has the format:
col_number: hostname|subsystem[|device_id]|counter_name
where
- col_number is an aribtrary number
- hostname is the fully qualified NSD server hostname
- subsystem is the type of component being measured (CPU, memory, network, disk)
- device_id is optional and represents the instance of the subsystem being measured (e.g., CPU core ID, network interface, or disk identifier)
- counter_name is the specific metric being measured
-
class
tokio.connectors.mmperfmon.
Mmperfmon
(*args, **kwargs)[source]¶ Bases:
tokio.connectors.common.SubprocessOutputDict
Representation for the mmperfmon query command. Generates a dict of form:
{ timestamp0: { "something0.nersc.gov": { "key0": value0, "key1": value1, ... }, "something1.nersc.gov": { ... }, ... }, timestamp1: { ... }, ... }
-
__repr__
()[source]¶ Returns string representation of self
This does not convert back into a format that attempts to resemble the mmperfmon output because the process of loading mmperfmon output is lossy.
-
load
(cache_file=None)[source]¶ Load either a tarfile, directory, or single mmperfmon output file
Tries to load self.cache_file; if it is a directory or tarfile, it is handled by self.load_multiple; otherwise falls through to the load_str code path.
-
load_cache
(cache_file=None)[source]¶ Loads from one of two formats of cache files
Because self.save_cache() outputs to a different format from self.load_str(), load_cache() must be able to ingest both formats.
-
load_multiple
(input_file)[source]¶ Load one or more input files from a directory or tarball
Parameters: - input_file (str) – Path to either a directory or a tarfile containing
- text files, each of which contains the output of a single (multiple) –
- invocation. (mmperfmon) –
-
load_str
(input_str)[source]¶ Parses the output of the subprocess output to initialize self
Parameters: input_str (str) – Text output of the mmperfmon query
command
-
to_dataframe_by_host
(host)[source]¶ Returns data from a specific host as a DataFrame
Parameters: host (str) – Hostname from which a DataFrame should be constructed Returns: All measurements from the given host. Columns correspond to different metrics; indexed in time. Return type: pandas.DataFrame
-
to_dataframe_by_metric
(metric)[source]¶ Returns data for a specific metric as a DataFrame
Parameters: metric (str) – Metric from which a DataFrame should be constructed Returns: All measurements of the given metric for all hosts. Columns represent hosts; indexed in time. Return type: pandas.DataFrame
-
-
tokio.connectors.mmperfmon.
get_col_pos
(line, align=None)[source]¶ Return column offsets of a left-aligned text table
For example, given the string:
Row Timestamp cpu_user cpu_sys mem_total 123456789x123456789x123456789x123456789x123456789x123456789x
would return:
[(0, 4), (15, 24), (25, 33), (34, 41), (44, 53)]
for
align=None
.Parameters: Returns: List of tuples of integer offsets denoting the start index (inclusive) and stop index (exclusive) for each column.
Return type:
-
tokio.connectors.mmperfmon.
value_unit_to_bytes
(value_unit)[source]¶ Converts a value+unit string into bytes
Converts a string containing both a numerical value and a unit of that value into a normalized value. For example, “1 MB” will convert to 1048576.
Parameters: value_unit (str) – Of the format “float str” where float is the value and str is the unit by which value is expressed. Returns: Number of bytes represented by value_unit Return type: int