tokio.connectors.nersc_isdct module

Connect to NERSC’s Intel Data Center Tool for SSDs outputs

Processes and aggregates the output of Intel Data Center Tool for SSDs outputs in the format generated by NERSC’s daily script. The NERSC infrastructure runs ISDCT in a verbose way on every burst buffer node, then collects all the resulting output text files from a node into a directory bearing that node’s nid (e.g., nid01234/*.txt). There is also an optional timestamp file contained in the toplevel directory. Also processes .tar.gz versions of these collected metrics.

class tokio.connectors.nersc_isdct.NerscIsdct(input_file)

Bases: tokio.connectors.common.CacheableDict

Dictionary subclass that self-populates with ISDCT output data

__init__(input_file)

Load the output of a NERSC ISDCT dump.

Parameters:input_file (str) – Path to either a directory or a tar(/gzipped) directory containing the output of NERSC’s ISDCT collection script.

_synthesize_metrics()

Calculate additional metrics not presented by ISDCT.

Calculates additional convenient metrics that are not directly presented by ISDCT, then adds the resulting key-value pairs to self.

diff(old_isdct, report_zeros=True)

Highlight differences between self and another NerscIsdct.

Subtract each counter for each serial number in this object from its counterpart in old_isdct. Return the changes in each numeric counter and any serial numbers that have appeared or disappeared.

Parameters:

• old_isdct (NerscIsdct) – object with which we should be compared
• report_zeros (bool) – If True, report all counters even if they showed no change. Default is True.

Returns:

Dictionary containing the following keys:

• added_devices - device serial numbers which exist in self but not old_isdct
• removed_devices - device serial numbers which do not exist in self but do in old_isdct
• devices - dict keyed by device serial numbers and whose values are dicts of keys whose values are the difference between old_isdct and self

Return type:

dict

load()

Wrapper around the filetype-specific loader.

Infer the type of input being given, dispatch the correct loading function, and populate keys/values.

load_native()

Load ISDCT output from a tar(.gz).

Load a collection of ISDCT outputs as created by the NERSC ISDCT script. Assume that ISDCT output files each contain a single output from a single invocation of the isdct tool, and outputs are grouped into directories named according to their nid numbers (e.g., nid00984/somefile.txt).

to_dataframe(only_numeric=False)

Express self as a dataframe.

Parameters:only_numeric (bool) – Only output columns containing numeric data of True; otherwise, output all columns. Returns:Dataframe indexed by serial number and with ISDCT counters as columns Return type:pandas.DataFrame

tokio.connectors.nersc_isdct._decode_nersc_nid(path)

Convert path to ISDCT output into a nid.

Given a path to some ISDCT output file, somehow figure out what the nid name for that node is. This encoding is specific to the way NERSC collects and preserves ISDCT outputs.

Parameters:path (str) – path to an ISDCT output text file Returns:Node identifier (e.g., nid01234) Return type:str

tokio.connectors.nersc_isdct._merge_parsed_counters(parsed_counters_list)

Merge ISDCT outputs into a single object.

Aggregates counters from each record based on the NVMe device serial number, with redundant counters being overwritten.

Parameters:parsed_counters_list (list) – List of parsed ISDCT outputs as dicts. Each list element is a dict with a single key (a device serial number) and one or more values; each value is itself a dict of key-value pairs corresponding to ISDCT/SMART counters from that device. Returns:Dict with keys given by all device serial numbers found in parsed_counters_list and whose values are a dict containing keys and values representing all unique keys across all elements of parsed_counters_list. Return type:dict

tokio.connectors.nersc_isdct._normalize_key(key)

Coerce all keys into a similar naming convention.

Converts Intel’s mix of camel-case and snake-case counters into all snake-case. Contains some nasty acronym hacks that may require modification if/when Intel adds new funny acronyms that contain a mix of upper and lower case letters (e.g., SMBus and NVMe).

Parameters:key (str) – a string to normalize Returns:snake_case version of key Return type:str

tokio.connectors.nersc_isdct._rekey_smart_buffer(smart_buffer)

Convert SMART values associated with one register into unique counters.

Take a buffer containing smart values associated with one register and create unique counters. Only necessary for older versions of ISDCT which did not output SMART registers in a standard “Key: value” text format.

Parameters:smart_buffer (dict) – SMART buffer as defined by parse_counters_fileobj() Returns:unique key:value pairs whose key now includes distinguishing device-specific characteristics to avoid collision from other devices that generated SMART data Return type:dict

tokio.connectors.nersc_isdct.parse_counters_fileobj(fileobj, nodename=None)

Convert any output of ISDCT into key-value pairs.

Reads the output of a file-like object which contains the output of a single isdct command. Understands the output of the following options:

• isdct show -smart (SMART attributes)
• isdct show -sensor (device health sensors)
• isdct show -performance (device performance metrics)
• isdct show -a (drive info)

Parameters:

• fileobj (file) – file-like object containing the output of an ISDCT command
• nodename (str) – name of node corresponding to fileobj, if known

Returns:

dict of dicts keyed by the device serial number.

Return type:

dict