tokio.tools.lfsstatus module

Given a file system and a datetime, return summary statistics about the OST fullness at that time

tokio.tools.lfsstatus._summarize_failover(fs_data)[source]

Summarize failover data for a single time record

Given an fs_data dict, generate a dict of summary statistics. Expects fs_data dict of the form generated by parse_lustre_txt.get_failovers:

{
    "abnormal_ips": {
        "10.100.104.140": [
            "OST0087",
            "OST0086",
            ...
        ],
        "10.100.104.43": [
            "OST0025",
            "OST0024",
            ...
        ]
    },
    "mode": 1
}
Parameters:fs_data (dict) – a single timestamp and file system record taken from the output of nersc_lfsstate.NerscLfsOstMap.get_failovers
Returns:summary metrics about the state of failovers on the file system
Return type:dict
tokio.tools.lfsstatus._summarize_fullness(fs_data)[source]

Summarize fullness data for a single time record

Given an fs_data dict, generate a dict of summary statistics. Expects fs_data dict of form generated by nersc_lfsstate.NerscLfsOstFullness:

{
     "MDT0000": {
         "mount_pt": "/scratch1",
         "remaining_kib": 2147035984,
         "target_index": 0,
         "total_kib": 2255453580,
         "used_kib": 74137712
     },
     "OST0000": {
         "mount_pt": "/scratch1",
         "remaining_kib": 28898576320,
         "target_index": 0,
         "total_kib": 90767651352,
         "used_kib": 60894630700
     },
     ...
}
Parameters:fs_data (dict) – a single timestamp and file system record taken from a nersc_lfsstate.NerscLfsOstFullness object
Returns:summary metrics about the state of the file system fullness
Return type:dict
tokio.tools.lfsstatus.get_failures(file_system, datetime_target, **kwargs)[source]

Get file system failures

Is a convenience wrapper for get_summary.

Parameters:
  • file_system (str) – Logical name of file system whose data should be retrieved (e.g., cscratch)
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
  • cache_file (str) – Basename of file to search for the requested data
Returns:

various statistics about the file system fullness
Return type:

dict
tokio.tools.lfsstatus.get_failures_lfsstate(file_system, datetime_target, cache_file=None)[source]

Get file system failures from nersc_lfsstate connector

Wrapper around the generic get_lfsstate function.

Parameters:
  • file_system (str) – Lustre file system name of the file system whose data should be retrieved (e.g., snx11025)
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
  • cache_file (str) – Basename of file to search for the requested data
Returns:

Whatever is returned by get_lfsstate
tokio.tools.lfsstatus.get_fullness(file_system, datetime_target, **kwargs)[source]

Get file system fullness

Is a convenience wrapper for get_summary.

Parameters:
  • file_system (str) – Logical name of file system whose data should be retrieved (e.g., cscratch)
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
Returns:

various statistics about the file system fullness
Return type:

dict
Raises:

tokio.ConfigError – When no valid providers are found
tokio.tools.lfsstatus.get_fullness_hdf5(file_system, datetime_target)[source]

Get file system fullness from an HDF5 object

Given a file system name (e.g., snx11168) and a datetime object, return summary statistics about the OST fullness.

Parameters:
  • file_system (str) – Name of file system whose data should be retrieved
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
Returns:

various statistics about the file system fullness
Return type:

dict
Raises:

ValueError – if an OST name is encountered which does not conform to a naming convention from which an OST index can be derived
tokio.tools.lfsstatus.get_fullness_lfsstate(file_system, datetime_target, cache_file=None)[source]

Get file system fullness from nersc_lfsstate connector

Wrapper around the generic get_lfsstate function.

Parameters:
  • file_system (str) – Lustre file system name of the file system whose data should be retrieved (e.g., snx11025)
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
  • cache_file (str) – Basename of file to search for the requested data
Returns:

Whatever is returned by tokio.tools.lfsstatus.get_lfsstate()
tokio.tools.lfsstatus.get_lfsstate(file_system, datetime_target, metric, cache_file=None)[source]

Get file system fullness or failures

Given a file system name (e.g., snx11168) and a datetime object

  1. locate and load the lfs-df (fullness) or ost map (failures) file
  2. find the sample immediately preceding the datetime (don’t find one that overlaps it
  3. return summary statistics about the OST fullness or OST failures
Parameters:
  • file_system (str) – Lustre file system name of the file system whose data should be retrieved (e.g., snx11025)
  • datetime_target (datetime.datetime) – Time at which requested data should be retrieved
  • metric (str) – either “fullness” or “failures”
  • cache_file (str) – Basename of file to search for the requested data
Returns:

various statistics about the file system fullness
Return type:

dict
Raises:
  • ValueError – if metric does not contain a valid option
  • IOError – when no valid data sources can be found for the given date