tokio.connectors.slurm module

Connect to Slurm via Slurm CLI outputs.

This connector provides Python bindings to retrieve information made available through the standard Slurm saccount and scontrol CLI commands. It is currently very limited in functionality.

class tokio.connectors.slurm.Slurm(jobid=None, *args, **kwargs)[source]

Bases: tokio.connectors.common.SubprocessOutputDict

Dictionary subclass that self-populates with Slurm output data

Presents a schema that is keyed as:

    taskid: {
        slurmfield1: value1
        slurmfield2: value2

where taskid can be any of

  • jobid
  • jobid.<step>
  • jobid.batch
__init__(jobid=None, *args, **kwargs)[source]

Load basic information from Slurm

Parameters:jobid (str) – Slurm Job ID associated with data this object contains
Variables:jobid (str) – Slurm Job ID associated with data contained in this object

Serialize object in the same format as sacct.

Returns:Serialized version of self in a similar format as the sacct output so that this object can be circularly serialized and deserialized.
Return type:str

Convert own keys into native Python objects.

Scan self and convert special keys into native Python objects where appropriate. If no keys are given, scan everything. Do NOT attempt to recast anything that is not a string–this is to avoid relying on expand_nodelist if a key is already recast since expand_nodelist does not function outside of an environment containing Slurm.

Parameters:*target_keys (list, optional) – Only convert these keys into native Python object types. If omitted, convert all keys.

Initialize self from a JSON-encoded string.

Parameters:json_string (str) – JSON representation of self

Return the top-level jobid(s) contained in object.

Retrieve the jobid(s) contained in self without any accompanying taskid information.

Returns:list of jobid(s) contained in self.
Return type:list of str

Return a list of all job nodes used.

Creates a list of all nodes used across all tasks for the self.jobid. Useful if the object contains only a subset of tasks executed by the Slurm job.

Returns:Set of node names used by the job described by this object
Return type:set

Find earliest start and latest end time for a job.

For an entire job and all its tasks, find the absolute earliest start time and absolute latest end time.

Returns:Two-item tuple of (earliest start time, latest end time) in whatever type self['start'] and self['end'] are stored
Return type:tuple

Initialize values either from cache or sacct


Retrieve a list of keys from sacct and insert them into self.

This always invokes sacct and can be used to overwrite the contents of a cache file.

Parameters:*keys (list) – Slurm attributes to include; names should be valid input to sacct –format CLI utility.

Load from either a json cache or the output of sacct


Convert self into a Pandas DataFrame.

Returns a Pandas DataFrame representation of this object.

Returns:DataFrame representation of the same schema as the Slurm sacct command.
Return type:pandas.DataFrame

Return a json-encoded string representation of self.

Serializes self to json using _RECAST_KEY_MAP to convert Python types back into JSON-compatible types.

Returns:JSON representation of self
Return type:str
class tokio.connectors.slurm.SlurmEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)[source]

Bases: json.encoder.JSONEncoder

Encode sets as lists and datetimes as ISO 8601.


Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
        iterable = iter(o)
    except TypeError:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)
tokio.connectors.slurm._RECAST_KEY_MAP = {'end': (<function <lambda>>, <function <lambda>>), 'nodelist': (<function expand_nodelist>, <function compact_nodelist>), 'start': (<function <lambda>>, <function <lambda>>)}

Methods to convert Slurm string outputs into Python objects

This table provides the methods to apply to various Slurm output keys to convert them from strings (the default Slurm output type) into more useful Python objects such as datetimes or lists.

  • value[0] is the function to cast to Python
  • value[1] is the function to cast back to a string

Convert a string of nodes into compact representation.

Wraps scontrol show hostlist nid05032,nid05033,... to compress a list of nodes to a Slurm nodelist string. This is effectively the reverse of expand_nodelist()

Parameters:node_string (str) – Comma-separated list of node names (e.g., nid05032,nid05033,...)
Returns:The compact representation of node_string (e.g., nid0[5032-5159])
Return type:str

Expand Slurm compact nodelist into a set of nodes.

Wraps scontrol show hostname nid0[5032-5159] to expand a Slurm nodelist string into a list of nodes.

Parameters:node_string (str) – Node list in Slurm’s compact notation (e.g., nid0[5032-5159])
Returns:Set of strings which encode the fully expanded node names contained in node_string.
Return type:set
tokio.connectors.slurm.jobs_running_between(start, end, keys=None)[source]

Generate a list of Slurm jobs that ran between a time range

  • start (datetime.datetime) – Find jobs that ended at or after this time
  • end (datetime.datetime) – Find jobs that started at or before this time
  • state (str) – Any valid sacct state
  • keys (list) – List of Slurm fields to return for each running job

Slurm object containing jobs whose runtime overlapped with the start and end times

Return type:



Convert output of sacct -p into a dictionary.

Parses the output of sacct -p and return a dictionary with the full (raw) contents.

Parameters:sacct_str (str) – stdout of an invocation of sacct -p
Returns:Keyed by Slurm Job ID and whose values are dicts containing key-value pairs corresponding to the Slurm quantities returned by sacct -p.
Return type:dict