tokio.connectors.esnet_snmp module¶

Provides interfaces into ESnet’s SNMP REST API

Documentation for the REST API is here:

http://es.net/network-r-and-d/data-for-researchers/snmp-api/

Notes

This connector relies either on the esnet_snmp_url configuration value being set in the pytokio configuration or the PYTOKIO_ESNET_SNMP_URI being defined in the runtime environment.

Examples

Retrieving the data of multiple endpoints (ESnet routers) and interfaces is a common pattern. To do this, the EsnetSnmp object should be initialized with only the intended start/end times, and the object should be asynchronously populated using calls to EsnetSnmp.get_interface_counters:

import datetime
import tokio.connectors.esnet_snmp

ROUTER = 'sunn-cr5'
INTERFACE = 'to_nersc_ip-d_v4'
TARGET_DATE = datetime.datetime.today() - datetime.timedelta(days=1)

# Because the ESnet API treats the end date as inclusive, we subtract
# one second to avoid counting the first measurement of the following
# day.
esnetsnmp = tokio.connectors.esnet_snmp.EsnetSnmp(
    start=TARGET_DATE,
    end=TARGET_DATE + datetime.timedelta(days=1, seconds=-1))
for direction in 'in', 'out':
    esnetsnmp.get_interface_counters(
        endpoint=ROUTER,
        interface=INTERFACE,
        direction=direction,
        agg_func='average')

for direction in 'in', 'out':
    bytes_per_sec = list(esnetsnmp[ROUTER][INTERFACE][direction].values())
    total_bytes = sum(bytes_per_sec) * esnetsnmp.timestep
    print("%s:%s saw %.2f TiB %s" % (
        ROUTER,
        INTERFACE,
        total_bytes / 2**40,
        direction))

For simple queries, it is sufficient to specify the endpoint, interface, and direction directly in the initialization:

esnetsnmp = tokio.connectors.esnet_snmp.EsnetSnmp(
    start=TARGET_DATE,
    end=TARGET_DATE + datetime.timedelta(days=1, seconds=-1)
    endpoint=ROUTER,
    interface=INTERFACE,
    direction="in")
print("Total bytes in: %.2f" % (
    sum(list(esnetsnmp[ROUTER][INTERFACE]["in"].values())) / 2**40))

class tokio.connectors.esnet_snmp.EsnetSnmp(start, end, endpoint=None, interface=None, direction=None, agg_func=None, interval=None, **kwargs)[source]¶

Bases: tokio.connectors.common.CacheableDict

Container for ESnet SNMP counters

Dictionary with structure:

{
    "endpoint0": {
        "interface_x": {
            "in": {
                timestamp1: value1,
                timestamp2: value2,
                timestamp3: value3,
                ...
            },
            "out": { ... }
        },
        "interface_y": { ... }
    },
    "endpoint1": { ... }
}

Various methods are provided to access the data of interest.

__init__(start, end, endpoint=None, interface=None, direction=None, agg_func=None, interval=None, **kwargs)[source]¶

Retrieves data rate data for an ESnet endpoint

Initializes the object with a start and end time. Optionally runs a REST API query if endpoint, interface, and direction are provided. Assumes that once the start/end time have been specified, they should not be changed.

Parameters:

start (datetime.datetime) – Start of interval to retrieve, inclusive
end (datetime.datetime) – End of interval to retrieve, inclusive
endpoint (str, optional) – Name of the ESnet endpoint whose data is being retrieved
interface (str, optional) – Name of the ESnet endpoint interface on the specified endpoint
direction (str, optional) – “in” or “out” to signify data input into ESnet or data output from ESnet
agg_func (str, optional) – Specifies the reduction operator to be applied over each interval; must be one of “average,” “min,” or “max.” If None, uses the ESnet default.
interval (int, optional) – Resolution, in seconds, of the data to be returned. If None, uses the ESnet default.
kwargs (dict) – arguments to pass to super.__init__()

Variables:

start (datetime.datetime) – Start of interval represented by this object, inclusive
end (datetime.datetime) – End of interval represented by this object, inclusive
start_epoch (int) – Seconds since epoch for self.start
end_epoch (int) – Seconds since epoch for self.end

_insert_result()[source]¶

Parse the raw output of the REST API and update self

ESnet’s REST API will return an object like:

{
    "agg": "30",
    "begin_time": 1517471100,
    "end_time": 1517471910,
    "cf": "average",
    "data": [
        [
            1517471100,
            5997486471.266666
        ],
...
        [
            1517471910,
            189300026.8
        ]
    ]
}

Parameters:	result (dict) – JSON structure returned by the ESnet REST API
Returns:	True if data inserted without errors; False otherwise
Return type:	bool

gen_url(endpoint, interface, direction, agg_func=None, interval=None, **kwargs)[source]¶

Builds parameters to be passed to requests.get() to retrieve data from the REST API.

This is factored out so that the URI generation can be tested without a working REST endpoint.

Parameters:	endpoint (str) – Name of ESnet endpoint (usually a router identifier) interface (str) – Name of the ESnet endpoint interface direction (str) – “in” or “out” to signify data input into ESnet or data output from ESnet agg_func (str or None) – Specifies the reduction operator to be applied over each interval; must be one of “average,” “min,” or “max.” If None, uses the ESnet default. interval (int or None) – Resolution, in seconds, of the data to be returned. If None, uses the ESnet default.
Returns:	kwargs to be passed directly to requests.get()
Return type:	dict

get_interface_counters(endpoint, interface, direction, agg_func=None, interval=None, **kwargs)[source]¶

Retrieves data rate data for an ESnet endpoint

This is factored the way it is so that it can be subclassed to use a faked remote REST endpoint for testing.

Parameters:	endpoint (str) – Name of ESnet endpoint (usually a router identifier) interface (str) – Name of the ESnet endpoint interface direction (str) – “in” or “out” to signify data input into ESnet or data output from ESnet agg_func (str or None) – Specifies the reduction operator to be applied over each interval; must be one of “average,” “min,” or “max.” If None, uses the ESnet default. interval (int or None) – Resolution, in seconds, of the data to be returned. If None, uses the ESnet default. kwargs (dict) – Extra parameters to pass to requests.get()
Returns:	raw return from the REST API call
Return type:	dict

load_json(*args, **kwargs)[source]¶

Loads input from serialized JSON

Need to coerce timestamp keys back into ints from strings

to_dataframe(multiindex=False)[source]¶

Return data as a Pandas DataFrame

Parameters:	multiindex (bool) – If True, return a DataFrame indexed by timestamp, endpoint, interface, and direction

tokio.connectors.esnet_snmp._get_interval_result(result)[source]¶

Parse the raw output of the REST API output and return the timestep

Parameters:	result (dict) – the raw JSON output of the ESnet REST API
Returns:	int or None

tokio.connectors.esnet_snmp._get_url(url, params, **kwargs)[source]¶

Wraps the requests.get() call and returns the results as a decoded JSON object.

Parameters:	url (str) – URL to request params (dict) – Same as requests.get(params) kwargs – Passed directly to requests.get()
Returns:	Result of the remote query
Return type:	dict