tokio.connectors.nersc_globuslogs module

Retrieve Globus transfer logs from NERSC’s Elasticsearch infrastructure

Connects to NERSC’s OMNI service and retrieves Globus transfer logs.

class tokio.connectors.nersc_globuslogs.NerscGlobusLogs(*args, **kwargs)

Bases: tokio.connectors.es.EsConnection

Connection handler for NERSC Globus transfer logs

classmethod from_cache(*args, **kwargs)

Initializes an EsConnection object from a cache file.

This path is designed to be used for testing.

Parameters:cache_file (str) – Path to the JSON formatted list of pages

query(start, end, must=None, scroll=True)

Queries Elasticsearch for Globus logs

Accepts a start time, end time, and an optional “must” field which can be used to apply additional term queries. For example, must may be:

[
    {
        "term": {
            "TASKID": "none"
        }
    },
    {
        "term: {
            "TYPE": "STOR"
        }
    }
]

which would return only those queries that have no associated TASKID and were sending (storing) data.

Parameters:

• start (datetime.datetime) – lower bound for query (inclusive)
• end (datetime.datetime) – upper bound for query (exclusive)
• must (list or None) – list of dictionaries to be inserted as additional term-level query parameters.
• scroll (bool) – Use the scrolling interface if True. If False, source_filter/filter_function/flush_every/flush_function are ignored.

query_timeseries(query_template, start, end, scroll=True)

Craft and issue query that returns all overlapping records

Parameters:

• query_template (dict) – a query object containing at least one @timestamp field
• start (datetime.datetime) – lower bound for query (inclusive)
• end (datetime.datetime) – upper bound for query (exclusive)
• scroll (bool) – Use the scrolling interface if True. If False, source_filter/filter_function/flush_every/flush_function are ignored.

query_type(start, end, xfer_type)

Wraps query() with a type restriction

Convenience method to constrain a query to a specific transfer type.

Parameters:

• start (datetime.datetime) – lower bound for query (inclusive)
• end (datetime.datetime) – upper bound for query (exclusive)
• xfer_type (str) – constrain results to this transfer type (STOR, RETR, etc). Case sensitive.

query_user(start, end, user)

Wraps query() with a user restriction

Convenience method to constrain a query to a specific user.

Parameters:

• start (datetime.datetime) – lower bound for query (inclusive)
• end (datetime.datetime) – upper bound for query (exclusive)
• user (str) – constrain results to this username

to_dataframe()

Converts self.scroll_pages to a DataFrame

Returns:Contents of the last query’s pages Return type:pandas.DataFrame