tokio.connectors.cachingdb module

This module provides generic infrastructure for retrieving data from a relational database that contains immutable data. It can use a local caching database (sqlite3) to allow for reanalysis on platforms that cannot access the original remote database or to reduce the load on remote databases.

class tokio.connectors.cachingdb.CachingDb(dbhost=None, dbuser=None, dbpassword=None, dbname=None, cache_file=None)[source]

Bases: object

Connect relational database with an optional caching layer interposed.

__init__(dbhost=None, dbuser=None, dbpassword=None, dbname=None, cache_file=None)[source]

Connect to a relational database.

If instantiated with a cache_file argument, all queries will go to that SQLite-based cache database. If this class is not instantiated with a cache_file argument, all queries will go out to the remote database.

If none of the connection arguments (db*) are specified, do not connect to a remote database and instead rely entirely on the caching database or a separate call to the connect() method.

Parameters:
  • dbhost (str, optional) – hostname for the remote database
  • dbuser (str, optional) – username to use when connecting to database
  • dbpassword (str, optional) – password for authenticating to database
  • dbname (str, optional) – name of database to use when connecting
  • cache_file (str, optional) – Path to an SQLite3 database to use as a caching layer.
Variables:
  • saved_results (dict) – in-memory data cache, keyed by table names and whose values are dictionaries with keys rows and schema. rows are a list of row tuples returned from earlier queries, and schema is the SQL statement required to create the table corresponding to rows.
  • last_hit (int) – a flag to indicate whether the last query was found in the caching database or the remote database
  • cache_file (str) – path to the caching database’s file
  • cache_db (sqlite3.Connection) – caching database connection handle
  • cache_db_ps (str) – paramstyle of the caching database as defined by PEP-0249
  • remote_db – remote database connection handle
  • remote_db_ps (str) – paramstyle of the remote database as defined by PEP-0249
_query_mysql(query_str, query_variables)[source]

Run a query against the MySQL database and return the full output.

Parameters:
  • query_str (str) – SQL query expressed as a string
  • query_variables (tuple) – parameters to be substituted into query_str if query_str is a parameterized query
_query_sqlite3(query_str, query_variables)[source]

Run a query against the cache database and return the full output.

Parameters:
  • query_str (str) – SQL query expressed as a string
  • query_variables (tuple) – parameters to be substituted into query_str if query_str is a parameterized query
close()[source]

Destroy connection objects.

Close the remote database connection handler and reset state of remote connection attributes.

close_cache()[source]

Close the cache database handler and reset caching db attributes.

connect(dbhost, dbuser, dbpassword, dbname)[source]

Establish remote db connection.

Connects to a remote MySQL database and defines the connection handler and paramstyle attributes.

Parameters:
  • dbhost (str) – hostname for the remote database
  • dbuser (str) – username to use when connecting to database
  • dbpassword (str) – password for authenticating to database
  • dbname (str) – name of database to use when connecting
connect_cache(cache_file)[source]

Open the cache database file and set the handler attribute.

Parameters:cache_file (str) – Path to the SQLite3 caching database file to be used.
drop_cache(tables=None)[source]

Flush saved results from memory.

If tables are specified, only drop those tables’ results. If no tables are provided, flush everything.

Parameters:tables (list, optional) – List of table names (str) to flush. If omitted, flush all tables in cache.
query(query_str, query_variables=(), table=None, table_schema=None)[source]

Pass a query through all layers of cache and return on the first hit.

If a table is specified, the results of this query can be saved to the cache db into a table of that name.

Parameters:
  • query_str (str) – SQL query expressed as a string
  • query_variables (tuple) – parameters to be substituted into query_str if query_str is a parameterized query
  • table (str, optional) – name of table in the cache database to save the results of the query
  • table_schema (str, optional) – when table is specified, the SQL line to initialize the table in which the query results will be cached.
Returns:

Tuple of tuples corresponding to rows of fields as returned by the SQL query.
Return type:

tuple
save_cache(cache_file)[source]

Commit the in-memory cache to a cache database.

This method is currently very memory-inefficient and not good for caching giant pieces of a database without something wrapping it to feed it smaller pieces.

Note

This manipulates the cache_db* attributes in a dirty way to prevent closing and re-opening the original cache db. If the self.open_cache() is ever changed to include tracking more state, this function must also be updated to retain that state while the old cache db state is being temporarily shuffled out.

Parameters:cache_file (str) – Path to the cache file to be used to write out the cache contents. This file will temporarily pre-empt the cache_file attribute and should be a different file.
tokio.connectors.cachingdb.get_paramstyle_symbol(paramstyle)[source]

Infer the correct paramstyle for a database.paramstyle

Provides a generic way to determine the paramstyle of a database connection handle. See PEP-0249 for more information.

Parameters:paramstyle (str) – Result of a generic database handler’s paramstyle attribute
Returns:The string corresponding to the paramstyle of the given database connection.
Return type:str