Cacher#

class sdgx.cachers.base.Cacher(blocksize, *args, **kwargs)[source]#

Bases: object

Base class for cachers

Cacher is used to cache raw data and processed data to prevent repeat read or process.

You can treat Cacher as a warrper of DataConnector

clear_cache()[source]#

Clear all cache

clear_invalid_cache()[source]#

Clear invalid cache.

It’s useful when data source has been changed. Subclass can try to inspect cache and only clear invalid cache. Also, it may clear all cache when not sure or not support.

is_cached(offset: int) → bool[source]#

Check if the data is cached

iter(chunksize: int, data_connector: DataConnector) → Generator[DataFrame, None, None][source]#

Load data from data_connector or cache in chunk

load(offset: int, chunksize: int, data_connector: DataConnector) → DataFrame[source]#

Load data from data_connector or cache

load_all(data_connector: DataConnector) → DataFrame[source]#

Load all data from data_connector or cache

NoCache#

class sdgx.cachers.base.NoCache(blocksize, *args, **kwargs)[source]#

Bases: Cacher

No cache means to proxy data_connector

clear_cache()#

Clear all cache

clear_invalid_cache()#

Clear invalid cache.

It’s useful when data source has been changed. Subclass can try to inspect cache and only clear invalid cache. Also, it may clear all cache when not sure or not support.

is_cached(offset: int) → bool[source]#

Always return False

iter(chunksize: int, data_connector: DataConnector) → Generator[DataFrame, None, None][source]#

Proxy to data_connector.iter

load(offset: int, chunksize: int, data_connector: DataConnector) → DataFrame[source]#

Proxy to data_connector.read

load_all(data_connector: DataConnector) → DataFrame[source]#

Proxy to data_connector.read