cache
- class BoxedValue(value: sensai.util.cache.TValue)[source]
Bases:
Generic
[sensai.util.cache.TValue
]Container for a value, which can be used in caches where values may be None (to differentiate the value not being present in the cache from the cached value being None)
- __init__(value: sensai.util.cache.TValue)
- class KeyValueCache(*args, **kwds)[source]
Bases:
Generic
[sensai.util.cache.TKey
,sensai.util.cache.TValue
],abc.ABC
- abstract set(key: sensai.util.cache.TKey, value: sensai.util.cache.TValue)
Sets a cached value
- Parameters
key – the key under which to store the value
value – the value to store; since None is used indicate the absence of a value, None should not be used a value
- abstract get(key: sensai.util.cache.TKey) Optional[sensai.util.cache.TValue]
Retrieves a cached value
- Parameters
key – the lookup key
- Returns
the cached value or None if no value is found
- class InMemoryKeyValueCache[source]
Bases:
sensai.util.cache.KeyValueCache
[sensai.util.cache.TKey
,sensai.util.cache.TValue
],Generic
[sensai.util.cache.TKey
,sensai.util.cache.TValue
]A simple in-memory cache (which uses a dictionary internally).
This class can be instantiated directly, but for better typing support, one can instead inherit from it and provide the types of the key and value as type arguments. For example for a cache with string keys and integer values:
class MyCache(InMemoryKeyValueCache[str, int]): pass
- __init__()
- set(key: sensai.util.cache.TKey, value: sensai.util.cache.TValue)
Sets a cached value
- Parameters
key – the key under which to store the value
value – the value to store; since None is used indicate the absence of a value, None should not be used a value
- get(key: sensai.util.cache.TKey) Optional[sensai.util.cache.TValue]
Retrieves a cached value
- Parameters
key – the lookup key
- Returns
the cached value or None if no value is found
- empty()
- class PersistentKeyValueCache(*args, **kwds)[source]
Bases:
sensai.util.cache.KeyValueCache
[sensai.util.cache.TKey
,sensai.util.cache.TValue
],Generic
[sensai.util.cache.TKey
,sensai.util.cache.TValue
],abc.ABC
- class PersistentList(*args, **kwds)[source]
Bases:
Generic
[sensai.util.cache.TValue
],abc.ABC
- abstract append(item: sensai.util.cache.TValue)
Adds an item to the cache
- Parameters
item – the item to store
- abstract iter_items() Iterator[sensai.util.cache.TValue]
Iterates over the items in the persisted list
- Returns
generator of item
- class DelayedUpdateHook(fn: Callable[[], Any], time_period_secs, periodically_executed_fn: Optional[Callable[[], Any]] = None)[source]
Bases:
object
Ensures that a given function is executed after an update happens, but delay the execution until there are no further updates for a certain time period
- __init__(fn: Callable[[], Any], time_period_secs, periodically_executed_fn: Optional[Callable[[], Any]] = None)
- Parameters
fn – the function to eventually call after an update
time_period_secs – the time that must pass while not receiving further updates for fn to be called
periodically_executed_fn – a function to execute periodically (every timePeriodSecs seconds) in the busy waiting loop, which may, for example, log information or apply additional executions, which must not interfere with the correctness of the execution of fn
- handle_update()
Notifies of an update and ensures that the function passed at construction is eventually called (after no more updates are received within the respective time window)
- class PeriodicUpdateHook(check_interval_secs: float, no_update_time_period_secs: Optional[float] = None, no_update_fn: Optional[Callable[[], Any]] = None, periodic_fn: Optional[Callable[[], Any]] = None)[source]
Bases:
object
Periodically checks whether a function shall be called as a result of an update, the function potentially being non-atomic (i.e. it may take a long time to execute such that new updates may come in while it is executing). Two function all mechanisms are in place:
a function which is called if there has not been a new update for a certain time period (which may be called several times if updates come in while the function is being executed)
a function which is called periodically
- __init__(check_interval_secs: float, no_update_time_period_secs: Optional[float] = None, no_update_fn: Optional[Callable[[], Any]] = None, periodic_fn: Optional[Callable[[], Any]] = None)
- Parameters
check_interval_secs – the time period, in seconds, between checks
no_update_time_period_secs – the time period after which to execute noUpdateFn if no further updates have come in. This must be at least as large as checkIntervalSecs. If None, use checkIntervalSecs.
no_update_fn – the function to call if there have been no further updates for noUpdateTimePeriodSecs seconds
periodic_fn – a function to execute periodically (every checkIntervalSecs seconds) in the busy waiting loop, which may, for example, log information or apply additional executions, which must not interfere with the correctness of the execution of fn
- handle_update()
Notifies of an update, making sure the functions given at construction will be called as specified
- class PicklePersistentKeyValueCache(pickle_path, version=1, save_on_update=True, deferred_save_delay_secs=1.0)[source]
Bases:
sensai.util.cache.PersistentKeyValueCache
[sensai.util.cache.TKey
,sensai.util.cache.TValue
]Represents a key-value cache as a dictionary which is persisted in a file using pickle
- __init__(pickle_path, version=1, save_on_update=True, deferred_save_delay_secs=1.0)
- Parameters
pickle_path – the path of the file where the cache values are to be persisted
version – the version of cache entries. If a persisted cache with a non-matching version is found, it is discarded
save_on_update – whether to persist the cache after an update; the cache is saved in a deferred manner and will be saved after deferredSaveDelaySecs if no new updates have arrived in the meantime, i.e. it will ultimately be saved deferredSaveDelaySecs after the latest update
deferred_save_delay_secs – the number of seconds to wait for additional data to be added to the cache before actually storing the cache after a cache update
- save()
Saves the cache in the file whose path was provided at construction
- get(key: sensai.util.cache.TKey) Optional[sensai.util.cache.TValue]
Retrieves a cached value
- Parameters
key – the lookup key
- Returns
the cached value or None if no value is found
- set(key: sensai.util.cache.TKey, value: sensai.util.cache.TValue)
Sets a cached value
- Parameters
key – the key under which to store the value
value – the value to store; since None is used indicate the absence of a value, None should not be used a value
- class SlicedPicklePersistentList(directory, pickle_base_name, num_entries_per_slice=100000)[source]
Bases:
sensai.util.cache.PersistentList
Object handling the creation and access to sliced pickle caches
- __init__(directory, pickle_base_name, num_entries_per_slice=100000)
- Parameters
directory – path to the directory where the sliced caches are to be stored
pickle_base_name – base name for the pickle, where slices will have the names {pickleBaseName}_sliceX.pickle
num_entries_per_slice – how many entries should be stored in each cache
- append(item)
Append item to cache :param item: entry in the cache
- iter_items() Iterator[Any]
Iterate over entries in the sliced cache :return: iterator over all items in the cache
- clear()
Clears the cache if it exists
- cache_exists() bool
Does this cache already exist :return: True if cache exists, False if not
- class SqlitePersistentKeyValueCache(path, table_name='cache', deferred_commit_delay_secs=1.0, key_type: sensai.util.cache.SqlitePersistentKeyValueCache.KeyType = KeyType.STRING, max_key_length=255)[source]
Bases:
sensai.util.cache.PersistentKeyValueCache
[sensai.util.cache.TKey
,sensai.util.cache.TValue
]- class KeyType(value)
Bases:
enum.Enum
An enumeration.
- STRING = ('VARCHAR(%d)',)
- INTEGER = ('LONG',)
- __init__(path, table_name='cache', deferred_commit_delay_secs=1.0, key_type: sensai.util.cache.SqlitePersistentKeyValueCache.KeyType = KeyType.STRING, max_key_length=255)
- Parameters
path – the path to the file that is to hold the SQLite database
table_name – the name of the table to create in the database
deferred_commit_delay_secs – the time frame during which no new data must be added for a pending transaction to be committed
key_type – the type to use for keys; for complex keys (i.e. tuples), use STRING (conversions to string are automatic)
max_key_length – the maximum key length for the case where the key_type can be parametrised (e.g. STRING)
- set(key: sensai.util.cache.TKey, value: sensai.util.cache.TValue)
Sets a cached value
- Parameters
key – the key under which to store the value
value – the value to store; since None is used indicate the absence of a value, None should not be used a value
- get(key: sensai.util.cache.TKey) Optional[sensai.util.cache.TValue]
Retrieves a cached value
- Parameters
key – the lookup key
- Returns
the cached value or None if no value is found
- iter_items()
- class SqlitePersistentList(path)[source]
Bases:
sensai.util.cache.PersistentList
- __init__(path)
- append(item)
Adds an item to the cache
- Parameters
item – the item to store
- iter_items()
Iterates over the items in the persisted list
- Returns
generator of item
- class CachedValueProviderMixin(cache: Optional[sensai.util.cache.KeyValueCache[sensai.util.cache.TKey, sensai.util.cache.TValue]] = None, cache_factory: Optional[Callable[[], sensai.util.cache.KeyValueCache[sensai.util.cache.TKey, sensai.util.cache.TValue]]] = None, persist_cache=False, box_values=False)[source]
Bases:
Generic
[sensai.util.cache.TKey
,sensai.util.cache.TValue
,sensai.util.cache.TData
],abc.ABC
Represents a value provider that can provide values associated with (hashable) keys via a cache or, if cached values are not yet present, by computing them.
- __init__(cache: Optional[sensai.util.cache.KeyValueCache[sensai.util.cache.TKey, sensai.util.cache.TValue]] = None, cache_factory: Optional[Callable[[], sensai.util.cache.KeyValueCache[sensai.util.cache.TKey, sensai.util.cache.TValue]]] = None, persist_cache=False, box_values=False)
- Parameters
cache – the cache to use or None. If None, caching will be disabled
cache_factory – a factory with which to create the cache (or recreate it after unpickling if persistCache is False, in which case this factory must be picklable)
persist_cache – whether to persist the cache when pickling
box_values – whether to box values, such that None is admissible as a value
- cached(fn: Callable[[], sensai.util.cache.T], pickle_path, function_name=None, validity_check_fn: Optional[Callable[[sensai.util.cache.T], bool]] = None, backend='pickle', protocol=4, load=True, version=None) sensai.util.cache.T [source]
Calls the given function unless its result is already cached (in a pickle), in which case it will read the cached result and return it.
Rather than directly calling this function, consider using the decorator variant
pickle_cached()
.- Parameters
fn – the function whose result is to be cached
pickle_path – the path in which to store the cached result
function_name – the name of the function fn (for the case where its __name__ attribute is not informative)
validity_check_fn – an optional function to call in order to check whether a cached result is still valid; the function shall return True if the result is still valid and false otherwise. If a cached result is invalid, the function fn is called to compute the result and the cached result is updated.
backend – pickle or joblib
protocol – the pickle protocol version
load – whether to load a previously persisted result; if False, do not load an old result but store the newly computed result
version – if not None, previously persisted data will only be returned if it was stored with the same version
- Returns
the result (either obtained from the cache or the function)
- pickle_cached(cache_base_path: str, filename_prefix: Optional[str] = None, filename: Optional[str] = None, backend='pickle', protocol=4, load=True, version=None)[source]
Function decorator for caching function results via pickle.
Add this decorator to any function to cache its results in pickle files. The function may have arguments, in which case the cache will be specific to the actual arguments by computing a hash code from their pickled representation.
- Parameters
cache_base_path – the directory where the pickle cache file will be stored
filename_prefix – a prefix of the name of the cache file to be created, to which the function name and, where applicable, a hash code of the function arguments as well as the extension “.cache.pickle” will be appended. The prefix need not end in a separator, as “-” will automatically be added between filename components.
filename – the full file name of the cache file to be created; if the function takes arguments, the filename must contain a placeholder ‘%s’ for the argument hash
backend – the serialisation backend to use (see dumpPickle)
protocol – the pickle protocol version to use
load – whether to load a previously persisted result; if False, do not load an old result but store the newly computed result
version – if not None, previously persisted data will only be returned if it was stored with the same version
- PickleCached(cache_base_path: str, filename_prefix: Optional[str] = None, filename: Optional[str] = None, backend='pickle', protocol=4, load=True, version=None)[source]
Function decorator for caching function results via pickle.
Add this decorator to any function to cache its results in pickle files. The function may have arguments, in which case the cache will be specific to the actual arguments by computing a hash code from their pickled representation.
- Parameters
cache_base_path – the directory where the pickle cache file will be stored
filename_prefix – a prefix of the name of the cache file to be created, to which the function name and, where applicable, a hash code of the function arguments as well as the extension “.cache.pickle” will be appended. The prefix need not end in a separator, as “-” will automatically be added between filename components.
filename – the full file name of the cache file to be created; if the function takes arguments, the filename must contain a placeholder ‘%s’ for the argument hash
backend – the serialisation backend to use (see dumpPickle)
protocol – the pickle protocol version to use
load – whether to load a previously persisted result; if False, do not load an old result but store the newly computed result
version – if not None, previously persisted data will only be returned if it was stored with the same version
- class LoadSaveInterface[source]
Bases:
abc.ABC
- abstract save(path: str) None
- abstract classmethod load(path: str) sensai.util.cache.T
- class PickleLoadSaveMixin[source]
Bases:
sensai.util.cache.LoadSaveInterface
- save(path: Union[str, pathlib.Path], backend='pickle')
Saves the instance as pickle
- Parameters
path –
backend – pickle, cloudpickle, or joblib
- classmethod load(path: Union[str, pathlib.Path], backend='pickle')
Loads a class instance from pickle
- Parameters
path –
backend – pickle, cloudpickle, or joblib
- Returns
instance of the present class