torch_opt
- class Optimiser(value)[source]
Bases:
enum.Enum
An enumeration.
- SGD = ('sgd', torch.optim.SGD)
- ASGD = ('asgd', torch.optim.ASGD)
- ADAGRAD = ('adagrad', torch.optim.Adagrad)
- ADADELTA = ('adadelta', torch.optim.Adadelta)
- ADAM = ('adam', torch.optim.Adam)
- ADAMW = ('adamw', torch.optim.AdamW)
- ADAMAX = ('adamax', torch.optim.Adamax)
- RMSPROP = ('rmsprop', torch.optim.RMSprop)
- RPROP = ('rprop', torch.optim.Rprop)
- LBFGS = ('lbfgs', torch.optim.LBFGS)
- classmethod from_name(name: str) sensai.torch.torch_opt.Optimiser
- classmethod from_name_or_instance(name_or_instance: Union[str, sensai.torch.torch_opt.Optimiser]) sensai.torch.torch_opt.Optimiser
- class NNLossEvaluator[source]
Bases:
abc.ABC
Base class defining the interface for training and validation loss evaluation.
- class Evaluation
Bases:
abc.ABC
- abstract start_epoch() None
Starts a new epoch, resetting any aggregated values required to ultimately return the epoch’s overall training loss (via getEpochTrainLoss) and validation metrics (via getValidationMetrics)
- abstract compute_train_batch_loss(model_output, ground_truth, x, y) torch.Tensor
Computes the loss for the given model outputs and ground truth values for a batch and aggregates the computed loss values such that :meth:
getEpochTrainLoss
can return an appropriate result for the entire epoch. The original batch tensors X and Y are provided as meta-information only.- Parameters
model_output – the model output
ground_truth – the ground truth values
x – the original batch input tensor
y – the original batch output (ground truth) tensor
- Returns
the loss (scalar tensor)
- abstract get_epoch_train_loss() float
- Returns
the epoch’s overall training loss (as obtained by collecting data from individual training batch data passed to computeTrainBatchLoss)
- abstract process_validation_batch(model_output, ground_truth, x, y) None
Processes the given model outputs and ground truth values in order to compute sufficient statistics for velidation metrics, which at the end of the epoch, shall be retrievable via method getValidationMetrics
- Parameters
model_output – the model output
ground_truth – the ground truth values
x – the original batch input tensor
y – the original batch output (ground truth) tensor
- Returns
the loss (scalar tensor)
- abstract get_validation_metrics() Dict[str, float]
- abstract start_evaluation(cuda: bool) sensai.torch.torch_opt.NNLossEvaluator.Evaluation
Begins the evaluation of a model, returning a (stateful) object which is to perform the necessary computations.
- Parameters
cuda – whether CUDA is being applied (all tensors/models on the GPU)
- Returns
the evaluation object
- abstract get_validation_metric_name() str
- Returns
the name of the validation metric which is to be used to determine the best model (key for the ordered dictionary returned by method Evaluation.getValidationMetrics)
- class NNLossEvaluatorFixedDim[source]
Bases:
sensai.torch.torch_opt.NNLossEvaluator
,abc.ABC
Base class defining the interface for training and validation loss evaluation, which uses fixed-dimension outputs and aggregates individual training batch losses that are summed losses per batch (averaging appropriately internally).
- class Evaluation(criterion, validation_loss_evaluator: sensai.torch.torch_opt.NNLossEvaluatorFixedDim.ValidationLossEvaluator, output_dim_weights: Optional[torch.Tensor] = None)
Bases:
sensai.torch.torch_opt.NNLossEvaluator.Evaluation
- __init__(criterion, validation_loss_evaluator: sensai.torch.torch_opt.NNLossEvaluatorFixedDim.ValidationLossEvaluator, output_dim_weights: Optional[torch.Tensor] = None)
- start_epoch()
Starts a new epoch, resetting any aggregated values required to ultimately return the epoch’s overall training loss (via getEpochTrainLoss) and validation metrics (via getValidationMetrics)
- compute_train_batch_loss(model_output, ground_truth, x, y) torch.Tensor
Computes the loss for the given model outputs and ground truth values for a batch and aggregates the computed loss values such that :meth:
getEpochTrainLoss
can return an appropriate result for the entire epoch. The original batch tensors X and Y are provided as meta-information only.- Parameters
model_output – the model output
ground_truth – the ground truth values
x – the original batch input tensor
y – the original batch output (ground truth) tensor
- Returns
the loss (scalar tensor)
- get_epoch_train_loss() float
- Returns
the epoch’s overall training loss (as obtained by collecting data from individual training batch data passed to computeTrainBatchLoss)
- process_validation_batch(model_output, ground_truth, x, y)
Processes the given model outputs and ground truth values in order to compute sufficient statistics for velidation metrics, which at the end of the epoch, shall be retrievable via method getValidationMetrics
- Parameters
model_output – the model output
ground_truth – the ground truth values
x – the original batch input tensor
y – the original batch output (ground truth) tensor
- Returns
the loss (scalar tensor)
- get_validation_metrics() Dict[str, float]
- start_evaluation(cuda: bool) sensai.torch.torch_opt.NNLossEvaluatorFixedDim.Evaluation
Begins the evaluation of a model, returning a (stateful) object which is to perform the necessary computations.
- Parameters
cuda – whether CUDA is being applied (all tensors/models on the GPU)
- Returns
the evaluation object
- abstract get_training_criterion() torch.nn.Module
Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).
- abstract get_output_dim_weights() Optional[numpy.ndarray]
- abstract create_validation_loss_evaluator(cuda: bool) ValidationLossEvaluator
- Parameters
cuda – whether to use CUDA-based tensors
- Returns
the evaluator instance which is to be used to evaluate the model on validation data
- get_validation_metric_name() str
Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.
- Returns
the name of the metric
- class ValidationLossEvaluator
Bases:
abc.ABC
- abstract start_validation_collection(ground_truth_shape)
Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.
- Parameters
ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)
- abstract process_validation_result_batch(output, ground_truth)
Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)
- Parameters
output – the model’s output
ground_truth – the corresponding ground truth
- abstract end_validation_collection() collections.OrderedDict
Computes validation metrics based on the data previously processed.
- Returns
an ordered dictionary with validation metrics
- class NNLossEvaluatorRegression(loss_fn: sensai.torch.torch_opt.NNLossEvaluatorRegression.LossFunction = LossFunction.L2LOSS, validation_tensor_transformer: Optional[sensai.torch.torch_data.TensorTransformer] = None, output_dim_weights: Optional[Sequence[float]] = None, apply_output_dim_weights_in_validation=True, validation_metric_name: Optional[str] = None)[source]
Bases:
sensai.torch.torch_opt.NNLossEvaluatorFixedDim
,sensai.util.string.ToStringMixin
A loss evaluator for (multi-variate) regression.
- class LossFunction(value)
Bases:
enum.Enum
An enumeration.
- L1LOSS = 'L1Loss'
- L2LOSS = 'L2Loss'
- MSELOSS = 'MSELoss'
- SMOOTHL1LOSS = 'SmoothL1Loss'
- __init__(loss_fn: sensai.torch.torch_opt.NNLossEvaluatorRegression.LossFunction = LossFunction.L2LOSS, validation_tensor_transformer: Optional[sensai.torch.torch_data.TensorTransformer] = None, output_dim_weights: Optional[Sequence[float]] = None, apply_output_dim_weights_in_validation=True, validation_metric_name: Optional[str] = None)
- Parameters
loss_fn – the loss function to use
validation_tensor_transformer – a transformer which is to be applied to validation tensors (both model outputs and ground truth) prior to computing the validation metrics
output_dim_weights – vector of weights to apply to then mean loss per output dimension, i.e. for the case where for each data point, the model produces n output dimensions, the mean loss for the i-th dimension is to be computed separately and be scaled with the weight, and the overall loss returned is the weighted average. The weights need not sum to 1 (normalisation is applied).
apply_output_dim_weights_in_validation – whether output dimension weights are also to be applied to to the metrics computed for validation. Note that this may not be possible if a validationTensorTransformer which changes the output dimensions is used.
validation_metric_name – the metric to use for model selection during validation; if None, use default depending on lossFn
- create_validation_loss_evaluator(cuda)
- Parameters
cuda – whether to use CUDA-based tensors
- Returns
the evaluator instance which is to be used to evaluate the model on validation data
- get_training_criterion()
Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).
- get_output_dim_weights() Optional[numpy.ndarray]
- class ValidationLossEvaluator(cuda: bool, validation_tensor_transformer: Optional[sensai.torch.torch_data.TensorTransformer], output_dim_weights: numpy.ndarray, apply_output_dim_weights: bool)
Bases:
sensai.torch.torch_opt.NNLossEvaluatorFixedDim.ValidationLossEvaluator
- __init__(cuda: bool, validation_tensor_transformer: Optional[sensai.torch.torch_data.TensorTransformer], output_dim_weights: numpy.ndarray, apply_output_dim_weights: bool)
- start_validation_collection(ground_truth_shape)
Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.
- Parameters
ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)
- process_validation_result_batch(output, ground_truth)
Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)
- Parameters
output – the model’s output
ground_truth – the corresponding ground truth
- end_validation_collection()
Computes validation metrics based on the data previously processed.
- Returns
an ordered dictionary with validation metrics
- get_validation_metric_name()
Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.
- Returns
the name of the metric
- class NNLossEvaluatorClassification(loss_fn: sensai.torch.torch_opt.NNLossEvaluatorClassification.LossFunction)[source]
Bases:
sensai.torch.torch_opt.NNLossEvaluatorFixedDim
A loss evaluator for classification
- class LossFunction(value)
Bases:
enum.Enum
An enumeration.
- CROSSENTROPY = 'CrossEntropy'
- NLL = 'NegativeLogLikelihood'
- create_criterion() Callable
- get_validation_metric_key() str
- classmethod default_for_output_mode(output_mode: sensai.torch.torch_enums.ClassificationOutputMode)
- __init__(loss_fn: sensai.torch.torch_opt.NNLossEvaluatorClassification.LossFunction)
- create_validation_loss_evaluator(cuda)
- Parameters
cuda – whether to use CUDA-based tensors
- Returns
the evaluator instance which is to be used to evaluate the model on validation data
- get_training_criterion()
Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).
- get_output_dim_weights() Optional[numpy.ndarray]
- class ValidationLossEvaluator(cuda: bool, loss_fn: sensai.torch.torch_opt.NNLossEvaluatorClassification.LossFunction)
Bases:
sensai.torch.torch_opt.NNLossEvaluatorFixedDim.ValidationLossEvaluator
- __init__(cuda: bool, loss_fn: sensai.torch.torch_opt.NNLossEvaluatorClassification.LossFunction)
- start_validation_collection(ground_truth_shape)
Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.
- Parameters
ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)
- process_validation_result_batch(output, ground_truth)
Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)
- Parameters
output – the model’s output
ground_truth – the corresponding ground truth
- end_validation_collection()
Computes validation metrics based on the data previously processed.
- Returns
an ordered dictionary with validation metrics
- get_validation_metric_name()
Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.
- Returns
the name of the metric
- class NNOptimiserParams(loss_evaluator: Optional[sensai.torch.torch_opt.NNLossEvaluator] = None, gpu: Optional[int] = None, optimiser: Union[str, sensai.torch.torch_opt.Optimiser] = 'adam', optimiser_lr=0.001, early_stopping_epochs=None, batch_size=None, epochs=1000, train_fraction=0.75, scaled_outputs=False, use_shrinkage=True, shrinkage_clip=10.0, shuffle=True, optimiser_args: Optional[Dict[str, Any]] = None)[source]
Bases:
sensai.util.string.ToStringMixin
- REMOVED_PARAMS = {'cuda'}
- RENAMED_PARAMS = {'batchSize': 'batch_size', 'earlyStoppingEpochs': 'early_stopping_epochs', 'lossEvaluator': 'loss_evaluator', 'optimiserClip': 'optimiser_clip', 'optimiserLR': 'optimiser_lr', 'scaledOutputs': 'scaled_outputs', 'shrinkageClip': 'shrinkage_clip', 'trainFraction': 'train_fraction', 'useShrinkage': 'use_shrinkage'}
- __init__(loss_evaluator: Optional[sensai.torch.torch_opt.NNLossEvaluator] = None, gpu: Optional[int] = None, optimiser: Union[str, sensai.torch.torch_opt.Optimiser] = 'adam', optimiser_lr=0.001, early_stopping_epochs=None, batch_size=None, epochs=1000, train_fraction=0.75, scaled_outputs=False, use_shrinkage=True, shrinkage_clip=10.0, shuffle=True, optimiser_args: Optional[Dict[str, Any]] = None)
- Parameters
loss_evaluator – the loss evaluator to use
gpu – the index of the GPU to be used (if CUDA is enabled for the model to be trained); if None, default to first GPU
optimiser – the optimiser to use
optimiser_lr – the optimiser’s learning rate
early_stopping_epochs – the number of epochs without validation score improvement after which to abort training and use the best epoch’s model (early stopping); if None, never abort training before all epochs are completed
batch_size – the batch size to use; for algorithms L-BFGS (optimiser=’lbfgs’), which do not use batches, leave this at None. If the algorithm uses batches and None is specified, batch size 64 will be used by default.
train_fraction – the fraction of the data used for training (with the remainder being used for validation). If no validation is to be performed, pass 1.0.
scaled_outputs – whether to scale all outputs, resulting in computations of the loss function based on scaled values rather than normalised values. Enabling scaling may not be appropriate in cases where there are multiple outputs on different scales/with completely different units.
use_shrinkage – whether to apply shrinkage to gradients whose norm exceeds
shrinkageClip
, scaling the gradient down toshrinkageClip
shrinkage_clip – the maximum gradient norm beyond which to apply shrinkage (if
useShrinkage
is True)shuffle – whether to shuffle the training data
optimiser_args – keyword arguments to be passed on to the actual torch optimiser
- classmethod from_dict_or_instance(nn_optimiser_params: Union[dict, sensai.torch.torch_opt.NNOptimiserParams]) sensai.torch.torch_opt.NNOptimiserParams
- classmethod from_dict(params: dict) sensai.torch.torch_opt.NNOptimiserParams
- classmethod from_either_dict_or_instance(nn_optimiser_dict_params: dict, nn_optimiser_params: Optional[sensai.torch.torch_opt.NNOptimiserParams])
- class NNOptimiser(params: sensai.torch.torch_opt.NNOptimiserParams)[source]
Bases:
object
- __init__(params: sensai.torch.torch_opt.NNOptimiserParams)
- Parameters
params – parameters
- fit(model: TorchModel, data: Union[sensai.torch.torch_data.DataUtil, List[sensai.torch.torch_data.DataUtil], sensai.torch.torch_data.TorchDataSetProvider, List[sensai.torch.torch_data.TorchDataSetProvider], sensai.torch.torch_data.TorchDataSet, List[sensai.torch.torch_data.TorchDataSet], Tuple[sensai.torch.torch_data.TorchDataSet, sensai.torch.torch_data.TorchDataSet], List[Tuple[sensai.torch.torch_data.TorchDataSet, sensai.torch.torch_data.TorchDataSet]]], create_torch_module=True) TrainingInfo
Fits the parameters of the given model to the given data, which can be a list of or single instance of one of the following:
a DataUtil or TorchDataSetProvider (from which a training set and validation set will be obtained according to the trainFraction parameter of this object)
a TorchDataSet which shall be used as the training set (for the case where no validation set shall be used)
a tuple with two TorchDataSet instances, where the first shall be used as the training set and the second as the validation set
- Parameters
model – the model to be fitted
data – the data to use (see variants above)
create_torch_module – whether to newly create the torch module that is to be trained from the model’s factory. If False, (re-)train the existing module.
- class TrainingInfo(best_epoch: Optional[int] = None, log: Optional[List[str]] = None, training_loss_sequence: Optional[Sequence[float]] = None, validation_metric_sequence: Optional[Sequence[float]] = None, total_epochs=None)[source]
Bases:
object
- __init__(best_epoch: Optional[int] = None, log: Optional[List[str]] = None, training_loss_sequence: Optional[Sequence[float]] = None, validation_metric_sequence: Optional[Sequence[float]] = None, total_epochs=None)
- get_training_loss_series() pandas.core.series.Series
- get_validation_metric_series() pandas.core.series.Series
- plot_all() matplotlib.figure.Figure
Plots both the sequence of training loss values and the sequence of validation metric values