• Engineering Practices  »  Model Monitoring
• Showcase  »  Implementation Notes

Model Monitoring

Model Performance Estimation

The NannyML library implements two approaches to model performance estimation:

• Confidence-based performance estimation (CBPE) for binary and multiclass classification tasks
• Direct Loss Estimation (DLE) for regression tasks

Since the ML problem for the showcase is a classification task, the CBPE approach is applied.

Reporting

Both model performance estimation and data drift detection algorithm in NannyML operate on chunks of model predictions. Therefore, it is straightforward to generate periodic reports on the model's performance as part of a monitoring workflow.

The showcase implements a simple reporting mechanism that builds a containerized monitoring dashboard of the reported performance metrics based on the prediction stored in the inference log.

Similarly, these reports could be implemented as a scheduled workflow that generates reports on a daily, weekly, or monthly basis.