Risk management is one of the core requirements of a MLOps process.

Understand the risks
Document the risks
Mitigate the risks

Risks and their stakes are different for each organization. In certain industries, particularly the finance sector, the Model Risk Management (MRM) function is crucial for regulatory compliance.

You should periodically assess:

• What if the model is unavailable for a given period of time?
• What are the consequences if the model makes a bad prediction?
• What if the model acts in the worst way imaginable?
• What if a user can extract training data or the internal logic of the model?
• What happens when the model accuracy or fairness decreases over time?
• What are the financial, business, legal, safety, and reputation risks?
• What if the people necessary to maintain the model leave the team?

Predictive model risk originates from e.g.

• Low quality of training data
• High difference between production data and training data
• Errors in designing; data prep, training, or evaluating
• Bugs in the runtime framework
• Expected error rates, but failures have higher consequences than expected
• Misuse of the model or misinterpretation of its outputs
• Legal risk from copyright infringement or liability for the model output
• Reputation risk due to unintended bias
• Adversarial attacks

The probability of risk and its magnitude can be amplified by:

• Broad usage of the model
• A rapidly changing environment
• Complex interactions between models
• Unknown data sources which maybe subject to regulation

Interactions between models is the most challenging source of risk. This increases the unknown unknowns a lot, but not the most common issue.

Sources

• Introducing MLOps: How To Scale Machine Learning In The Enterprise