Health Metrics and Model Health; AirPro Phase 2 Services
AirPro is a leader in automotive diagnostic scanning and advanced driver assistance systems (ADAS) calibration. In a previous case study, we delved into NLP Logix’s work with AirPro on their calibration technology, the Auggie AirPro: A Case Study that Steps into Augmented Reality. This continuation explores the model health and maintenance aspects of this solution.
Model performance monitoring is a way to ensure deployed models are sustainable and performing optimally over time. Confirming that the deployed model aligns with expectations and remains stable is pivotal. NLP Logix conducts rigorous quality checks through audits. The auditing process guarantees a consistent evaluation of the model’s quality and performance in real-world scenarios. This approach is integral to maintaining a dependable and high-quality standard for the model throughout its deployment and usage.
Gnarly Problem
Much like any model, anticipated degradation over time is a natural aspect of the AirPro Solution for the Auggie. An illustration of this is the emergence of new car models with distinct features, angles, and positioning that cameras must detect. Any changes to technology will inevitably bleed into production, prompting a need to assess and proactively manage these changes across all deployed AirPro models.
Strategic Approach
NLP Logix aims to ensure that model health is maintained and examined with these changes in mind. Necessary changes can be made when it becomes apparent that the model health is dipping. Utilizing a systematic process, the Auggie metrics are evaluated retrospectively to confirm they are holding prospectively.
The approach for the Auggie model maintenance was on health-based retraining, not time-based retraining. Necessary data was pulled for testing from AirPro’s environment and brought into NLP Logix’s environment. Testing was then conducted to compare how closely a human’s response were, to what the Auggie’s response were, when presented the same problem. This evaluation gives insight into the performance of the Auggie. NLP Logix then analyzes and reports this data back to AirPro with how accurate the current Auggie model is performing.
Operational Success
In this case health-based training allows for a more adaptive (nuanced) and data-driven approach to managing and maintaining model performance.
With time-based training the model is retrained at fixed intervals, regardless of the model’s performance. With health-based training, there is a manual check conducted at a set time interval of the current state. This frequent analysis enables a more responsive strategy—only retraining the model when necessary. If the model is performing well, the approach is to refrain from making unnecessary adjustments, adhering to the principle of not fixing what isn’t broken.
Learn more about: Advanced Analytics • NLP Logix
Model performance is closely monitored for accuracy. Retrain alerts are triggered before the product becomes negatively impacted. In the case of the Auggie, when the max error is calculated above a .5 camera width, it is suggested to retrain the model.
