![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/09\/brian-lozach.jpeg\")
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/04\/Medium-Blog.png\")
<\/a><\/span><\/div>TL; DR<\/h2><\/div>When dealing with forecasts models, the best approach is often to iterate continuously, adding some data sources, improving feature engineering, tweeking model parameters\u2026 Most of the time Data Scientists tends to be fixed to only one KPI (i.e.<\/em> RMSE, Forecast Accuracy\u2026). There is often lot more information behind those KPIs that needs to be analyzed to improve prediction. Building an appropriate visualization tool is a great mean to deep-dive into the model behavior, spot quickly pain-points of your model, and thus gain in accuracy efficiently.<\/p>\n<\/div>This article will develop the key questions one should ask him\/herself when evaluating a forecasting model, then present must-have visualizations to answer these questions and finally propose a quick implementation under a unified tool to gather all these visualization using Streamlit.<\/p>\n<\/div>
Context<\/h2><\/div>This article sums up what we learned from building a unified visualization tool to help Data Scientists, Software Engineers, Product Owners & Demand Planner (business experts) develop sell-in forecasting models for 10+ business units in a food & beverage company. Our models made forecasts at a Daily x Warehouse x Product level, for the coming 14 weeks. These were developed using boosting methods, and take into account product characteristics, historical sales, events and promotional data.<\/p>\n<\/div>
Key questions to ask when evaluating a Forecasting model<\/h2><\/div>1. Is the model good compared to the baseline?<\/strong>
\nHaving access to current predictions (for instance demand planner forecasts) on the same scope is very helpful. It allows a good understanding of the business behaviour on a specific period, product or location. The more you interview the Business, the more you gain insights, and the more you can implement the right features.<\/p>\n<\/div>2. Chasing over or under-predictions and drops in Forecast Accuracy<\/strong>
\nMust have questions are: Do I catch the global trend? Does the model catch known recurrent events like holidays, warehouses closure, school holidays? Do I have some accuracy drops on particular periods?
\nWhy is it important?<\/p>\n<\/div>- <\/i><\/span>If the model over-predicts, it induces stock increase and thus stock costs.<\/div><\/li>
- <\/i><\/span>If the model under-predicts, it induces out-of-stocks periods, thus missing sales opportunities and decreasing customer satisfaction.<\/div><\/li><\/ul>
Spotting such events is a great way to reach efficiently more accurate predictions. These are often well known by demand planners and quite simple to implement in your model when you have the information. For example, in many of our business units, some products were sold for school lunches. Introducing and preparing a feature representing school holidays led to a great increase in our accuracy on these particular periods.<\/p>\n<\/div>
3. Dealing with product specificities<\/strong>
\nIs my performance homogeneous on my products brands \/ families? Are there any other distinction between my products (products sold only during promotions periods, best-sellers vs. low volumes products, products )?
\nWhy is it important ?<\/p>\n<\/div>- <\/i><\/span>To validate the good behavior of the model on the whole scope.\u00a0<\/strong>Depending on the business need, a minimal accuracy can be required on the whole scope.<\/div><\/li>
- <\/i><\/span>\n
To identify critical products, that can be analyzed more precisely, and thus gain in accuracy.<\/p>\n<\/div><\/li><\/ul>
These questions help you understand the business more precisely. For example, splitting models based on the importance of the products in terms of volume often lead to the increase of the performance. Indeed, the demand for regular products is very different from the demand for promotional products or less common ones, which can be highly correlated with promotion periods or have a very sparse sales profile. In most of our cases, we trained distinct models to address those different types of products.<\/p>\n<\/div>
4. Do the constitutive effects are correctly taken into account?<\/strong>
\nIs the model correctly capturing promotions effects? Are there cannibalization effects? Does the model adapt well to exogenous phenomena (ex: strikes)?
\nWhy is it important?<\/p>\n<\/div>- <\/i><\/span>In most of the cases, promotions are an important part to drive demand and can lead to great spikes in sales data.<\/div><\/li>
- <\/i><\/span>It can lead to big waste \/ out-of-stocks phenomena.<\/div><\/li>
- <\/i><\/span>A more realistic model taking into account sophisticated effects will more likely be adopted by your future users.<\/div><\/li><\/ul>
When dealing with forecasts models, the best approach is often to iterate continuously, adding some data sources, improving feature engineering, tweeking model parameters\u2026 Most of the time Data Scientists tends to be fixed to only one KPI (i.e.<\/em> RMSE, Forecast Accuracy\u2026). There is often lot more information behind those KPIs that needs to be analyzed to improve prediction. Building an appropriate visualization tool is a great mean to deep-dive into the model behavior, spot quickly pain-points of your model, and thus gain in accuracy efficiently.<\/p>\n<\/div> This article will develop the key questions one should ask him\/herself when evaluating a forecasting model, then present must-have visualizations to answer these questions and finally propose a quick implementation under a unified tool to gather all these visualization using Streamlit.<\/p>\n<\/div> This article sums up what we learned from building a unified visualization tool to help Data Scientists, Software Engineers, Product Owners & Demand Planner (business experts) develop sell-in forecasting models for 10+ business units in a food & beverage company. Our models made forecasts at a Daily x Warehouse x Product level, for the coming 14 weeks. These were developed using boosting methods, and take into account product characteristics, historical sales, events and promotional data.<\/p>\n<\/div> 1. Is the model good compared to the baseline?<\/strong> 2. Chasing over or under-predictions and drops in Forecast Accuracy<\/strong> Spotting such events is a great way to reach efficiently more accurate predictions. These are often well known by demand planners and quite simple to implement in your model when you have the information. For example, in many of our business units, some products were sold for school lunches. Introducing and preparing a feature representing school holidays led to a great increase in our accuracy on these particular periods.<\/p>\n<\/div> 3. Dealing with product specificities<\/strong> To identify critical products, that can be analyzed more precisely, and thus gain in accuracy.<\/p>\n<\/div><\/li><\/ul> These questions help you understand the business more precisely. For example, splitting models based on the importance of the products in terms of volume often lead to the increase of the performance. Indeed, the demand for regular products is very different from the demand for promotional products or less common ones, which can be highly correlated with promotion periods or have a very sparse sales profile. In most of our cases, we trained distinct models to address those different types of products.<\/p>\n<\/div> 4. Do the constitutive effects are correctly taken into account?<\/strong>Context<\/h2><\/div>
Key questions to ask when evaluating a Forecasting model<\/h2><\/div>
\nHaving access to current predictions (for instance demand planner forecasts) on the same scope is very helpful. It allows a good understanding of the business behaviour on a specific period, product or location. The more you interview the Business, the more you gain insights, and the more you can implement the right features.<\/p>\n<\/div>
\nMust have questions are: Do I catch the global trend? Does the model catch known recurrent events like holidays, warehouses closure, school holidays? Do I have some accuracy drops on particular periods?
\nWhy is it important?<\/p>\n<\/div>
\nIs my performance homogeneous on my products brands \/ families? Are there any other distinction between my products (products sold only during promotions periods, best-sellers vs. low volumes products, products )?
\nWhy is it important ?<\/p>\n<\/div>
\nIs the model correctly capturing promotions effects? Are there cannibalization effects? Does the model adapt well to exogenous phenomena (ex: strikes)?
\nWhy is it important?<\/p>\n<\/div>
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/09\/article-brian-blog.png\" "\\"article-brian-blog\\"")
<\/span><\/div>