As a forecaster it is very useful to know how good your forecasts are.
It is straightforward to calculate the error of previous forecasts. For example, I can calculate the Mean Absolute Error of my week-ahead forecasting of daily email arrivals at a customer operation. Let’s say the MAE is 37 for the month of April. This may have been the lowest error I have ever achieved for any month, but how do I know whether or not this level of accuracy is a good result?
Further, I may find that Mean Absolute Error increases significantly at times, even though there has been no change in the forecasting method or the attention I have given to it. How can I ascertain whether the increase to error is due to the inherent variation in the data rather than poor performance of the model?
One way of answering these questions is to examine the quality of the forecasting model by comparing a forecast to what a simple method would have delivered, and see how much more accuracy your forecasting actually provided.
When making a short term forecast of daily data, a simple method might be to take last week’s data of the same day. Or take the mean of the last four values from the same day of week. One method I have used for a long time is to take a weighted average of the last four values of the same day of week using the weights 0.4, 0.3, 0.2 and 0.1, which I call the 4321 method.
Compare the error of your forecasting model with the error that would have resulted from a simple forecast, and see how much more accurate it is. Importantly, make sure that your simple forecast only uses data that was available when you were making your original forecast!
Then you will understand how much more accurate your forecasting is compared to the simple method.
When I visit organisations to help improve operations planning, I often calculate what the error from the 4321 forecast would have been and compare it to the error actually achieved. In some cases, I have found that the 4321 forecast strongly outperforms the model the client has been using, which suggests there are significant opportunities to improve the forecasting accuracy, and that there are commercial gains to be made.
