If you're accustomed to using technical analysis for your trading decisions, the leap to AI and ML models might seem a bit daunting. Don't be afraid. Understanding the key metrics and results from these models is less complicated than you'd think. In fact, many of the core principles are similar to what you already know!
What is a Model selection? There are usually many “good” explanations or as it’s formally called “models” of why a cryptocurrency price moves up and down. And based on each model, you could have different price predictions. For instance, if you create two predictive models—Model A emphasizes market sentiment, while Model B focuses on technical indicators—then a negative shift in market sentiment would more likely trigger a price drop prediction from Model A, as opposed to Model B.
Why Model Selection is important?
Accuracy: The best model is the best representation of how price dynamics work. This means that you will get the most accurate predictions with these kinds of models.
Consistency: Sometimes you may obtain accurate price predictions from a bad model, however, you can’t rely on that. Imagine someone who guesses that Cardano’s price will increase tomorrow, it might happen just by chance. In contrast, selecting the best model will guarantee consistently accurate predictions.
How to Model Select the best model?
1. Prediction error: Good models must have a low prediction error, this is your gold standard. But there is more you can know about the model from its evaluation results. In the “Prediction Error Per Point” graph, you can look for certain patterns:
A.Continuously increasing error: This is a bad sign. As mentioned in a previous article, the algorithm is trained on a portion of the data and gets tested on the following part of the data. When the prediction error is low at the beginning of the evaluation period and it goes gradually up, it means that the model gives good predictions over the short term but is becoming gradually insufficient to explain the price dynamics. Therefore, this model will not give you accurate predictions.
B. Spike in Prediction Error. This is usually because of a sudden price pump or dump which the model failed to predict. Most price pumps and dumps are impossible to predict, so a single spike in prediction error shouldn’t be a problem. However, if the prediction error continued to be high after the spike, this shows that the model is now insufficient. C. Single-sided prediction error. Sometimes, the model predictions tend to be always lower or higher than the real market price. In this case, you will see that the prediction error is always positive or always negative. This is valuable information for you, so when you look at the price predictions, you can have a better idea if the prediction is overestimated or underestimated.
2. Direction Accuracy: For some trade markets like binary options, traders are more interested in the predicted price direction, than the exact predicted price. In this case – and overall- you should choose models with higher direction accuracy. For more information about direction accuracy, check our previous article. 3. Log-likelihood: When comparing Econometric models with each other, models that have higher log-likelihoods are expected to provide higher accuracy. But be aware that log-likelihood is calculated on the training part of the data, so it’s less reliable than the other metrics which are calculated on the evaluation/test part of the data.
Conclusion Understanding model evaluation metrics and learning how to select the best models is an essential skill in AI / ML Analytics. This helps you gauge the reliability of the AI-based predictions. So, when you build a new model, you should first compare it with others and choose the best one, before you move to obtaining price predictions.
So the next time you’re on Coincharted or any other AI-driven trading platform, don’t be intimidated by these new metrics. Embrace them, understand them, and incorporate them into your trading strategy as you would with good old technical analysis.
