0

I am new to both statistics and machine learning in general. I've tried to construct a price prediction model using the RNN-LSTM architecture. For this problem I have a dataset of one-minute closing prices of Bitcoin. The goal of this project is to quantify the accuracy of the prediction model. And this is my issue. I built the model, plotted it and applied the MAE metric, but I am not sure what to make of the resulted value, or rather how to compute the actual accuracy of the model in the range of 0-100%. The closing price values range from roughly 6670-6770 Is this possible? I would very much appreciate any suggestions. Thank you very much!

Code:

from sklearn.metrics import mean_absolute_error
mean_absolute_error(y_test, preds)

result: 6415.621912643506

FrantisekG
  • 3
  • 2
  • 1
    Welcome to CV and congratulations on your first post. Don't think of your model as a lookup table, a single line, where you put in a single input and get out a single output. Think of it as you put in a single input, and then it puts out a distribution of values, like a bell curve. Your fit tells you the "highest point" of that bell curve, the most likely of many likely values. You need to consider the sides, the prediction interval. That MAE tells you more about the spread. It gives you a sense of how far off your model is most of the time; how fat or narrow your distribution is. – EngrStudent Jan 10 '19 at 13:07

1 Answers1

1

There are many possible forecast accuracy measures, e.g. here and here.

Whether a given MAE is good enough for your needs is something only you can answer, based on what you plan on doing with your forecast. You can use accuracy measures to compare forecasts and select a more accurate model over a less accuracy one. (Use a holdout sample for this, do not calculate and interpret in-sample accuracy.)

There is no common accuracy measure that is scaled to lie in the interval between 0 and 100%. If your closing price is 6000, a forecast of 12000 or 0 will be 100% off... but you could also have a forecast of 18000 and be 200% off.

You may be interested in How to know that your machine learning problem is hopeless?

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thank you for your answer. I might have not been clear in the question. What I am trying to uncover is how, in most of the articles and papers written on the topic of creating a prediction price model, do the authors calculate the accuracy. For example here: http://intelligentonlinetools.com/blog/2018/01/19/machine-learning-stock-prediction-lstm-keras/ The author concludes an accuracy of 97% from test MSE = 0.0163, this to me is not clear. Thank you. – FrantisekG Jan 10 '19 at 10:17
  • Ah. Thank you. The page you link actually says "with testing MSE 0.015 and accuracy 97%", and does not say that the accuracy was calculated *based on* the MSE, so I don't know whether this assumption of yours is warranted. I am not aware of a *standard* way to calculate "accuracy" for numerical targets. The MSE, in contrast, is well defined, and accuracy is also defined for categorical predictions (and [a bad KPI](https://stats.stackexchange.com/q/312780/1352)). So it essentially seems to boil down to that page being unclear. – Stephan Kolassa Jan 10 '19 at 10:29