0

I created sample training data with set of random numbers and their squares. But when I predict square of a new number, none of the sklearn models are predicting it correctly. Given below is my sample code.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC


x = []
y = []

# Generate lot of random integers for train data
x = np.random.randint(10, 1000, (500, 1))
# print (x)

# Generate square for each integer and save in train data
for i in x:
    y.append(i[0] * i[0])

# print(x, y)

# List of Algorithms
models = []
models.append(('LinearRegression', LinearRegression()))
models.append(('LogisticRegression', LogisticRegression()))
models.append(('KNeighborsClassifier', KNeighborsClassifier()))
models.append(('DecisionTreeClassifier', DecisionTreeClassifier()))
models.append(('GaussianNB', GaussianNB()))
models.append(('SVC', SVC()))


# Loop through models and identify the best model to predict square
for name, model in models:
    model.fit(x, y)
    x_predict = 7
    y_predict = model.predict(x_predict)
    print(name, y_predict)

My Output is as below:

LinearRegression [-164374.36815163]
LogisticRegression [100]
KNeighborsClassifier [100]
DecisionTreeClassifier [100]
GaussianNB [100]
SVC [100]

What am I doing wrong? Is it not possible to use sklearn models to predict simple square pattern? Please help.

Mosu
  • 3
  • 2
  • You shouldn't even be considering logistic regression, or any of the "classifier" models for this exercise. Logistic regression, and in general classification, is for predicting the conditional probability of a *binary* response. It kind of looks like you threw a kitchen sink worth of models at a problem without any real thought into what was appropriate, this is not how predictive modeling works. – Matthew Drury Sep 10 '16 at 16:39

2 Answers2

0

Oh dear -- looks like you fell into the extrapolation-trap!

Remember that most statistical methods do no "predicting" at all -- it's actually quite an underappriciated -- but nuanced -- idea.* What most are actually doing is simply finding most representative values, according to association.

Looking at your code you are generating a training set by picking n=500 random integers in the values range of 10 to 1000, then you ask your models to extrapolate outside of that range to predict the square of a value of 7 -- which is not in the range [10, 1000]

I ran your code by changing the range from 0 to 20 as follows: 

# Generate lot of random integers for train data
x = np.random.randint(0, 50, (500, 1))

And found better results: 

('LinearRegression', array([-64.08766168]))
('LogisticRegression', array([36]))
('KNeighborsClassifier', array([49]))
('DecisionTreeClassifier', array([49]))
('GaussianNB', array([49]))
('SVC', array([49]))

Please note that your results are poor for both linear and logistic regression, as should be expected. Both of these methods are strictly linear, by construction, and would therefore poorly predict an exponential function. The others are closer to "picking correctly based on past observations" (to simplify extremely), and so long as your dataset well covers the x-value to predict (7, in this case), you should get good results. Improve your results by generating more training data around the value of 7 by modifying the upper and lower bounds of the "randint" function.

*For deeper discussion on the idea see the eloquent answer to this stack exchange question

Community
  • 1
Omeed
  • 91
  • 3
  • 1
    Hello Omeed, Thank you for the explanation on extrapolation trap. I was not aware of that. By the way, you got many models to predict 49 accurately because the random train data you generated also have 7 and its square. If you remove it from train data, then the predictions are not so close :) – Mosu Sep 10 '16 at 16:16
  • Yeah @Mosu makes a good point. Generating random floats instead of random ints might be a more meaningful (and useful) way to explore this idea. – Omeed Sep 10 '16 at 19:42
0

It's possible to use PolynomialFeatures if you suspect Y is affected by higher degrees of x.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
import numpy as np

# fit to an order-2 polynomial data
model = Pipeline([('poly', PolynomialFeatures(degree=2)),
                  ('linear', LinearRegression(fit_intercept=False))])

x = np.random.randint(10, 1000, (500, 1))
y = x ** 2

model = model.fit(x, y)

x_predict = [[7]]
y_predict = model.predict(x_predict)
print(y_predict)
Ay Mon
  • 1