Regression with same values for dependent variables?

Question

I am performing a regression task on a relatively small dataset (4000 observations). These 4000 observations are grouped in such a way, that if I look at the dependent variable, there are only about 170 distinct values, implying that for ~20 distinct sets of independent variables, I should have the same value in the dependent variable.

My initial approach, so far, has been to rely on a simple linear regression, however when I plot a scatter for the out-of-sample predicted v/s actual values, there seems to be no fit. Other models such as the tree-based RandomForest and XGBoost models show similar results.

I have tried multiple approaches, including relaxing the hyperparameters for my model and using k-fold cross validation to assess the performance over multiple sets, however the performance of the models remains the same.

I cannot expand this dataset, and reducing it to only contain 170 values isn't an option either given how the problem is structured (we want to see the effect of the independent variables in each observation on the dependent variable). What other techniques and methods should I look at to improve my model's performance?

Also, when I look at the in-sample predictions, I see that for my XGBoost model, for the same, unique dependent variable, it predicts the same value, eventhough the independent variables are slightly different. Shouldn't the model be predicting, even on the in-sample, slightly different values per observation?

More on the data - The independent variables are price and specifications of products and the dependent variable is the product price of another company, in the same category as the product whose features are used as independent variables.

Now, I have different products similar to the product's price that I am predicting for, and, the assumption here is, that, given a comparable product, I want to see what the price would be for a product in the same space which is yet to be released. With different comparables for each product, I should ideally be getting a distribution of prices as the output of my regression.

Answer 1

Let us write a simple linear regression model, using a random effect $\alpha$ common for the observations with a common response (so it would have 170 different realizations.) $$ Y_i=\mu+\alpha+X_i^T\beta +\epsilon_i $$ where $\epsilon_i$ is the error term. No consider one of the groups with a common response. Within that group we have $$\underbrace{Y_i-\mu-\alpha}_{\text{constant!}}=X_i^T\beta+\epsilon_i $$ so the covariables $X_i$ and $\epsilon_i$ cannot be independent. That can explain the estimation problems you see! since such dependence can destroy consistency properties of the least-squares estimator.

If you have, say, some independent calculation (not based on the regression modelling) of product costs, you could use that as an instrument in IV (Instrumental variables) estimation. Search this site. I have no experience with such methods, so take this only as a suggestion.

Addition: Since you are using cross-validation, for your data, to get a realistic cross-validation, keep the groups with common response together.