This question was inspired by this answer to the question: Linear regression: any non-normal distribution giving identity of OLS and MLE?
The setting is the following: assume the following regression model:
$$Y = f_{\beta}(X) + \epsilon,$$
where $\epsilon$ is a zero-mean random variable and $f_\beta$ a function of $x$ parametrized by $\beta$. We observe data $(x_i, y_i)_{i=1}^n$ and want to find an estimate for $\beta$ in the above model. This can be done with least squares regression:
$$\hat \beta = \arg \min_\beta \sum_{i=1}^n (y_i - f_\beta(x_i))^2. $$
Now, this corresponds to assuming a family of densities of the following form:
$$ p(y|x;\beta) =c g(x, y) \exp(-\kappa(y - f_\beta(x))^2),$$
where $g(x,y)$ does not depend on $\beta$ and $c$ is a constant, and performing MLE over $\beta$ with the above family: $\hat \beta = \arg \max_\beta \prod_{i=1}^n p(y_i|x_i;\beta) $.
Note that $g(x,y)$ above does not matter in maximizing the likelihood, so in principle it is unconstrained. However, I would like to understand if there is any condition $g(x,y)$ has to satisfy. Specifically, let us assume that the above model is valid for $\beta = \beta^\star$, ie the data were actually generated in that way for that value of $\beta$. This also means that the expected value $E[Y|X=x] = f_{\beta^\star} (x)$. Therefore, I believe the expectation of $Y$ under $p(y|x;\beta^\star)$ should be equal to $f_{\beta^\star} (x)$, ie:
$$ \int y\ p(y|x;\beta^\star) dy = c \int y\ g(x,y) e^{-\kappa(y - f_{\beta^\star}(x))^2} dy = f_{\beta^\star}(x), \ \forall\ x.$$
My question is: can we say anything about $g(x,y)$ given the latter expression? My intuition is that, as that holds for any $x$, if $f_{\beta^\star}(x)$ is not constant, there may be some constraints on how $g$ depends on both arguments, but I don't know how to go farther and am not sure about that.
