So, I've read some posts on covariance in this site and I think I have finally managed to get my head around the concept. However, there is still one thing about it that I really wish I had more intuition on, namely, when calculating the covariance between two random variables $X$ and $Y$, why do we average the product of their deviations from their respective means? In other words, why do we calculate
$$E[(X-E[X])(Y-E[Y])]$$
instead of just
$$E[XY]$$
? Wouldn't the latter calculation be enough to tell wether, on average, $X$ goes up when $Y$ goes up and vice-versa? Why would we wish to consider $(X-E[X])$ instead of just $X$? Thanks very much in advance.