Yes, You can compute the correlation without using the whole dataset as follows.
Let
$$
\begin{aligned}
X = x_{ij}&~ i=1,\dots,n;~j = 1,\dots,p\text{ Whole dataset}\\
X_{1j}& \implies \text{oldest observation to be removed at feature }X_j\\
X_{(+)j}&\implies \text{new observation to be added at feature }X_j\\
\mu^{(t)}_j &\implies \text{the current mean of feature }X_j\\
\sigma^{(t)}_{jk} &\implies \text{the current covariance between }X_j \text{ and } X_k\\
\end{aligned}
$$
More definitions:
Suppose
$$
\begin{aligned}
D_j =& X_{(+)j} - X_{1j}\quad \text{the difference between the new observation and the old observation}\\
E_j =& X_{(+)j} - \mu_{j}^{(t+1)}\quad \text{the difference between the new observation and the updated mean}\\
F_j = & X_{1j} - \mu_{j}^{(t+1)}\quad \text{the difference between the old observation and the updated mean}
\end{aligned}
$$
Then:
$$
\begin{aligned}
\mu_j^{(t+1)} &= \frac{1}{n}\sum_{i=1}^n X_{ij} + \frac{X_{(+)j} - X_{1j}}{n}=\mu_{j}^{(t)} + \frac{D_j}{n} \\
\end{aligned}
$$
And
$$
\begin{aligned}
\sigma_{ij}^{(t+1)} =& \frac{1}{n-1}\sum_{i=1}^n \left(X_{ij}-\mu_{j}^{(t+1)}\right)\left(X_{ik}-\mu_{k}^{(t+1)}\right) + \frac{\left(X_{(+)j} - \mu_{j}^{(t+1)}\right)\left(X_{(+)k} - \mu_{k}^{(t+1)}\right)}{n-1}-
\frac{\left(X_{1j} - \mu_{j}^{(t+1)}\right)\left(X_{1k} - \mu_{k}^{(t+1)}\right)}{n-1}\\
=& \frac{1}{n-1}\sum_{i=1}^n \left(X_{ij}-\mu_{j}^{(t)} - \frac{D_j}{n}\right)\left(X_{ik}-\mu_{k}^{(t)} - \frac{D_k}{n}\right) + \frac{E_jE_k-F_jF_k}{n-1}\\
=&\sigma_{jk}^{t} + \frac{D_jD_k}{n(n-1)} + \frac{E_jE_k-F_jF_k}{n-1}\\
\end{aligned}
$$
Thus the Updating formula in Matrix notations to be used in Python
$$
\begin{aligned}
D =&~ X_{(+)} - X_{1}\quad \text{A vector of all the feature differences} \\
\mu^{t+1} =& ~\mu^{(t)} + \frac{D}{n}\\
E =& ~X_{(+)} - \mu^{(t+1)}\\
F = & ~X_{1} - \mu^{(t+1)}\\
\Sigma^{(t+1)} =& ~\Sigma^{(t)} + \frac{DD^\top}{n(n-1)} + \frac{EE^\top - FF^\top}{n-1}\\
S =& \sqrt{Diag(\Sigma^{(t+1)})}\\
R^{(t+1)} = & \Sigma^{(t+1)} {\oslash} SS^\top\quad \text{where }\oslash \text{ is the elementwise division.}
\end{aligned}
$$
Python code:
Note that we do have the data, previous mean, previous variance, number of observations:
import numpy as np
#Create a fake data
np.random.seed(48)
X = np.random.normal(np.random.uniform(-20, 100, 4),
np.random.uniform(2,5,4), (20, 4))
n = X.shape[0]# Will never Change!! You adding one obs and removing one
mu = X.mean(0)
covar = np.cov(X.T)
corr = np.corrcoef(X.T) # Current Correlation(Not Updated Direcctly)
# Convert covariance matrix to correlation
def cov2cor(Sigma):
S = np.sqrt(np.diag(Sigma))[:, None]
return Sigma / S.dot(S.T)
# Function to update the mean and covariance.
def update(Xnew, Xold, mu, cov, n):
D = (Xnew - Xold).ravel()[:, None]
mu_new = mu.ravel()[:, None] + D/n
E = Xnew.ravel()[:, None] - mu_new
F = Xold.ravel()[:, None] - mu_new
cov_new = cov + (D @ D.T/n + E @ E.T - F @ F.T)/(n-1)
return {'mu':mu_new.ravel(), 'covar':cov_new}
new_obs = np.array([-10, 100, 50, 30])
new = update(new_obs,X[0], mu, covar, n = X.shape[0]) # X[0] is the one to be removed
updated_cor = cov2cor(new['covar'])
print(updated_cor)
#compare to
## Updated Data:
Y = np.r_[X[1:], new_obs[None, :]] # Used Y instead of X. just to keep the two different
print(np.corrcoef(Y.T))
[[ 1. 0.30645714 0.06955658 -0.22865679]
[ 0.30645714 1. 0.51049261 0.45552015]
[ 0.06955658 0.51049261 1. 0.6746911 ]
[-0.22865679 0.45552015 0.6746911 1. ]]
[[ 1. 0.30645714 0.06955658 -0.22865679]
[ 0.30645714 1. 0.51049261 0.45552015]
[ 0.06955658 0.51049261 1. 0.6746911 ]
[-0.22865679 0.45552015 0.6746911 1. ]]
```
