I have a discrete random variable $Z$. Every possible outcome of $Z$ has a given probability $p(z)$ and a value given by some normal distribution with unknown mean, but known variance $z_i \sim \mathcal{N}(\mu_{z_i},\,\sigma_{z_i}^{2})$. As a simple example, let's say $Z$ can take three values $(\mathcal{N}(\mu_{z_1},1^{2}), \mathcal{N}(\mu_{z_2},2^{2}), \mathcal{N}(\mu_{z_3},3^{2}))$ with probabilities $(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$.
I am interested in estimating the expected value of $Z$ by sampling a fixed number of samples from $Z$ (sampling budget is equal to $N$). However, I can decide whether I want sample from $Z$, or from some $z_i$ directly (here, I am constrained to choosing only $z_i$ that have been sampled earlier by sampling from $Z$). I am interested in 'optimal' distribution of budget between sampling from $Z$ and from some specific $z_i$, where 'optimal' is defined as a minimal variance estimator of $E(Z)$. Going back to the example - given a budget of three samples, I can first sample two values from $Z$, and then either sample a third value from $Z$ or sample another $z_i$ that has the biggest variance.
Denoting $E(\hat{Z})$ as sample mean of $Z$, $VAR(Z)$ as variance of $Z$ given known expected values of each $z_i$ and given i.d.d. sampling of $N$ samples from $Z$, the total variance of $E(\hat{Z})$ is equal to $VAR(E(\hat{Z})) = \frac{Var(Z)+ E_{z_i \sim Z}(Var(z_i))}{N}$.
Questions:
If I can also sample from $z_i$ directly, is $VAR(E(\hat{Z})) = \frac{Var(Z)+ E_{z_i \sim Z}(\frac{Var(z_i)}{N_{z_i}})}{N}$ correct? $N_{z_i}$ denotes number of samples drawn directly from $z_i$. What if I can draw only from $z_i$ that have been drawn from $N$ samples?
If the above is correct, can I just calculate the partial derivatives of variance with respect to $N$ and some $N_{z_i}$ and draw $z_i$ if the derivatives imply so?
Apologies if the question is messy - this is my first time asking question on such forum.
Many thanks, Michael
