Suppose I have an episodic Markov Decision Process where all episodes start in the same state, $s_0$. I also have a parameterized policy $\pi_\theta$, and I'm trying to find a $\theta$ such that the performance of the policy is maximized in this environment.
Let's examine two different ways of defining performance for the policy. The first one is simply the value (expected accumulated reward) of the policy in the initial state: $$ J_1(\theta)=V_{\pi_\theta}(s_0) $$
The second one is the average reward for the same policy. $$ J_2(\theta)=\mathbb E_{S\sim d_{\pi_\theta}, A\sim \pi_\theta(\cdot|S)}\big[R(S,A)\big] $$ where $d_{\pi_\theta}$ is the on-policy distribution for policy $\pi_\theta$.
If I remember Sutton's book correctly, then $\nabla_\theta J_1(\theta)\propto\nabla_\theta J_2(\theta)$. Is this really the case? In simpler words, does maximizing the average reward imply that we're also maximizing the expected return at the starting state? If so, how can one prove this?
Note
This question might seem similar to Average expected reward vs expected reward for start-state, but the similarity is superficial. I am aware that $V_{\pi_\theta}(s_0)\neq\mathbb E_{S\sim d_{\pi_\theta}}[V_{\pi_\theta}(S)]$, which seems to be the source of confusion for the asker. Instead of asking if such quantities are equal, I'm asking if maximizing one leads to maximization of the other.
