So if we have a Poisson distribution with a rate lambda we know that the prior is a gamma with alpha,beta. But suppose we didn't know that the prior was a gamma. How would we do the derivation please?
I think it's pretty straightforward and I'm just not seeing it.
Consider the posterior as prior$\times$likelihood and we want the posterior to be of the same form as the prior, so the likelihood must combine with the prior in a way that leaves the prior unchanged in form, apart from the values of the parameters.
In the case of the Poisson, we can do this one by inspection.
Recall that the likelihood is a function of the parameter, not the data (the data are fixed - we condition on them):
$\mathcal{L}(\lambda;\mathbf{x})\propto \lambda^{S} \exp(-n\lambda)\propto \lambda^{(S+1)-1} \exp(-n\lambda)$
(where $S$ is an obvious function of the data). We can immediately see that the likelihood is (up to scaling constants) of the form of a gamma density in $\lambda$.
This suggests to us that perhaps a gamma density could be a conjugate prior, since that also consists of a power of $\lambda$ and $\exp(-k\lambda)$, both of which will combine when we take the product:
$p(\theta|\mathbf{x})\propto \mathcal{L}(\lambda;\mathbf{x})\propto \lambda^{S} \exp(-n\lambda)\cdot \lambda^{\alpha-1}\exp(-\lambda \theta)$
$\hspace{1.4cm}\propto \lambda^{(S+\alpha)-1} e^{-\lambda(n+\theta)}$
Often this is all that it takes - looking at the likelihood as a function of the parameter and (in simple cases) recognizing the form of density where products of such densities are scaled densities of the same kind (as is the case with normal or gammas). In other cases, you will instead need to recognize that there's some density which the likelihood combines with in a product which will also be of the same form as the prior (even though the likelihood itself perhaps isn't in that form).
For another example, see this discussion of a conjugate prior for the Laplace (in the sense that the posterior is in the same form as the prior).
