That's an interesting question. My research group has been using the distribution you refer to for some years in our publicly available bioinformatics software. As far as I know, the distribution does not have a name and there is no literature on it. While the paper by Chandra et al (2012) cited by Aksakal is closely related, the distribution they consider seems to be restricted to integer values for $r$ and they don't seem to give an explicit expression for the pdf.
To give you some background, the NB distribution is very heavily used in genomic research to model gene expression data arising from RNA-seq and related technologies. The count data arises as the number of DNA or RNA sequence reads extracted from a biological sample that can be mapped to each gene. Typically there are tens of millions of reads from each biological sample that are mapped to about 25,000 genes. Alternatively one might have DNA samples from which reads are mapped to genomic windows. We and others have popularized an approach whereby NB glms are fitted to the sequence reads for each gene, and empirical Bayes methods are used to moderate the genewise dispersion estimators (dispersion $\phi=1/r$). This approach has been cited in tens of thousands of journal articles in the genomic literature, so you can get an idea of how much it gets used.
My group maintains the edgeR R sofware package. Some years ago we revised the whole package so that it works with fractional counts, using a continuous version of the NB pmf. We simply converted all the binomial coefficients in the NB pmf to ratios of gamma functions and used it as a (mixed) continuous pdf. The motivation for this was that sequence read counts can sometimes be fractional because of (1) ambiguous mapping of reads to the transcriptome or genome and/or (2) normalization of counts to correct for technical effects. So the counts are sometimes expected counts or estimated counts rather than observed counts. And of course the read counts can be exactly zero with positive probability. Our approach ensures that the inference results from our software are continuous in the counts, matching exactly with discrete NB results when the estimated counts happen to be integers.
As far as I know, there is no closed form for the normalizing constant in the pdf, nor are there closed forms for the mean or variance. When one considers that there is no closed form for the integral
$$\int_0^\infty \frac{1}{\Gamma(x)}dz$$
(the Fransen-Robinson constant) it is clear that there cannot be for the integral of the continuous NB pdf either.
However it seems to me that traditional mean and variance formulas for the NB should continue to be good approximations for the continuous NB. Moreover the normalizing constant should vary slowly with the parameters and therefore can be ignored as having negligible influence in the maximum likelihood calculations.
One can confirm these hypotheses by numerical integration. The NB distribution arises in bioinformatics as a gamma mixture of Poisson distributions (see the Wikipedia negative binomial article or McCarthy et al below).
The continuous NB distribution arises simply by replacing the Poisson distribution with its continuous analog with pdf
$$f(x;\lambda)=a(\lambda)\frac{e^{-\lambda}\lambda^x}{\Gamma(x+1)}$$
for $x\ge 0$ where $a(\lambda)$ is a normalizing constant to ensure the density integrates to 1.
Suppose for example that $\lambda=10$. The Poisson distribution has pmf equal the above pdf on the non-negative integers and, with $\lambda=10$, the Poisson mean and variance are equal to 10.
Numerical integration shows that $a(10)=1/0.999875$ and the mean and variance of the continuous distribution are equal to 10 to about 4 significant figures.
So the normalizing constant is virtually 1 and the mean and variance are almost exactly the same as for the discrete Poisson distribution.
The approximation is improved even more if we add a continuity correction, integrating from $-1/2$ to $\infty$ instead of from 0.
With the continuity correction, everything is correct (normalizing constant is 1 and moments agree with discrete Poisson) to about 6 figures.
In our edgeR package, we do not need to make any adjustment for the fact that there is mass at zero, because we always work with conditional log-likelihoods or with log-likelihood differences and any delta functions cancel out of the calculations. This is typical BTW for glms with mixed probability distributions. Alternatively, we could consider the distribution to have no mass at zero but to have support starting at -1/2 instead of at zero. Either theoretical perspective leads to the same calculations in practice.
Although we make active use of the continuous NB distribution, we haven't published anything on it explicitly. The articles cited below explain the NB approach to genomic data but don't discuss the continuous NB distribution explicitly.
In summary, I am not surprised that the article you are studying obtained reasonable results from a continualized version of the NB pdf, because that is our experience also. The key requirement is that we should be modelling the means and variances correctly and that will be fine provided the data, whether integer or not, exhibits the same form of quadratic mean-variance relationship that the NB distribution does.
References
Robinson, M., and Smyth, G. K. (2008). Small sample estimation of negative binomial dispersion, with applications to SAGE data. Biostatistics 9, 321-332.
Robinson, MD, and Smyth, GK (2007). Moderated statistical tests for assessing differences in tag abundance. Bioinformatics 23, 2881-2887.
McCarthy, DJ, Chen, Y, Smyth, GK (2012). Differential expression analysis of multifactor RNA-Seq experiments with respect to biological variation. Nucleic Acids Research 40, 4288-4297.
Chen, Y, Lun, ATL, and Smyth, GK (2014). Differential expression analysis of complex RNA-seq experiments using edgeR. In: Statistical Analysis of Next Generation Sequence Data, Somnath Datta and Daniel S Nettleton (eds), Springer, New York, pages 51--74. Preprint
Lun, ATL, Chen, Y, and Smyth, GK (2016). It's DE-licious: a recipe for differential expression analyses of RNA-seq experiments using quasi-likelihood methods in edgeR. Methods in Molecular Biology 1418, 391-416. Preprint
Chen Y, Lun ATL, and Smyth, GK (2016). From reads to genes to pathways: differential expression analysis of RNA-Seq experiments using Rsubread and the edgeR quasi-likelihood pipeline. F1000Research 5, 1438.
