This model is possible with the brms R package which is an interface to R:
A slight modification of one of the examples from https://cran.r-project.org/web/packages/brms/vignettes/brms_distreg.html shows essentially what is involved in setting up and fitting the model
## load package
library('brms')
## load data
zinb <- read.csv("http://stats.idre.ucla.edu/stat/data/fish.csv")
## fit a model with constant zero inflation
fit_zinb1 <- brm(count ~ s(persons, k = 4) + s(child, k = 4) + camper,
data = zinb, family = zero_inflated_negbinomial(),
chains = 4, cores = 4,
control = list(adapt_delta = 0.999))
## plot the marginal effects
plot(marginal_effects(fit_zinb1))
## model summary
summary(fit_zinb1, WAIC = FALSE)
## fit a model where the zero-inflation is a constant plus a linear
## effect of number of children
fit_zinb2 <- brm(bf(count ~ s(persons, k = 4) + s(child, k= 4) + camper,
zi ~ s(child, k = 4)),
data = zinb, family = zero_inflated_negbinomial(),
chains = 4, cores = 4,
control = list(adapt_delta = 0.999))
## plot the marginal effects
plot(marginal_effects(fit_zinb2))
## model summary
summary(fit_zinb2, WAIC = FALSE)
There's more to this that shown above (you need to do some amount of model checking, look at posterior predictive checks etc) but it gives you an idea of what's involved in fitting a GAM --- you'll need to look at the help for brms to find the syntax for random effects.
The other alternative is to use the gamlss package, which has the ZINBI family for zero-inflated NegBin and as with brms you can model all of the parameters of the distribution with linear predictors.