complex survey in python

Question

I have been using the excellent R survey package for survival analysis of complex survey data. I have the necessity to migrate to python, and have found that the Python package lifelines gives the possibility to define sampling weights and clusters in the CoxPHFitter. For example, reusing pieces of codes from their tutorial, I would use:

import pandas as pd
from lifelines import CoxPHFitter

df = pd.DataFrame({
    'T': [5, 3, 9, 8, 7, 4, 4, 3, 2, 5, 6, 7],
    'E': [1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0],
    'weights': [1.1, 0.5, 2.0, 1.6, 1.2, 4.3, 1.4, 4.5, 3.0, 3.2, 0.4, 6.2],
    'month':  [1, 3, 9, 8, 7, 4, 4, 3, 2, 5, 6, 7],
    'age':    [4, 3, 9, 8, 7, 4, 4, 3, 2, 5, 6, 7],
    'id':     [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2]
})

cph = CoxPHFitter()
cph.fit(df, 'T', 'E', weights_col='weights', cluster_col='id', robust=True)
cph.print_summary()

to have a Cox Proportional Hazard model. Would this be equivalent of using svycoxph?

N.B: I'd add the tag lifelines but it does not exists and I do not have the minimum reputation (300) to create it. I'd appreciate if anybody would edit this question adding that tag.

score 4 · Answer 1 · answered Jun 20 '20 at 14:10

4

lifelines' author here.

I am not familiar with the inner workings of svycoxph, so I can't comment authoritatively on the similarities of it and CoxPHFitter. Looking at its docs page, there are some difference that lifelines doesn't handle, for example

accounts for the reduction in variance from stratified sampling and the increase in variance from having only a small number of clusters

I would treat lifelines' CoxPHFitter as most similar to coxph in the survival library, and compare what differences coxph has from svycoxph. Does that help?

answered Jun 20 '20 at 14:10

Cam.Davidson.Pilon

11,476
5
47
75

Thank you for your answer, and thanks for `lifelines`, great tool too. I will try the comparison you suggest. Do I interpret the documentation correctly in my question, beyond the variance issue? That is, would the point estimate be correct if in the `CoxPHFitter` I use the `id` to model the unique clusters and the `weights_col` are the sample weights? – giotto Jun 23 '20 at 20:07
Thanks for the kind words. To answer your question: I believe so, yes. – Cam.Davidson.Pilon Jun 23 '20 at 21:09

score 3 · Accepted Answer · answered Jun 28 '20 at 07:44

As @Cam.Davidson.Pilon says, there are some differences. But not a lot.

There's one big difference: svycoxph can account for stratified sampling, which coxph and (apparently) lifelines cannot. This can be important if you have good stratification, but ignoring stratification is conservative.

Another difference is that svycoxph can distinguish between sampling with and without replacement. That's probably not a big issue, since with a regression model you usually want inference about the data generating process rather than the finite population.

There will also be very minor differences of the $n/(n-1)$ sort, where the formulas aren't identical between clustered/longitudinal data analysis and survey sampling.

So, stratification is the potentially important issue. If you're analysing data from NHANES, stratification could matter. Most of the time it won't -- except that you'll get different answers from what survey statisticians get and you might have to explain why.

complex survey in python

2 Answers2