4

I am trying to find a correlation between a given data set containing redshifts and turnover frequencies (I have a list of 320 galaxies, and the redshift and turnover frequency (a turnover frequency is the frequency of the peak in the radio spectrum) is given for each (so have 320 data points)), but a third of the turnover frequency measurements are upper limits. I am looking at using the lifelines package (depends on pandas) to analyse this, but so far I have only seen tutorials that consider only one data set (eg. age at death).

So far I have only been able to include the uncensored measurements in fitting a trendline.

enter image description here

The red points are upper limits. As you can see, the correlation is extremely weak.

Can lifelines be used to find relationships between two variables like as described above?

If so, could I use the KaplanMeierEstimator function?

If not, does anyone have a suggestion for a survival analysis package for python that can analyse my data? (I know of Rpy2, but have had trouble installing)

Cam.Davidson.Pilon
  • 11,476
  • 5
  • 47
  • 75
Matt Majic
  • 163
  • 4
  • 3
    Hey Matt, can you describe your dataset better and what terms like "turnover frequency" are? – Cam.Davidson.Pilon Apr 21 '15 at 12:47
  • Why not exclude the censored points? The censoring is extreme. Most of your censored points have upper limits at the **low** end of your frequency data range. That makes them not very useful to improve your already very weak correlation. And you'd have to make a difficult analysis if you wish to include these points since the censoring is not the same for all points and also seems to be a function of red shift for a few groups at log TF 8, 8.5 and 9. – Sextus Empiricus Dec 10 '17 at 01:13
  • To solve this problem I would use a brute force computational method (custom made, an astronomer should like to do such thing not?). Try out many different linear relationships and determine the likelihood. Then you can incorporate many more effects than just the censored data due to upper limits (e.g. possibly your observations have more censoring, e.g. low TF galaxies at high red shift may be more difficult to observe?). – Sextus Empiricus Dec 10 '17 at 01:25

1 Answers1

2

If I understand you problem correctly, then no, there is not a "censored" correlation function in lifelines.

Edit: I've changed my thoughts on this: you can use the concordance index as a measure of correlation between a censored variable and an uncensored variable. Actually, to get a correlation value between -1 and 1, multiply the c-index by 2 and subtract 1 (also called the Sommer's D value).

Cam.Davidson.Pilon
  • 11,476
  • 5
  • 47
  • 75
  • Thanks. Do you happen to know of any software for this at all? – Matt Majic Apr 23 '15 at 00:24
  • Well ok actually you can use lifelines, but it's a hack on the definition of correlation. There is a measure called the concordance index which is used to measure survival prediction. Instead of comparing values, it compares ranks: if x_i < x_j in one dataset, is y_i < y_j in the other dataset? It deals gracefully with right-censored data as well. It's a value between 0 and 1 (1 being perfect, 0.5 is random, 0 is opposite); if you normalize the concordance index to be between -1 and 1, you get a sorta correlation (but please don't call it that). Here's a script to do it in lifelines: – Cam.Davidson.Pilon Apr 23 '15 at 19:32
  • https://gist.github.com/CamDavidsonPilon/73578eabd5b34b3f086d – Cam.Davidson.Pilon Apr 23 '15 at 19:32
  • Thanks, I managed to use this function and got ci = 0.605 – Matt Majic Apr 24 '15 at 12:50