10

I am measuring for the existence of response in cell signal measurements. What I did was first apply a smoothing algorithm (Hanning) to the time series of data, then detect peaks. What I get is this: time series of cell signal response

If I wanted to make the detection of the response a bit more objective than "yeah you see a raise in the continuous drop", what would be the best approach? Is it to get the distance of the peaks from a baseline determined by linear regression?

(I am a python coder and have almost no understanding of statistics)

Thank you

ttnphns
  • 51,648
  • 40
  • 253
  • 462
Radek
  • 201
  • 2
  • 5
  • 4
    I don't think there exist "the best approach". There are many ways to analyse or report peaks in time series. Your question gives no clue to understand what you might be after. Maybe you consult with articles in your field, for hints or benchmarks. – ttnphns Oct 25 '11 at 09:28
  • I do not know what information to provide to give you the clues. Basically you have a graph that has a downward trend (you get less response from a cell as time goes on) and somewhere in the middle you might see a raise in output. That is it. Do you think that it is basically up to me to subjectively say that say 10% increase in response = what I am after? – Radek Oct 25 '11 at 16:24
  • 1
    Assuming that you sometimes see the behavior as shown, and sometimes just continuous decrease (approximately), you'll have much better chances of getting a reasonable answer here if you replace you one large graph by 6-10 small ones, where one half has this increase and the other half doesn't. – AVB Apr 27 '12 at 04:14
  • Can it have more than one local maximum (bump)? – Emre Apr 27 '12 at 18:27
  • 1
    why don't you post your data and I will take a crack at this . The presumptive filtering that you and others have suggested have side effects. If you want an objective way of handling this I might be able to give you some pointers. But it all starts with the data not with presumption ! – IrishStat Apr 27 '12 at 19:50
  • You won't get any help from statistics here. – Aksakal Feb 02 '16 at 14:28

2 Answers2

2

So it sounds like from your October 25th comment that you are interested in algorithmically finding and characterizing two main features: the initial response decay followed by a cycle of increased response and subsequent decay. I assume that the data are observed at discrete time intervals.

Here is what I would try:

  1. Use a routine like numpy.ma.polyfit to fit, say, a 4th degree polynomial through your data. This should account for the initial drop followed by the rise/drop, but smooth out the numerous but minor fluctuations. Hopefully this degree of polynomial would be flexible enough to fit other, similar series well. The main goal I think would be to get a function that accounts for the major pattern you are looking for.
  2. Use Python routines for computing the derivative of the polynomial function fit to the data. Example routines are scipy.misc.derivative and numpy.diff. You are looking for the time values where the 1st derivative is zero, indicating a possible local min or max of the function. A second derivative test could be used to confirm which point corresponds to a min or max. Presumably you will have three such points if the graph you showed is representative. Note that the sage project could be very valuable here.

  3. At this point you'll have the time values associated with

    a. the start of the initial decay

    b. the start of the upswing

    c. the start of the second decay

You can then do what you want analytically to assess the changes.

It may be best to let the data speak for itself: across multiple series, when you apply this method, what is the typical size change at the upswing, when does it typically occur into the decay period, and how long does it last? And what does the distribution of this upswing look like in terms of where, how big, and how long? Knowing these statistics, you can better characterize a particular upswing as being within tolerance, with respect to where in time it occurs as well as it size and duration. The key from my understanding would be to easily identify where these changes are occurring. The rest of what I have described is straight-forward to calculate.

Josh Hemann
  • 3,384
  • 1
  • 21
  • 18
  • 4
    Polynomials won't work with these data unless you make the degree so large that they threaten to introduce spurious peaks. – whuber Nov 28 '11 at 19:37
  • Perhaps I should clarify my answer that he needs to still apply the Hanning function and then do the polynomial fit. The Hanning window may have to be changed to get a more smooth function. Or are you saying that a low degree polynomial won't work for the smoothed data? – Josh Hemann Nov 28 '11 at 19:58
  • 5
    A low degree polynomial definitely will not work, Josh. You need a local smoother--think of a kernel smooth or certain kinds of splines--and it needs *not* to be a polynomial, which has terrible properties. (Polynomials can introduce spurious, huge peaks between what otherwise appear to be well-behaved data series.) – whuber Nov 28 '11 at 20:27
  • @whuber, while I agree that fitting a polynomial globally would probably be a bad idea, the Taylor expansion of a function around a point $f(x)=f(x_0) + (x-x_0)f'(x_0) + \frac{(x-x_0)^2}{2!}f''(x_0) + ...$ is a low degree polynomial and thus the coefficients from a local quadratic fit should correspond to estimating the first derivative. Therefore, wouldn't local quadratic regression be the the most straight-forward, if not "best", way to go? – Sameer Jun 26 '12 at 04:09
  • 3
    Thank you for clarifying that point, @Sameer. I agree that a *local* low-degree polynomial fit could be effective and I did not mean to imply the opposite in my previous comment (which intended "polynomial" to be understood as a *global* fit). As to whether it is "best," though, I have to agree with a comment by ttnphns to the original question: it all depends. I would expect local quadratic fits with dense series of data to closely approximate Gaussian kernel smooths, which gives us one approach. Another sense of "best" is the BLUP of kriging, which can be spline-like. – whuber Jun 26 '12 at 13:45
1

Here are some ideas but I off the top my head that just may work...

Derivatives: If you take your array and subtract the elements from each other to get an array of one less points, but that's the first derivative. If you now smooth that and look for the sign change, that may detect your bump.

Moving averages: Perhaps using 2 lagged (exponential or windowed) moving averages might reveal the large bump while ignoring the small one. Basically, the width of the smaller window moving average must be greater than the width of of the bumps you want to ignore. The wider EMA must be wider but not too wide to detect the bump.

You look for when they cross and subtract the lag (window/2) and that's an estimate where your bump is. http://www.stockopedia.com/content/trading-the-golden-cross-does-it-really-work-69694/

Linear models: Do a series of linear models of sufficient width that are several little bumps wide, let's say 100 points. Now loop thru the data set generating linear regressions on the X variable. Just look at the coefficient of X and see where the big sign change happened. That is a big bump.

The above is just conjecture is on my part and there are probably better ways of doing it.

Chris
  • 1,151
  • 9
  • 26