1

I'd like to measure the duration of a function call. The function call has one parameter, n. If I were to graph an average of the function call's duration, with n on the x axis, I would also provide the standard error for each n.

The function call duration is, however, extremely short. It seems more appropriate to time, say, 100 calls. Is this approach completely orthogonal to the earlier use of the standard error? Does an average of the 100 calls now count as 1 measurement? Does it have a standard error?

whuber
  • 281,159
  • 54
  • 637
  • 1,101
user643722
  • 113
  • 3

1 Answers1

2

Does an average of the 100 calls now count as 1 measurement?

Basically you can measure anything and define the measurements to be the values of an (unknown) variable. In your case this variable X:="average duration across 100 functions calls given parameter n", so 100 calls count as one measurement.

Does it have a standard error?

Yes, given you have enough data (2 points), you can measure the standarddeviation/error of every variable. The question is whether the value is meaningful, which is only the case when the distribution is symmetric. If you are not sure about that you are better of considering the percentiles of the data or a robust scale measurement (see On univariate outlier tests (or: Dixon Q versus Grubbs))

Is this approach completely orthogonal to the earlier use of the standard error?

No. Let's assume a normal distribution $N(\mu,\sigma^2)$. When one repeatly draws a sample of 100 values from this distribution and calculates the average across this values, is this the same as estimating the mean from $N(\mu,\sigma^2)$ based on samples of size 100. The distribution of this estimation is t-distributed (approximately normal) with mean=$\mu$ and variance=$(\frac{\sigma}{\sqrt{n}})^2$

In summary: I'd check that the average duration of function calls is approximately normal (or symmetrically) distributed and just report the error. If this is not true, I'd report the percentiles instead. In the latter case (or in general) a boxplot provides a good visualization.

Community
  • 1
mlwida
  • 9,922
  • 2
  • 45
  • 74
  • Thankyou. I do though want to express the time in terms of one function call. I'm afraid I find your answer difficult to follow. – user643722 Dec 17 '11 at 14:49
  • @user643722 I do not understand. Then treat the duration of one function call as single measurement instead of the average duration across 100 calls ? Without stating what exactly yo do not understand, I cannot adjust my answer. But maybe I am on the wrong track and only someone else can help us out. – mlwida Dec 18 '11 at 19:47
  • Thanks again. I'll think your answers over. The small time I measure for one function call varies considerably. It seems sensible to measure the duration for 100, and average. But would I improve things by doing this n times, to obtain a standard error? Or would it be equally good to measure the duration of n*100 functions, and average? – user643722 Dec 18 '11 at 22:18
  • @user643722 The latter is more precise and hence recommended. – mlwida Dec 19 '11 at 08:56
  • Ok. But then I have no standard error, just the time for n*100 function calls. – user643722 Dec 19 '11 at 09:36
  • @user643722 I am in chat "ten fold" now. Please join if you want to continue the discussion. – mlwida Dec 19 '11 at 09:45
  • @user643722 since you did not show up: If you use n*100 functions calls, you have n*100 measurements and hence both an average and a standard-error. This is more precise than having n measurements (with each measurement=average across 100 calls) and calculating average and standard error across this n values. Frankly, I cannot follow your line of thought. – mlwida Dec 19 '11 at 10:42