2

Is it even possible to use a statistical test in this case? Specifically, I'm analysing cities in two different regions. The aim is to determine if there is a statistically significant difference between the amount of cities that have UK names in each region.

The data look like this:

                 region1    region2
UK_named_cities  553        3255
total_cities     9155       55327

The percentages of cities with UK names:

region 1: 6.07%
region 2: 5.88%

The difference is 0.19%.

My first thought was an $χ2$ test, which yielded this result:

Test Statistic (X-squared): 0.2840672425979418
p-value: 0.5940477093071286
Degrees of Freedom: 1

I'm not sure this is right though. I don't have expected and observed counts as such. And a p-value of 0.59 doesn't seem that intuitive when there is a difference (albeit a very small one). I can't see how a t-test or z-test or reporting CI's would apply either.

Fundamentally, I'm confused. Can I only report that there is indeed a difference, but not test it for significance?

RDJ
  • 485
  • 1
  • 4
  • 14
  • 1
    Closely related questions: [Does it make sense to compute confidence intervals and to test hypotheses when data from whole population is available?](http://stats.stackexchange.com/q/68886/22228) and [Resources for when population data is available](http://stats.stackexchange.com/questions/116483/resources-for-when-population-data-is-available) – Silverfish Sep 18 '16 at 11:15

2 Answers2

3

There are occasions where you can use frequentist inferential methods as you have used when you are envisioning a possible repetition of the experiment somewhere. But for this case there is not repetition to envision nor possible error in estimates so I would not do a significance test and just report the existing small difference in the two proportions. The difference you have will be judged by most to not be very relevant. You might do better with a covariate-adjusted analysis where the covariate is the year in which the city was founded. With binary logistic regression you could estimate the odds of a UK name, holding age constant. You can compare that to the raw (marginal) odds ratio with the data already in hand.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
1

At least to one prevalent interpretation of (frequentist) statistics, statistical inference is a process where a parameter of a population is estimated, based on a sample of the population. The test typically has two possible outcomes: no significance, in which case (usually) nothing can be stated about the population with confidence; or a significant effect, which, if the null hypothesis is that of no differences, means we can, with confidence/guaranteed long-run error rates, reject the null hypothesis.

The null hypothesis is (usually) this: "The value of the parameter in the population is zero." If you reject this hypothesis, that means you reject the possibility of the population parameter being zero. If, for example, you want to estimate if a new population you have encountered in an Amazon forrest has a male/female ratio like Europeans, or an IQ of 100, you could sample 100 people, and if your test (of the null hypotheses: male/female ratio = 50, IQ = 100) rejects, you could conclude the population diverges from that null hypothesis, even though you have not exhaustively measured the population. That is the purpose of hypothesis tests: making a statement about people you have not measured.
So it might seem a bit banal that all the test allows us is to reject a single number as a hypothesis; but from another perspective, it's magical - we say something about people we have never met, and using these procedures, we'll only incorrectly say our hypothesis about how these people are is false (when it actually is true) 5% of the time!

What a hypothesis test is not is an answer for the question: is the effect relevant? Relevance might be all kinds of things. For example, the IQ in the population might be 102 (stable over time). This would be inconsequential for nearly all purposes, but the null hypothesis of the population having a mean IQ of 100 would still be wrong. If a divergence from the null by 2 points is relevant can not be established by statistical inference.

A related example might be if you want to use a hypothesis test to infer e.g. the future of the population or the genetic material in the population. Then, the population your test would be about would be not the sum of people currently belonging to that society, however; the "population" would be a sample.

jona
  • 1,694
  • 12
  • 21