Stats
Can I run a quantitative analysis on this data? (June 17, 2008).
I get lots of questions about how small a sample size can be before you
can't perform a quantitative analysis and instead are forced to summarize the
data in a qualitative fashion. The most recent question involved looking at
infants with feeding disorders. There were 29 of these infants, but a subgroup
of 5 had disorders so severe that they still required a feeding tube at 3
years of age. The researcher wanted to compare this group of 5 to the
remaining 24.
There's a fuzzy border between the sample sizes that are so small that you
describe them qualitatively and the sample sizes that are just large enough to
allow a quantitative comparison. It's based largely on emotional reactions,
which is fine. The problem, of course, is that a sample size that one person
"feels" is too small might be more than enough for another person.
Although there's very little quantitative justification for this border, I
do talk about sample sizes that are so small that only an "all or nothing"
comparison will achieve statistical significance.
By "all or nothing" I mean that the worst case in the first group is
still higher than the best case in the second group. A total sample size of 5
and 5 (10 total) represents an "all or nothing" threshold. Since you have
exceeded that value in the second group, you probably have sufficient
justification for running an analysis. Keep in mind, of course, that it would
take a Godzilla sized difference to achieve statistical significance, but that
shouldn't stop you.
If you or your boss are uncomfortable, I certainly would not object if you
used a qualitative discussion. There's no right or wrong answer here.
2008-07-14. Category: Small sample size
issues
