I was asked by BMJ to review a paper that involved a qualitative data
analysis. These reviews are confidential, so I don't want to describe the
paper in any detail. It is worthwhile, however, to note some of the
standards that others have suggested for assessing the quality of a
qualitative data analysis.
This is important for a peer-reviewer like myself. I need to be able
to assess whether the authors have produced a result that is sufficiently
rigorous to merit publication.
Perhaps the best paper in my files on this topic is
- Qualitative research in health care. Assessing quality in
qualitative research. Mays N, Pope C. British Medical Journal 2000:
320(7226); 50-2.
[Medline]
[Full
text]
[PDF]
The authors address a debate in the research community: should
qualitative research be judged by the same standards as quantitative
research. In particular, do concepts like validity, generalizability, and
reliability apply in the same ways?
Those who would answer "no" to this question, the antirealists,
argue that qualitative research represents a distinctive paradigm
and as such it cannot and should not be judged by conventional measures
of validity, generalisability, and reliability. At its core, this
position rejects naive realism---a belief that there is a single,
unequivocal social reality or truth which is entirely independent of the
researcher and of the research process; instead there are multiple
perspectives of the world that are created and constructed in the
research process.
The authors do not use the term, but the perspective of antirealists
reminds me of post-modern philosophy. There are numerous post-modern
critiques of evidence-based medicine, and I would like to summarize some
of these articles in a separate web page.
On the opposite side of the fence are the relativists, those who would
offer a conditional "yes" to the above question. According to the
authors, relativists believe that
assessment criteria are feasible but that distinctive ones are
required to evaluate qualitative research have put forward a range of
different assessment schemes. In part, this is because the choice and
relative importance of different criteria of quality depend on the topic
and the purpose of the research.
The criteria used by relativists to assess the quality of a
qualitative analysis include the following:
- Degree to which substantive and formal theory is produced and
the degree of development of such theory,
- Novelty of the claims made from the theory,
- Consistency of the theoretical claims with the empirical data
collected,
- Credibility of the account to those studied and to readers,
- Extent to which the description of the culture of the setting
provides a basis for competent performance in the culture studied,
- Extent to which the findings are transferable to other settings,
- Reflexivity of the account---that is, the degree to which the
effects of the research strategies on the findings are assessed or the
amount of information about the research process that is provided to
readers.
I personally do not find this list very helpful.
The authors then summarize a perspective of a third group, the subtle
realists, who hold a position somewhere between antirealists and the
relativists. Like the antirealists, this group does recognize that
there are multiple perspectives of the world that are created and
constructed in the research process,
but does argue that
there is an underlying reality which can be studied
and that while research may not be able attain the "truth", it is
capable of representing the underlying reality.
The difference between attaining truth and representing an underlying
relative is a subtle difference, and it is unclear to me whether this in
an important difference. But the criteria that the authors propose for
this group are promising.
- Triangulation. I have seen this term used narrowly to represent a
series of research studies that alternate between qualitative and
quantitative research, with each quantitative study building on the
knowledge of the previous qualitative study and each qualitative study
building on the knowledge of the previous quantitative study. The
authors of this paper take a broader perspective. Triangulation is the
use of multiple methods of data collection or the use of multiple
sources of data.
- Respondent validation. If a qualitative researcher really made a
hash of things, the people best able to point this out would be the
participants whose interviews and comments served as the raw data for
the qualitative data analysis. So you can validate a qualitative
analysis by showing a summary of the qualitative study to the original
sample and recording their comments and opinions about the summary
using the same qualitative analysis tools.
- Transparency. If you provide clarity in the process by which
various themes have been identified and developed, your reader should
be able to make informed judgments about the extent to which your work
was influenced by personal biases.
- Reflexivity. This is a term that the authors use without a good
definition. One source defines it as an "awareness of the
researcher's contribution to the construction of meanings throughout
the research process, and an acknowledgment of the impossibility of
remaining 'outside of' one's subject matter while conducting research."
When you are documenting your qualitative research study, you should
therefore define your own prior assumptions and experience.
- Attention to negative data. The authors also define this as deviant
case analysis. If you develop a theme in a qualitative research study,
there will be some raw data that supports this theme and some that
undermines this theme. By giving extra attention to the non-supportive
data and trying to reconcile it with your results, you produce a
produce that does a better job of representing the totality of the data
rather than an isolated viewpoint.
- Fairness. Is there sufficient diversity in your sample to insure
that an isolated viewpoint does not predominate.
The authors do point out that all of these criteria have limitations,
but I believe it still represents a valuable and interesting list of
perspectives to consider as you are reviewing a qualitative research
study.
Other resources:
The last source is particularly relevant to me.
07/08/2008.
