ROC curve for an imperfect gold standard (March 12, 2008).

Someone asked me about how to use an ROC curve if you have more than two categories. Apparently the gold standard that the researchers were using was known to be imperfect, so they wanted an intermediate category (possible disease).

There's a lot of literature about less than perfect gold standards, and you should familiarize yourself with that first. Creating an intermediate category is not the best way to handle an imperfect gold standard. Often the best approach when there is an imperfect gold standard is to apply a second or third different (but still imperfect, of course) gold standard.

As far as I know, there is no way to adapt the ROC curve to more than two groups. You can, however, use a different model, such as ordinal logistic regression to see how well your diagnostic test predicts in the three categories.

If all of this seems too complicated, consider dropping the middle group or combining it with one of the other two groups. You already know that it is less than ideal, but it may be the only simple option available to you.

2008-07-14. Category: Diagnostic testing