I attended a webinar, "Should (Can?) Statistical Consultants, be Independent?" presented
by Janet Wittes.
Statisticians are like Humpty-Dumpty (When I use a word...), in that there are certain
words (validated, prespecified, intent-to-treat, and independent) have special meanings just
to statisticians. The talk focused on the last word, independent. Independent is not ignorant
or uninterested (although disinterested meaning lack of conflict of interest is good).
As statisticians are we an advocate, like a lobbyist, lawyer, or expert witness? An
advocate's job is to fashion the best result for the client. An advocate who identifies with
their client, they will use the word "we" in their conversation. An advocate should not be
independent. An advocate and a client share the same interest.
Contrast an advocate versus a consultant. Are you a hired gun? Is your fee contingent
based? Does your job depend on the answer? Noted that a "hired gun" is not a "prostitute".
The contrast listed below help to distinguish an advocate (first choice in the pair) from a
consultant (second choice in the pair).
-
Lobbyist vs. Advisor
-
Best result vs. "Truth"
-
Uses first person vs. uses second person
-
Fires client who lies vs. Fires client who ignores our advice
-
Not independent vs. Independent
A third role is "collaborator". A collaborator is a "team member" and uses the pronoun
"we" but is still interested in truth rather than the best result.
It is important in a consulting environment to feel valuable, but perhaps more
importantly, it is important we be perceived as important by our clients. It is critical that
we "sit at the table" meaning participate actively rather than passively. It is important to
talk and we need to talk in English, not in Statistics.
Dr. Wittes offered a two dimensional grid. To the left is a reviewer and to the right is a
team member. To the top is an advocate and to the bottom is a critic. A critic can actually
be valuable in the sense of serving as a devil's advocate. Independence is on the lower
portion of the grid (but not necessarily at the very bottom because an extreme critic is
actually an advocate of the opposite side).
Statisticians fall naturally into the role of critics, but it is hard not to let personal
feelings move us into the role of advocates. Our clients want us to be advocates, but they
grudgingly accept our role as critics.
I asked two questions. First, when does the "best result" not coincide with the "truth"?
Dr. Wittes distinguished between "not going beyond the truth" for an advocate versus
"producing a neutral result without a particular spin" for a consultant.
I also asked whether the distinction of an advocate versus a consultant changes when the
goal of the project is to develop new software tools as opposed to producing a particular
data analysis? Dr. Wittes agreed that new tools (and she added new methodologies) that
advocacy is more troublesome when the
Another person wanted to draw a distinction between consultation and collaboration by the
depth of the involvement, and Dr. Wittes agreed and elaborated on this point. A collaboration
needs to really get inside the problem and understand it thoroughly.
Another person distinguished between these terms based on whether you actually see the
data. To offer suggestions without seeing the data means you are a consultant. If you see the
data, you are involved to a greater extent to become a collaborator.
A fourth person worried that by translating our hard work into simple language, perhaps we
are oversimplifying and hurting ourselves by hiding many of the complexities of the problem
and making it seem that the work is easy enough for anyone to do. Dr. Wittes agreed and
offered an analogy of scientific approaches that use "cartoons" to explain mechanisms and
pathways.
A fifth person asked do you see a difference between consulting with companies versus
consulting with regulators. Dr. Wittes saw a huge difference. Regulators have to operate in a
different framework.
A sixth person asked if a consultant who has been paid as part of the grant be included or
excluded as an author on any research publications. Dr. Wittes pointed out that the degree of
collaboration and interaction will determine this. Sometimes our contributions are
undervalued and we are not given enough credit for our work and sometimes our contributions
are overvalued and a trivial effort is overstated. Sometimes researchers want our names on
publications for political reasons. If there is a misperception about whether you are "part
of the team" then you need to fix things.
Dr. Wittes then offered her ten rules (guidelines) for consulting.
1. Know thyself. Know what kind of analyses that you prefer, that you accept, that
you grudgingly accept, that you hate, and that you wouldn't be associated with. Be honest
with yourself--there have to be things that you really hate to do.
There is tension in your work when you are in a regulated environment. It is easy if you
and the regulators are on the same side. But don't hide behind a regulator and use them as an
excuse. When you disagree with the regulators, life is very hard. When you are more
conservative than the regulators, you are in an especially difficult position because a less
rigorous approach would still be accepted by the regulator. She offered the method "Last
Observation Carried Forward" is not ideal, but which is still acceptable in some regulatory
environments.
2. Learn the relevant nouns. In theory, we can do the data analysis using abstract
labels like x, y, and z. We never really know the scientific or technical area as well as the
experts. But we work more effectively and are more credible when we know the language. You
should not play a statistical "trust me" game.
3. Keep up to date. You need to attend meetings and courses. You need to read
journals and books. Examine your own behavior--are you only using the methods that you
learned in grad school or are you applying new methods?
4. Have some humility. You need a reasonable but not a degrading amount of
humility. It's okay to say "I don't know."
5. Work with people you like and respect. Or you can train them to be likeable. An
outsider can easily turn down work, because there will always be someone else who will take
up the job. This is harder for an insider, especially if you are the lone statistician. You
have to say yes to everyone, but you can use some balance. Spend more time and attention with
those people you like and with those collaborations that are most lucrative to you.
6. Don't let yourself be abused. There is hardly ever such a thing as a statistical
emergency. Again this is more difficult internally. Just be frank if you can't do
something--the worst case is they'll find someone else. If you always respond, then you will
get more and more abusive requests.
7. Train your clients to work in their interest. You need to teach your client how
to read (their is a lot of formal guidance available to scientists and technicians and they
need to be familiar with this information.) Teach them that accuracy and precision matter
(words and shades of meaning are very important to statisticians). Teach them that they can't
hide bad news (someday the truth will emerge and then it's a disaster). Keep your critical
eye fresh (know when you are being co-opted).
Dr. Wittes told a story about a situation where data indicated that there was a safety
problem with a certain drug. The people in the room minimized the problem and tried to argue
against the data. Dr. Wittes finally blurted out "I wouldn't want any of you to be my
doctor!" The conversation stopped for a while but when discussion resumed her comment was
ultimately ignored. At the break though, several of the other people (other non-doctors in
the room) spoke to her and thanked her for her honesty. The doctors needed to hear bad news
from an outsider.
8. Don't be all things to all people. Consulting is like a blind date. She offered
a case study where a Phase 3 trial was ambiguous and the client believes that the data meets
all the regulatory requirements, but the regulatory agency doesn't. You side with the agency.
Do you go to the FDA and have them believe something about you that isn't true? Do you
encourage the company not to submit and fail to serve their interests? A second case study
involves a paper that you are a co-author on where you disagree with the tone, they omit
something you want in, they give you too little time to respond. When do you remove your
name? There are no easy answers to these cases.
9. Listen to what the client wants and know what your client should want. Give them
no more or no less than what they want. But teach the client what he/she should want, and be
prepared to sever the relationship if the client is continually uninterested in important
issues. This is just intellectual honesty.
10. Avoid being the unwitting agent of someone else's destruction. You need to
understand who is hiring you and why. There may be some political undercurrents that you need
to be aware of. Are you being used as a tool or a pawn? Are you the first statistician
involved or are you being used to attack the work of a previous statistician? If you are
being hired as an outside consultant be sure to work closely with the internal statisticians.
In conclusion, Dr. Wittes asked what you give up when you become a consultant. You give up
the choice of problems that you work on and the intellectual ownership of problems. On the
other hand, you gain the chance to work on many things, and the ability to contribute widely.
For any given case, think about where you are on the reviewer/team member and critic/advocate
axes. Make sure that your client is comfortable with where you are, and do midcourse
corrections if there is a disparity in perceptions or if you are uncomfortable in your
current role.
One question was what you should do when you notice negative results or cautionary results
in a regulated environment. Do you point this out to the client? to the regulator? Dr. Wittes
said that you always disclose all information to your client. If the client does not want to
disclose this information to the regulator, then you need to educate them that the
statisticians at the regulatory agency are usually quite sharp and are likely to notice the
same things that you noticed. Another participant pointed out that if there are negative or
cautionary findings and you do not report them to the regulator, you will have a disaster on
your hands.
