Wednesday 4 January 2012

Beware of experts

Well, not so much "beware of experts" as "beware of definitions of experts".

It's fairly obvious that the way one recognises an "expert" in one field is not going to be the same that one recognises an "expert" in another field.

Obviously expertise and specialisation is a good thing. The problem is that identifying expertise in a field that you have little knowledge of is itself a tricky problem. So there are conventions as to what experts "look like" - some of those conventions are themselves directly rooted in privilege: they look like middle-aged middle-class white men - others are only indirectly so: academic experts tend to be found in research institutions because that's a major way that they become experts, but not everyone can get the chance to work at a research institution even if they have a strong aptitude in and interest for that particular field.

There are lots of problems with recognising expertise from the outside - a lot of my former posts on bad science reporting are caused by precisely that - and even on the inside experts will disagree with each other - but I think there's a particular issue in the way that academic expertise is identified and defined that matters.

This is: all science is basically statistical.

An expert in metallurgy is not an expert on every individual atom within a metal block (you need a different expert for the next block...) but on their statistical behaviour. An expert in economics is not tracking every penny separately but looking at the statistical movements of much larger sums of money over time. We recognise this.

An expert on psychology or health or other fields that consider the general behaviour of living creatures in particular conditions also only has statistical knowledge. We do not recognise this as much.

An atom of metal, or a penny, individually, may well not react in the overall way that a metallurgist or economist predicts that it will. This doesn't usually matter - the overall effect will still be what they predict. (And the individual variation between atoms or pennies is small)

When discussing people, however, a person not behaving in the way that the expert predicts is important. The expert's knowledge of the situation is statistical. They can tell that of 100 people, 90 will do a certain thing. The other 10 will not.

When attempts to predict are taken to the individual level they will regularly fail. The expert is an expert in general behaviour and situations. The individual is an expert in their own personal situation. So an individual's perception of their own situation - and what they should therefore do - should in general be treated as more reliable than the expert's1.

What actually happens is that the median behaviour the expert has identified - and their identification of such can be quite incorrect through their own privilege issues, of course, but even if all of that is somehow avoided - tends to become regarded as correct and deviation as incorrect. (And, further, stay regarded as correct even if the median behaviour later changes)

It also gets to a point about the difference between a claim that "X is a problem" and "X is an [adjective] problem" - which often gets intentionally ignored.

To make a claim that a particular thing affects a particular proportion of people, or that its prevalence has been changing over time or space, or that a particular counter-measure is in/effective one needs to do a very carefully designed study, probably more than once. This is really difficult to do well.

To make a claim that a particular thing occurs at all, seriously affects those it occurs to, and therefore should be countered ... you need to listen to a few people. This is also really difficult to do well, especially for people with the "anecdote is not evidence" attitude that works well in particular scientific fields and for statistical research and works very badly when exported from there.

Unfortunately, that sort of person is likely to fit right in to an academic environment where they can become a socially-recognised expert, because our expert recognition is insufficient.

We need a separation in the way that society recognises experts between "experts in statistical behaviour of people" and "experts in individual behaviour of people".

Footnote

1 The obvious exception is when the expert - who may in this context not be a socially-recognisable expert - has knowledge highly material to the individual's situation which they know the individual does not have. Of course, in most cases, not giving this information to the individual is intentionally forcing a mistake on their part.