Scientists' unethical use of media for propaganda
purposes
This web page is intended to draw the attention of scientists, the media,
and the public to a problem that, while being very familiar to some, is probably
unknown to the majority of visitors to this web page. Propagandising
of the kind described in the following bypasses the normal carefully
considered processes of science, and may well create
a distorted impression in the mind of the unsuspecting reader or
viewer.
The following account of a not untypical case concludes with recommendations
for the media.
1. A TV demonstration is fixed?
Many viewers of a recent Discovery Channel programme, previewed in a Guardian article, where the claims
of Natasha Demkina, a 17-year-old Russian girl who says she is able to "look
deep inside people's bodies, watch their organs at work and spot when things
are going wrong", were investigated, ended up with a strong impression that
the main test in the investigation had been deliberately set up with a view
to ensuring that she would fail it. The test involved her being given
a set of seven cards, with a medical condition indicated on each. Medical
subjects with these seven conditions (one of which was 'no condition'), each
bearing an identifying number, stood in a row and Natasha had to mark each
card with the number of the person whom she thought had the condition indicated
on the card. Despite the difficulties associated with the rigorous
and unfamiliar conditions imposed
by the experimenters, she identified four of the seven correctly. A
fairly straightforward calculation shows that the odds of getting 4 hits
or more out of 7 by chance are more than 50 to 1 against. Surely a case
for celebrating Natasha's success?
Well, no, actually. The experimental protocol, to which Natasha and
her agent had been asked to agree, rather curiously states:
"If Natasha correctly matches fewer than
5 target medical conditions, then the Test Proctor will declare that results
are more consistent with chance guessing and does not support any belief
in her claimed abilities."
Accordingly, it was announced that Natasha had 'failed the test'. In
the article about the programme in the Guardian, Richard Wiseman, one of
the investigators, emphasised this conclusion, declaring "a failure is a failure".
(added November 11th., 2004) The investigators'
own account is now available on the web: observe that the 50 to 1 statistic
does not feature anywhere in it. The fact that "everyone [had] agreed
to the written protocols" (including the above italicised condition) is given
as sufficient justification for asserting "[the] test, as preliminary as it
was, will likely close the chapter in this case". I think not: real
science does not work on a basis of getting someone to sign their agreement
to a long list of conditions, then later coming back saying "this is what
you signed; the challenge goes to us!".
2. Motivations I
On the face of it, it looks as if there was some kind of plot to discredit
the teenage claimed psychic by setting up the conditions to make it likely
that they could pass her off as a failure. Might the investigators have
simply made a computational error when they decided to place the cut off
point at 5 hits? Apparently not: Wiseman, when questioned about this,
appeared to know about the 50 to 1 statistic, but would not accept that the
cutoff point had been set wrongly. What then did he have to say about the
fact that Natasha achieved four hits? He admits that the result is
"interesting", but is then quoted as saying: "At best, she's done this a
lot and she has a real expertise at being able to look at people and make
reasonably accurate diagnoses." Perhaps realising that this statement
might not quite convince, he goes on to suggest that perhaps Natasha cheated,
a handy way to evade the issue.
A second suspect argument for Natasha having no special abilities (except
for her 'expertise at being able to look at people and make reasonably accurate
diagnoses') involved discrepancies in a preliminary test, but given the limited
extent to which these discrepancies were followed up it would be hard to
conclude just from these that Natasha was not actually picking up some problem
that the subjects had not written down on their card (having viewed the actual
programme, I now consider the argument based on these discrepancies to have
been grossly overblown).
In the words of Sven-Goran Eriksson, 'this is a nonsense'. A statistically
very significant result was obtained in the quantitative part of the investigation,
but the experimenters concealed the fact with their talk of 'failing the
test'. The investigation claims to be science, but fails almost every
test of good scientific practice. Does this matter for a TV programme?
I suggest that if a program pretends to be science then it does matter, and
its failure to conform to scientific norms is a serious matter.
3. Motivations II
If this line of enquiry leaves us uncertain, perhaps we should look elsewhere
to understand why the investigation took the form that it did. According
to the Guardian article, the experiment was designed by scientists working
for the Committee for the Scientific Investigation of Claims of the Paranormal
(CSICOP), an organisation of professional sceptics. In addition, the
leader of the investigation, Andrew Skolnick, is director
of the organisation CSMMH, the so-called Commission for Scientific Medicine
and Mental Health. This organisation is not an official body as one
might think from its title Commission, but rather a collection of people interested
in "the scientific examination of unproven alternative medicine and mental
health therapies", Their mission statement at http://www.csmmh.org/about.html
makes instructive reading, with references to 'aberrant remedies' and
'dubious research'. The situation now perhaps becomes a little clearer.
It would be consistent with the aims of members of such organisations
for the experiment to be designed to maximise the chances of being able to
discredit Natasha. Setting the threshold to 5, rather than the value
that would make more sense, namely 4, would help achieve this, as would the
scenario whereby it would be announced that Natasha had 'failed the test'.
Perhaps we need look no further to see why the threshold of 5 rather
than 4 was chosen?
4 Wiseman and the media
Wiseman's investigations are amazingly well covered by the media, which surely
cannot come about except by design. Other investigations follow a similar
pattern to the one described. Take the following report of one of his
mega-demonstrations. Here is an example:
No meeting of minds at telepathy trial
By David Derbyshire
(Daily Telegraph, 14 Dec. 2000)
THE world's biggest psychic experiment last week failed to
come up with a shred of evidence for telepathy.
But then, the true psychics would have known that anyway.
Over the course of 10 experiments, several hundred people failed to
project a set of images to volunteers in a sealed room several hundred feet
away. The volunteers should have got between two and three images right just
by random chance. In fact they scored one out of 10.
Derbyshire's jocular comment 'true psychics would have known that anyway'
is truer than he might have realised, since many professional parapsychologists
did predict in advance that the test would 'fail', just from their knowledge
of the test conditions.
We see the same features as in our earlier example: a highly publicised test
is carried out in a way that is unlikely to succeed, and the fact that it
does not succeed is trumpeted loudly as if it were a highly significant result:
a show dressed up to look like science.
5. Comments on the TV programme
I suspect that most viewers would have ended up being deeply sceptical as
to the way the 'test' was done, and wondered about the investigators' motives.
A comment made by Wiseman himself in the closing stages of the programme
would seem to sum up the investigators' mindset precisely:
"... it doesn't matter what [the experiment
achieves] in terms of testing. It's not about the results; this is about
belief".
It seems to me that the only correct conclusion to have come to in regard
to the test is that it was inconclusive; no viewer of any intelligence
would have felt that the fact that Natasha did not achieve 5 hits ('the score
that the experimenters had set'), under the conditions obtaining at the test,
meant much at all.
The three investigators saw fit to ignore the inconvenient statistics of
the outcome, and to talk instead as if the outcome of the test had refuted
Natasha's claims, which it clearly did not. Doing so was not only unscientific
but, under the circumstances, unethical.
6. Afterthoughts
(added November 28th., 2004) In correspondence, Andrew Skolnick has
expressed in strong words the view that the CSICOP/CSMMH experiment was good
science. while the critique on this web page was unscientific, assertions
that I found hard to take seriously. There is, however, one imaginable
perspective from which the sceptics' way of doing things could appear rational.
Let us examine it.
The scenario concerned is one where (a) one did not believe Natasha's claims
could possibly be true; (b) taking them to be true could have undesirable
consequences. If one thought that, then disproving the claims would
take a high priority. Suppose also (c) that there is a possibility that
it might appear as the outcome of an investigation that Natasha had the claimed
skill, when actually she did not (a situation that the sceptics assert is
the actuality), which could lead to the undesirable consequences presumed
in (b). In that case, it could appear rational to design the experiment
to minimise the probability of a false positive outcome as in (c).
This line of argument could have been the rationale underlying the choice
of 5 out of 7 as the cutoff point.
That line of thinking is problematic because, in minimising the probability
of a false positive in this way, the experimenters significantly enhanced
the probability of the alternative, equally undesirable, false negative, i.e.
declaring that Natasha appears to have no abilities when in fact she has
some. All that getting less than 5 successes really proved, under conditions
that were very unfavourable to the exercise of her skills (including, it
would appear, one of the subjects appearing to be laughing at her), was that
she could not get perfect results under arbitrary conditions (and even then,
there have been disputes as to whether her 'failures' were really failures;
for example, proper medical records have not been supplied to confirm that
the patients did not have the conditions that Natasha believed they had).
The very fact that there was no null hypothesis, other than the scientifically
suspect one that she could get a score of at least 5 under the unfamiliar
conditions of the experiment, makes the 'test' dubious from the scientific
point of view. And the experiment provided no justification at all for
Prof. Hyman to say as he did on the programme, "my hope for Natasha is that
she will grow up ... and give up this aspect of her life ... I don't think
it is good in the long run for any of us to be living an illusion".
Was the experiment CSICOP propaganda? Yes. Was the TV demonstration
'fixed'? In the sense discussed in this section, again yes.
(added Dec. 13th., 2004) Further analyses, by Julio Siqueira,
based in part on correspondence with the investigators, are now available:
7. Conclusions
Should one be suspicious of CSICOP's procedures? Let the reader decide.
The initial decision to consider a score of 4 hits out of 7 as 'failure'
when the probability of getting such a score is less than two per cent is
hardly in step with normal scientific practice. Having adopted it, and
used it to declare Natasha a failure ("she had the claim, we tested it, she
didn't pass the test"), the investigators moved on to what seems to have been
an automatic presumption of deception or self-deception ("people believe that
she can do it ... how come smart people can get to believe things that aren't
so?").
Manipulation of concepts such as 'failure', and the abuse of statistics,
are commonplace in the world of propaganda. Is that what is happening
here, or honest science?
The CSICOP organisation is not infrequently taken to have an authority that
it does not deserve. Such organisations are in reality pressure groups,
taking every chance they can get to press their beliefs in the media, often
in ways that have been characterised as misleading. Representatives
of the media need to be on their guard against this kind of thing. Some
recommendations directed toward this end follow:
- Keep your critical faculties active, and bear in mind the possibility
that what seems, on the face of it, to be a dispassionate scientific investigation
may in reality have an underlying unstated agenda. In the above case
for example, the choice of cut-off point, and the way the 'failure' was handled,
rather strongly suggests some such deliberate intent, but in other cases
it may be harder to judge.
- As a corollary, taking into account such uncertainty, it is well, in
writing reports, to avoid phrases such as 'scientists have shown': instead,
you can talk in terms of 'claims', which word carries less of a connotation
of scientists' pronouncements being absolute truth.
7. Links
Some sites and links giving critiques of these kinds of activities:
and here are links to the home pages of the two organisations discussed,
CSICOP and CSMMH.
The above report was produced by Brian Josephson
