See Carl Heneghan in action in the CEBM's workshop videos. 
autism and brain scan test: the real predictive value
Last edited 11th August 2010
A brain scan that detects autism in adults could mean much more straightforward diagnosis of the condition, scientists say. Reported the BBC, Sky the Guardian and many more.
I had great difficulty getting hold of this paper, it wasn’t published online at the time of the press release. I managed to get a copy via Ben Goldacre at Bad Science and Evidence Matters who sent me the full text. Given this problem in getting the paper, it is highly likely no one who released the story has actually read the paper.
The news all report the headline ‘The researchers detected autism with over 90% accuracy, the Journal of Neuroscience reports.’
Sounds impressive, but this is one of the most obvious mistakes to make in interpreting a diagnostic test result. Never mind this is not the correct study type.
What has happened is the sensitivity has been taken for the positive predictive value, which is what you want to know: if I have a positive test do I have the disease?
Sensitivity: The proportion of people with disease who have a positive test.
Positive predictive value (+PV): The proportion of people with a positive test who have disease.
So, for a prevalence of 1% the actual positive predictive value is 4.5%. That is about 5 in every 100 with a positive test would have autism. Even at a prevalence of 2%, only 8.5% would be correctly identified.
Suddenly, not that great a test. This has to be one of the worst examples of misinterpreting diagnostic test results in the media I’ve ever seen.
Tags:
- Carl Heneghan's blog
- Login or register to post comments
Many problems in the paper
Yes, the paper has many problems and weaknesses, mostly related to data-mining and small samples. With 20 NTs in the study, it is not possible to say anything about the real value in the general population if ASDs are at 1%. Also, with no people with relatives with ASDs (but without being diagnosed themselves), we have no idea how it would deal with diagnostic issues either. That a test can separate autistics from NTs is not even close to enough for a real diagnosrtic test. Another issue is that media seems to see this as the first step in finding "the defect" behind ASDs, when the method used needs many differences to provide good results. If it holds up, it is more like big support for the neurodiversity concept rather than a step in the direction of finding the cause of ASDs.
I've also commented this in this blog entry: http://blog.rdos.net/?p=90
+PV
Who would suggest using this test as a screening tool for the general population with an autism prevalence of 1%?
This has to be one of the worst examples of misinterpreting diagnostic results on the internet I've seen.
William Osler IV
Written about this too
Carl,
Thanks very much for pointing me in the direction of the paper. I've written about it here: http://post.ly/rKrS
Ian
Great put-down
This is news the media want to not hear.
Michael.