Reporting of subgroup analyses: look for the test of interaction
Today’s BMJ looks at the ‘influence of study characteristics on reporting of subgroup analyses in randomised controlled trials.’
The objective was to investigate what industry funding does to the reporting of subgroup analyses. The ‘so what’ factor, of why is this important in the first place, probably comes to mind.
Subgroups are common, and in modern trial publication appear pretty much all the time. They are used to try and determine if certain baseline characteristics may affect the treatment outcome. For example, women with heart disease have very different outcomes compared to men and often this results in less aggressive treatment.
Yet, subgroups are open to all sorts of misuse: if they aren’t predefined they should be treated with extreme caution. A number of subgroups have subsequently been shown to be false: for example In 1988, the early breast triallists collaborative group showed in tamoxifen trials, there was a clear reduction in mortality only among women 50 or older. Yet, later reviews show benefits were irrespective of age.
Xin Sun and colleagues selected RCTs published in 118 core clinical journals in 2007, including 469 RCTs. What they found was that the high impact journals, non-surgical trials and larger sample size led to more reporting of subgroups. Of interest, and why this study is worth reading, is when the primary outcome was not significant, pharma trials were more likely to report subgroup analyses than non-pharma trials.
The one learning point that readers of trials should stick to from now on, and why this is important, is pharma trials used a test for interaction less for subgroups than non-pharma trials. To compare treatment effects in subgroups in a RCT, such as by sex, a test of interaction should be used. Even if it looks to the reader the two treatment effects look very different, and the P value looks very different, the test of interaction may not be significant.
So, if the interaction test isn't significant, there is no observable subgroup effect. And, if it isn't reported at all then you should ignore the result.
Value-based pricing and putting epidemiology into practice
I see epidemiology as the study of health and disease in populations. It can be observational (e.g. cohort studies) or experimental (e.g. clinical trials). Last week, I blogged about the increase in research funding and how more “bang-for-buck” may be expected from public- and private-funded research in hard economic times. Moreover, the best kind of health research is the kind that influences practice, or even better, translates into the policy sphere. However, shortcomings in design of research on the one hand, and clinical practice and policy on the other, mean that much epidemiology is not used or useful.
Clinical trials attempt to tell us whether a drug has a benefit in a particular clinical setting. Cost-effectiveness studies go further, allowing us to assess the relative cost of a drug versus the benefit gained (often by quality-adjusted life-years, QALYs). This epidemiologic research is vital because drugs make up a huge proportion of healthcare spending, for both the individual and the health system. Amazingly, there is no system in the NHS (or for that matter in the world) for directly correlating the amount of money spent or profit made by a certain drug with the health effects of that drug. The pharmaceutical industry has enjoyed the unique status among other manufacturing industries where sale of its products is not determined by their value. This aspect may contribute to the many conflicts of interest, scope for manipulation and lack of clinical data.
There have been many calls for changes in physician-pharma relationships and in the way pharma deals with its intellectual property rights, but few calls for change in the way pharma interacts with health systems. Well that is about to change. The UK Department for Health this week ended a 3 month consultation about value-based pricing (VBP). The government is hoping to roll out this scheme in 2014 and plans to incentivise innovation in drug research by rewarding pharma companies based on the “value” of their products.
A non-profit organisation, Incentives for Global Health, to which I am a medical advisor, is promoting a similar proposal on a global scale: the Health Impact Fund. This global fund aims to incentivise drug development and distribution based on the health impact of the drugs.
Both value-based pricing and the Health Impact Fund must face scrutiny, improvement and testing. However, not only do they both represent opportunities to change the way in which epidemiologic data is collected and used in this research area, they have the potential to radically improve the way in which health systems use limited resources on the expensive resources which are drugs. Have a look at my podcast on IDEAS-LAB about value-based pricing
The inadequate evidence base for monitoring in chronic diseases
We often undertake projects with folk who visit the department, and for this systematic review, Ivan Moschetti visiting from Italy and Daniel Brandt from Canada did a great job, undertaking the majority of the spade-work.
Our conclusion is pretty straightforward: ‘Many guidelines for cardiovascular disease do not report clearly what to monitor and what to do if a change is detected. If no evidence is available to support a specific monitoring schedule, this should be explicit in the guideline, with a description of the new research that would fill the gap’.
A lot of money is spent on monitoring in chronic diseases, and based on what we found a considerable amount of it is wasted. Read more
Increase in UK and global research funding-value for money?
Last week, the UK government unveiled plans to put £775 million of extra research funding into the NHS over the next 5 years. In addition, the newly appointed Chief Medical Officer, Professor Dame Sally Davies, is also Director General of Research and Development and Chief Scientific Adviser for the Department of Health and NHS, and so has a longstanding passion for raising the profile of health research.
If we look around the world, health research funding has also grown substantially, with US$ 160.3 billion spent in 2005, up from US$ 125.8 billion in 2003. In 2005, the UK was the third largest spender on health research at US$4.2 billion, after the US and Japan. The World Health Report 2012, boldly titled “No Health Without Research” will focus on the role of research and evidence in the improvement of global health. Over the last 5 years, the World Health Organization has been making a sustained effort at the international level to encourage research initiatives in public health, including:
- The 2005 World Health Assembly Resolution 58.34, which called upon WHO member states to “establish or strengthen mechanisms to transfer knowledge in support of…. evidence-based health-related policies.”
- The Bamako Call to Action, which urged governments to allocate at least 2% of budgets of ministries of health to research and to earmark at least 5% of funding for research, including support for knowledge translation practices.
There is no doubt that more funding and resources, both in the UK and globally, will lead to more research and hopefully more evidence for clinical practice and policy, but will this lead to improvement of health or health systems? The Global Forum for Health Research found that 97% of spending on health R&D continues to be by high-income countries, with only 3% by low- and middle-income countries. Unsurprisingly, most of this money goes towards generating products, processes and services tailored to needs of the health-care markets of the richer countries. In the UK as well, there are differences of opinion over whether money in research is well spent and whether universities manage health research better than the NHS, and about the amount of resource that is wasted in the current research infrastructure.
There has been a trend towards more funding for translational research in recent years (i.e. research that actually changes practice), both globally and in the UK. Particularly since large amounts of public money are invested in research, the scientific machine must try harder to make research relevant to the health of all sectors of the population, and make the most efficient use of this research. Worldwide, ‘global health financing’ has aimed to address this issue, by channelling external finance towards the health sector of low and middle income countries in order to meet the needs of predominantly poor population groups. The increase in private funding for global health now accounts for 25% of all development aid for health. However, inadequate monitoring and tracking of resources may have complicated and fragmented health systems. This is an important lesson for UK research funding.
The next paradigm shift is the change in the manner in which research is communicated and disseminated. The usefulness of research cannot only be measured by publications because of lack of access to journals and lack of access to research institutions. Blogging, Twitter and mobile technologies are examples of media already filling a void in research communication and will play a growing role. New metrics for evaluation of health research which assess its health and broader societal impact may not be far off. The bottom line is that more funding for health research is very welcome but it must be adequately allocated, monitored and tracked to ensure that the resources are properly used and not wasted.
Alzheimer's and brain scans: not needed yet
"We have some of the best scientists and facilities in the world and today's announcement will help ensure we continue to be at the cutting edge of life sciences."
An advanced computer programme compares a patient's brain scan with a database of 1,200 existing images of brains already affected by the disease. Scientists say early tests show the new technique is 85% accurate and can deliver results in just 24 hours.
In relation to diagnosis the word accurate keeps cropping up, and when it does we normally reveal the test is useless. If you want a precursor then see why Autism can’t be diagnosed with brain scans in the guardian last year. It's seems to be a modern feature of news stories that there seems to be little checking of the original data.
This is what Kings College have to say:
The 'Automated MRI' software automatically compares or benchmarks someone’s brain scan image against 1200 others, each showing varying stages of Alzheimer’s disease. This collection of images is thought to be the largest of its kind in the world.
The scan has been developed by scientists at the IoP, together with colleagues from the Karolinska Hospital in Stockholm.
The system is being 'field tested' over the next 12 months with patients attending SLaM memory services in Croydon, Lambeth and Southwark. The ‘field test’ will also provide a supply of research grade images, which has important implications for the development of the next generation of drugs for dementia and individualised treatments.
My assumptions for now, given after searching 35 news articles without a single reference to any study, is that 85% accurate refers to the proportion of people with established disease who have a positive brain scan. This figure, if it is the correct interpretation, bears no resemblance to the real question that needs to be asked? If I have a positive brain scan do I have Alzheimer’s?
I'm not sure how these news stories help us, but for now, I'm glad I can still spot nonsense when I see it. When I can't, it looks like I'll need the Brain scan.