What matters is what works

Ben Goldacre, who writes the Bad Science column in the Guardian, makes a very fair point. In medical research, randomised control trials have been commonplace for many years, whereas in social policy, they are much less common. As a result, policies get announced and implemented which sound like they kind of ought to help sort out a problem, but without there being any real evidence about whether they actually work or not.

There's been a really good example of the power of this kind of research from the USA just recently. Researchers were looking at the important question about how best to improve the health of poor children in rich countries. Previous research has established that there was a strong link between low income and poor health.

So what they investigated was 'the effectiveness of direct provision of additional monies to socially or economically disadvantaged families in improving children's health, well-being and educational attainment'. They used several pieces of research in different American states which looked at welfare policies which offered poor families extra money, usually in return for the families meeting particular conditions.

You can read the research paper here. The conclusion was very interesting:

"On the basis of current evidence we have not been able to establish that direct financial benefits delivered as an intervention are effective in redressing this balance in the short term. It is plausible that studies reviewed here did not offer a significant 'dose' (an interventions of larger value or longer duration). This is in essence a statement of "no evidence of effect" rather than of "evidence of no effect" viewed in the context of the monetary value of the interventions studied. While this review has not found significant benefits associated with low-value, strictly conditional welfare reform, the implications for practice is that increasing family income remains a promising intervention.

<...>

The gaps in the research evidence remain in the evaluation of unconditional payments of higher value, with high quality child outcome measures. For those studies completed, data collection for outcomes of children in experimental families should be ongoing." (my italics)

The political consensus, shared by all three main parties in the UK and advisers such as David Freud, is that more 'low-value, strictly conditional welfare reform' is what is needed. It sounds kind of like it should work - giving families more money if they make sure their kids go to school, or if they go and get a job or things like that. But sometimes, things which sound kind of like they should work actually don't.

That's why these research findings are so important, because they suggest that our government is likely to spend billions of pounds on a set of policies when there is no statistically significant evidence that they will work. So what's the chance that instead, we learn from the evidence and try out a more promising line of research, as suggested by the experts - substantially higher payments for poor families with no strings attached. These should be on exactly the same terms, that it is subject to randomised control tests and if it works, it gets adopted, and if it doesn't we try something else. James Purnell, who is in charge of this area of policy, said that he was 'ideologically neutral' about welfare reform - here's a good chance to prove it.