Government announces 'results' of voice analysis trial - but what do they mean?

If you've picked up the paper, or seen the Technology site this morning, you'll have seen that I've written about the "voice risk analysis" system being used by 25 local councils to "detect" benefit fraud, and on which the Department of Work and Pensions has spent at least £1.5m.

The scientific basis for this system is hard to find. Two Swedish scientists who have investigated the underlying software suggest that its verifiability – that is, "is there a scientific basis for believing what this is telling you?" lies "at the astrology end of the [scientific] spectrum". Other tests by a different team of scientists suggest that on its own, the system's reliability – that is, "how often does it give the right answer?" – is about as good as flipping a coin (ie 50%, or chance).

When I was researching the story, I knew that Tony McNulty was due to give a written ministerial statement with the results of the first sets of trials, which ran from May 2007 to November 2007, and which he had previously said would be published. (I found it a bit hard to believe that it had taken 18 months to write up the results of a pilot study on which large amounts of money had been spent, and which was extended to another 18 councils in May 2008. Wouldn't you publish the evaluation first so that citizens could see whether they were getting any return on their taxes? Perhaps I haven't worked enough in government.)

Anyway, the results have now been published.

And pretty lousy reading they make. I think we need to call in the Bad Science police.

From the statement:

The Department is committed to ensuring that decisions to use or not use the technology are supported by evidence. The Department's evaluation is being conducted by departmental statisticians who are bound by a national code of practice.

That sounds good... Now, let's see.

In each trial customers provided information to support their claim over the telephone and staff used Voice Risk Analysis technology to help classify the call as high or low risk. A selection of customers from both groups was followed up with a face-to-face review that aimed to corroborate the operator's judgment by seeing whether this further investigation revealed changes that would have affected the benefit in payment.

There then follows a table which shows the comparison of those where the "low risk" calls - where the system had suggested there wasn't much "risk" the caller was lying - had a benefit change; and another where benefits were changed (it doesn't say whether up or down; my research indicated for some councils benefits paid went up) after a call judged "high risk".

The result: the "high risk" calls led to a larger percentage of changed benefits than the "low risk" ones.

This is a fantastic piece of Bad Science, and it's been presented in a written ministerial statement. I think Tony McNulty should be ashamed. Perhaps he is; perhaps that's why this was written, rather than spoken.

Why do I say that? Two reasons. First, the glaring error: there's no control group. We don't have a comparable group where the VRA system wasn't used, and where operators tried to decide by listening and using their human skills whether the caller was a "high risk" or "low risk".

It's like running a drugs trial without a placebo - just having people who are ill and who are well, giving your drug to both of them, and seeing which ones get well. Here's a clue: a number of the ill ones will get well. Does that mean the drug works? Without the placebo as a control, you can't tell.

This is why in the piece I wrote I asked whether any of the councils have tried a "blind test" - where you'd have a black box that would beep (as the VRA system does) to alert operators that it thought it had detected a "high risk" - except you'd just have a random system in there. The problem there is that ethically, you can't tell people you're using a system to monitor them if you know you've got something that doesn't. Of course, none of the councils had tried that.

Just to make things worse, Mr McNulty hasn't supplied the raw numbers, only percentages. A primary school child will tell you that you can't make comparisons between different datasets if they're all expressed in percentages.

That could have a serious effect. For instance, if (in the table) the councils of Lambeth and Wealden - which show the smallest difference between the "low risk" and "high risk" results - actually had by far the largest number of calls, then the differences shown by other councils would be overwhelmed in a meta-study which put them all together. You might therefore find that there's no significant difference between the two groups.

I'm astonished that the government's statisticians have allowed this incomplete report to be put out, and that nobody has suggested doing the simple test (interviewing benefit claimants who haven't been through the VRA system) that would allow a chi-squared test that would actually tell us definitively whether this system works.

I have asked the DWP for the full results, and whether (and if not, why not) any non-VRA followup interviews were carried out.

And if you think this is pointless - remember, it's DWP spending our money on a technology that remains unproven, and not even doing the proper tests to determine whether it works. There isn't even an estimate of savings. It's astonishing.

Today's best video