Medical records are difficult, because they often contain publicly known information mixed in with private stuff: think of Gordon Brown’s eye operations. In a famous case, Harvard professor Latanya Sweeney managed to identify the medical record of the governor of Massachusetts from “anonymous” records released by the Veterans’ Administration.
For years, officials did not want to know. The idea that you could stop worrying about privacy if you just delete people’s names is altogether too seductive. John Major’s government built a database of hospital records with names removed, postcode and date of birth still there – so most patients are easy to identify. After the BMA objected, the Caldicott committee was set up to look into the problem and pointed out that more than 60 information flows in the NHS were illegal. The following Labour government at least did not deny the science, but went for legal fixes. The Data Protection Act 1998 was given a huge loophole: database operators can pretend data are anonymous if they can’t re-identify the records – even if others can.
See also the background at the LBT blog:
The government has been pushing for this since last year, having appointed medical datamining enthusiast Tim Kelsey as its “transparency tsar”. There have been two consultations on how records should be anonymised, and how effective it could be; you can read our responses here and here (see also FIPR blog here). Anonymisation has long been known to be harder than it looks (and the Royal Society recently issued a authoritative report which said so). But getting civil servants to listen to X when the Prime Minister has declared for Not-X is harder still!
Despite promises that the anonymity mechanisms would be open for public scrutiny, CPRD refused a Freedom of Information request to disclose them, apparently fearing that disclosure would damage security. Yet research papers written using CPRD data will surely have to disclose how the data were manipulated. So the security mechanisms will become known, and yet researchers will become careless.