Late last year the Dr Foster company, which produces “unique consumer guides to health services,” released its annual Dr Foster Hospital Guide (BMJ 2011;343:d7791, doi:10.1136/bmj.d7791). Its press release, headlined “New report finds higher death rates at hospitals with fewer doctors at evenings and weekends,” stated, “The chances of survival are better in hospitals that have more senior doctors on site” (http://bit.ly/uGfG8e).

Roger Taylor, cofounder and director of Dr Foster, appeared on the BBC, saying, “It’s very worrying . . . What we’ve been able to do is to actually see where the problem is greatest, which hospitals have the biggest issue—which is highest death rates at weekends—and also identify that those are the hospitals where we don’t have the senior doctors in the hospitals at the weekends” (www.bbc.co.uk/news/health-15915874). He continued, “What we can really clearly see is that it’s about senior doctors, it’s about the more experienced staff, in the hospital, looking after patients, out of normal working hours.

“Almost every hospital does have a higher death rate at weekends, almost none it doesn’t occur at . . . We also list the hospitals that have fewer staff, there are some . . . Scarborough Hospital, George Eliot hospital, there’s a few hospitals where we do think there’s an issue that needs some pretty urgent attention.”

This was taken up by the press. “Hospitals with the fewest senior doctors on duty have the highest death rates, reflecting prior concerns about a lack of consultant cover, with junior doctors often left in charge,” reported theGuardian (http://bit.ly/t3vH0i) and, verbatim, the Independent (http://ind.pn/vwSnOC). The Daily Telegraph said, “Health experts said the figures . . . prove the link between the number of doctors on wards and the standard of care . . . Sir Richard Thompson, president of the Royal College of Physicians, said: ‘At last we have data that clearly links higher numbers of senior doctors in hospitals at the weekend with lower mortality rates’” (http://tgr.ph/vi9NwK).

But how reliable are the Dr Foster data? The company contacted 147 acute non-specialist trusts with a questionnaire about staffing, of which 135 provided usable data. The questionnaire asked the trusts what the number of medical staff scheduled to be on site and on call was on two midweek dates at 10 pm and two Sunday mornings at 11 am in early 2011. These staff were divided into nursing staff and junior and consultant medical staff categories. Hospital administrators were asked to fill in two columns, one each for either “on site” or “on call.”

Is this a reliable enough tool to generate data capable of drawing firm associations between mortality and staffing? No spot checks were done on wards at these times, which would have more reliably shown how many and what types of consultants were present. Just because consultants are on call does not mean that they are not present, and indeed it would be possible for a consultant working on call to be present for more hours in a hospital than a consultant contracted to work one session at a weekend. Also, weekends do not usually require presence of consultants at outpatient clinics or management or clinical meetings. Because there was no comparison with the consultant presence on wards during normal weekday hours, it is not possible to know whether the concentration of consultants working at weekends was different from the weekday pattern. What seems to have been an association may have been nothing of the sort.

Dr Foster says, “We work hard with the NHS to get the best possible data and although we recognise the data won’t always be perfect we do stand behind this data and feel, after the processes we went through to confirm and amend the data, that it is robust.” However, several hospitals were listed as having consultants on call but not on site on Sunday at 11 am. One consultant, who did not wish to be named, said that although Dr Foster had listed West Cumberland Hospital as having consultants on call but not scheduled to be on site at 11 am it would be “unheard of for less than four consultants” to be present then. Employees at other trusts have suggested that it is wrong to assume that on-call consultants would not be routinely present on weekend mornings. Other trusts have told me that the data for Dr Foster were gathered by assessing which consultants were paid to work a fixed session, such as orthopaedic surgeons’ or anaesthetists’ sessions, where other consultants would do the morning ward rounds and work as necessary after that, meaning that the Dr Foster data would not have represented staffing on the floor.

Dr Foster Intelligence, part of the Dr Foster group, is currently for sale (www.dh.gov.uk/en/MediaCentre/Pressreleases/DH_121537). It is part owned (48.75%) by the Department of Health for England, which obtained it in 2006 for £12m, a price, the House of Commons Committee of Public Accounts found, to be 33% to 53% higher than its valuation. It has also been criticised for large amounts of money paid to shareholders (http://bit.ly/sxE3F5).

The data that Dr Foster collects from the NHS comes to it free, and it sells the information on to NHS organisations: it is currently aligning itself to supply data to assist GP commissioning (http://bit.ly/uZtY5f). It is difficult to see why the data analysis that Dr Foster provides could not be done by epidemiologists and other researchers in the NHS.

It is also difficult not to notice the potential flaws in its data collection and the lack of substantive analysis or discussion of potential shortfalls in presentation of the data. Dr Foster presents its data as “robust” but does not tackle the potential flaws—this is not a fault of the data supplied but a fault of those analysing it.