Quite excited about Twitter journal club, which is 8pm on Sunday @twitjournalclub

The paper for TODAY(!) is “A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population“. At the time it was published ,in 2009 in the NEJM, I had concerns about it, here. A few other people did too, but criticisms seemed to sink away and the Surgical Checklist is now  a la mode  in most UK hospitals.

I think the evidence is far slimmer and more contentious that the paper concluded, or indeed most medical managers who have implemented it. The press release was victorious in tone, here;

“A collection of hospitals in eight cities around the globe has successfully demonstrated that the use of a simple surgical checklist during major operations can lower the incidence of deaths and complications by more than one third….

The rate of major complications in the study operating rooms fell from 11% in the baseline period to 7% after introduction of the checklist, a reduction of more than one third. Even more dramatically, inpatient deaths following major operations fell by more than 40 percent (from 1.5% to 0.8%) with implementation of the checklist.

“The results are startling,” said Gawande, senior author of the NEJM article. “They indicate that gaps in teamwork and safety practices in surgery are substantial in countries both rich and poor. With the annual global volume of surgery now exceeding even the volume of childbirth, the use of the WHO checklist could reduce deaths and disabilities by millions. There should be no time wasted in introducing these checklists to help surgical teams do their best work to save lives.” …

IHI President and CEO Donald Berwick said: “I cannot recall a clinical care innovation in the past 30 years that has shown results of the magnitude demonstrated by the surgical checklist. This is a change ready right now for adoption by every hospital that performs surgery.”

The study took eight hospitals, one each in Toronto, Canada; New Delhi, India; Amman, Jordan; Auckland, New Zealand; Manila, Philippines; Ifakara, Tanzania; London, England; and Seattle, WA. They recorded a before and after effect with the introduction of a ‘surgical checklist’. Right here there is a problem; we don’t know why these hospitals were selected except that they wanted to become pilot sites within the WHO Safe Surgery Saves Lives program. Right away this introduces bias; and such a range of hospitals means that it will be difficult to fairly compare them. It also means that unequal safety check systems will be present to start off with; different types of operation on different types of people with different risks and expertise is added into the mix.

Then we look at the results, here; focussing on St Mary’s Hospital, the only UK hospital included.

We have 525 patients before the checklist starts, 585 after. They appear to show a reduction of surgical site infection from 9.5% to 5.8%. They also show a rise in pneumonia rates from 1.0 to 1.7%, and a reduction in death rates from 2.1 to 1.7%. This is an absolute difference of  11.025 to 9.9 deaths; probably, with rounding up, a difference of one death between the two groups of 525 and 585 respectively. I don’t think this is a big enough difference in real terms to give us confidence that checklists work; we are talking about 1 death in over 500 operations. The group is not large enough to be confident about rare outcomes. This result is well within the bounds of normal chance variation.

This is before we’ve got onto the Hawthorne effect, which is observer bias; people do better because they are being observed. More on that here. This is one of the reasons why I think this trial would have been better with a control hospital, also being monitored but without the checklist, in the same vicinity as the experimental hospital. Participants could have been properly blinded as to the purposes of the trial. In any instance, it wasn’t accounted for, and could have been the entire reason for any improvements that were made. I don’t think it’s possible to fairly include all the disparate hospitals together to reach statistical significance.

I don’t think the conclusions this paper reaches – that surgical checklists save lives and should be immediately vital – are served out by the  statistics. Of course we need safety, and of course we should promote it. But this probably means more than checklists – staffing levels, intensive care beds, and bed overcrowding for starters,

My concerns is that managers will look in the UK towards checklists as succour when it’s staffing and bed numbers that require more attention.