The Surgical Checklist – twitter journal club

Quite excited about Twitter journal club, which is 8pm on Sunday @twitjournalclub

The paper for TODAY(!) is “A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population“. At the time it was published ,in 2009 in the NEJM, I had concerns about it, here. A few other people did too, but criticisms seemed to sink away and the Surgical Checklist is now  a la mode  in most UK hospitals.

I think the evidence is far slimmer and more contentious that the paper concluded, or indeed most medical managers who have implemented it. The press release was victorious in tone, here;

“A collection of hospitals in eight cities around the globe has successfully demonstrated that the use of a simple surgical checklist during major operations can lower the incidence of deaths and complications by more than one third….

The rate of major complications in the study operating rooms fell from 11% in the baseline period to 7% after introduction of the checklist, a reduction of more than one third. Even more dramatically, inpatient deaths following major operations fell by more than 40 percent (from 1.5% to 0.8%) with implementation of the checklist.

“The results are startling,” said Gawande, senior author of the NEJM article. “They indicate that gaps in teamwork and safety practices in surgery are substantial in countries both rich and poor. With the annual global volume of surgery now exceeding even the volume of childbirth, the use of the WHO checklist could reduce deaths and disabilities by millions. There should be no time wasted in introducing these checklists to help surgical teams do their best work to save lives.” …

IHI President and CEO Donald Berwick said: “I cannot recall a clinical care innovation in the past 30 years that has shown results of the magnitude demonstrated by the surgical checklist. This is a change ready right now for adoption by every hospital that performs surgery.”

The study took eight hospitals, one each in Toronto, Canada; New Delhi, India; Amman, Jordan; Auckland, New Zealand; Manila, Philippines; Ifakara, Tanzania; London, England; and Seattle, WA. They recorded a before and after effect with the introduction of a ‘surgical checklist’. Right here there is a problem; we don’t know why these hospitals were selected except that they wanted to become pilot sites within the WHO Safe Surgery Saves Lives program. Right away this introduces bias; and such a range of hospitals means that it will be difficult to fairly compare them. It also means that unequal safety check systems will be present to start off with; different types of operation on different types of people with different risks and expertise is added into the mix.

Then we look at the results, here; focussing on St Mary’s Hospital, the only UK hospital included.

We have 525 patients before the checklist starts, 585 after. They appear to show a reduction of surgical site infection from 9.5% to 5.8%. They also show a rise in pneumonia rates from 1.0 to 1.7%, and a reduction in death rates from 2.1 to 1.7%. This is an absolute difference of  11.025 to 9.9 deaths; probably, with rounding up, a difference of one death between the two groups of 525 and 585 respectively. I don’t think this is a big enough difference in real terms to give us confidence that checklists work; we are talking about 1 death in over 500 operations. The group is not large enough to be confident about rare outcomes. This result is well within the bounds of normal chance variation.

This is before we’ve got onto the Hawthorne effect, which is observer bias; people do better because they are being observed. More on that here. This is one of the reasons why I think this trial would have been better with a control hospital, also being monitored but without the checklist, in the same vicinity as the experimental hospital. Participants could have been properly blinded as to the purposes of the trial. In any instance, it wasn’t accounted for, and could have been the entire reason for any improvements that were made. I don’t think it’s possible to fairly include all the disparate hospitals together to reach statistical significance.

I don’t think the conclusions this paper reaches – that surgical checklists save lives and should be immediately vital – are served out by the  statistics. Of course we need safety, and of course we should promote it. But this probably means more than checklists – staffing levels, intensive care beds, and bed overcrowding for starters,

My concerns is that managers will look in the UK towards checklists as succour when it’s staffing and bed numbers that require more attention.

6 Responses to “The Surgical Checklist – twitter journal club”

  1. Martin Budden July 5, 2011 at 6:12 am #

    Related to this is the question: how much evidence is required before a new procedure, treatment or drug is introduced? If the evidence bar is set too high then there will be effective treatments whose introduction is blocked or delayed. If the evidence bar is too low then ineffective treatments will be introduced.

    My view is that the evidence bar is variable. Expensive treatments or procedures, or those with bad side effects need a high evidence bar. Cheap treatments or procedures with few or no side effects require a lower evidence bar.

    Checklists are cheap and have few side effects. They are widely used in safety critical areas outside medicine, pre-flight checklists used by pilots being the most obvious example (and probably the inspiration for medical checklists). There is even an argument that their efficacy outside medicine is sufficient evidence for their adoption within medicine.

    Your concern that checklists will divert attention from fixing other more important problems is of course warranted, but that is a fundamental problem of human nature – namely latching onto an easy fix to the exclusion of more important and difficult fixes. Your concern is not an argument against checklists themselves.

  2. Becca July 6, 2011 at 1:11 am #

    They may not be costly in terms of equipment or supplies, but how long do they take to perform? How much attention and time do they need from already overstretched staff? What else is being neglected in order to carry them out? These days you can be pretty sure that there is no slack in the system – if you give staff something extra to do, they’ll have to skip something they are currently doing to accommodate it.

  3. Margaret McCartney
    margaretmccartney July 6, 2011 at 9:18 pm #

    you’re right, Martin – on checklists, if there is evidence, we should use it, and I agree about how much /how risky an intervention is. As a paper I think this has major flaws, though, and I know surgeons in different areas of country complaining that the checklist has now reached monumental proportions of apparant importance and is audited constantly, and other important things simply aren’t. I think the evidence for this intervention in the UK is weak. The evidence for overcrowding resulting in increased m+m is stronger.
    Becca is right – pint and quarts. What gives up for this? It’s the handwashing argument; where does the 2minutes per patient per 40 patients a day come from? what don’t we do instead?
    If we took every weak paper up on it’s suggestions we would end up doing hardly any medicine. I am currently trying to find out how many forms nurses fill in when patients are admitted to hospital. If they are filling in 15 different risk assessments, they are not spending as much time with the patient – overall, where is the bigger harm?

  4. Martin Budden July 7, 2011 at 5:54 am #

    You comment:
    “the checklist has now reached monumental proportions of apparent importance and is audited constantly”
    “If they are filling in 15 different risk assessments, they are not spending as much time with the patient – overall, where is the bigger harm?”

    There are two complementary means to achieving quality: “rule based” and “culture based” (by “culture based” I mean where, within an organization, there is a culture of high quality – people pride themselves on the quality of their work, and producing quality work gains the respect of ones peers). Achieving quality in an organization requires both, but, in my view, the very highest quality is achieved when it is based primarily on culture.

    Unfortunately in many organizations quality control is often the responsibility of non-practitioners, and non-practitioners, often not having a full understanding of the subject, tend to focus on rule-based quality. What’s more, even within the rule base, they often put emphasis on the wrong rules – those that have high visibility and are easy to check, rather than those that actually have the most impact on quality. This is understandable – how can non-practitioners have any feel for which rules have the most impact? I see this rule-based myopia in my own field (computer software) all the time. It also seems to be widespread in medicine.

    On the checklists themselves: I’ve just had a look at the WHO Surgical Safety Checklist. I was shocked. The checklist seems so vague as to be useless. Some of the items seem to have the utility of the “how to fasten your seatbelt” instructions of the pre-flight safety demo given on airline flights. One can also imagine they are there for the same reason – something non-controversial that could be agreed on. I was at the very least expecting operation-specific checklists. Compare the WHO Surgical Safety Checklist ( ) with the checklist for a light aircraft ( ). Having said that, there are probably some useful checklists behind the scenes. For example, whoever preps the operating theatre must presumably have some sort of checklist of what equipment, drugs etc are required for the operation.

    “If we took every weak paper up on it’s suggestions we would end up doing hardly any medicine.” I was never suggesting this. In my view a single paper is not evidence at all, no matter how good that paper is. A paper needs to be followed up and the results independently confirmed before it can be regarded as sufficient evidence to even consider making change to established practice. And then, as I said before, the level of evidence needs to be weighed against the risk of the change.

  5. Margaret McCartney
    margaretmccartney July 11, 2011 at 8:23 pm #

    yes agree Martin – hospitals have always had some kind of check in the UK – every operation I’ve ever attended has, and the theatre nurse was always in charge of everything and you couldn’t start without her approval. that’s another reason why I think the study was weak; mixing of standards to start off with.
    thank you for the aircraft checklist too!

  6. Anne Marie Cunningham July 11, 2011 at 8:47 pm #

    Hello Martin,
    I hope you saw the discussion. I agree that the paper doesn’t say very much about the checklist and it is non-specific. But it seems that what was really being evaluated was an attempt at culture change through getting people together to talk about how the checklist would be implemented.
    Gawande tweeted a few good links that night.
    See here
    Both Gawande and Woodward were clear that the checklist should not be treated as just a checklist.