Breast screening: case control vs RCT – the problems

The Daily Telegraph reported yesterday that “Breast cancer screening ‘works and we should move on” – “Women should undergo breast cancer screening because it halves the chance of them dying of the disease, according to a new study that claims to draw a line under the controversy”.

So simple? No!

Here’s the paper. It’s a case control study. So you look back at women who have died of breast cancer, and take a comparable group of women, but who didn’t die of breast cancer. Then you find out if one group went for breast screening more than the other. The researchers found that there was  “an average 49% reduction in breast cancer mortality for women who are screened.”

I don’t think this means that we have proven whether breast screening works or not. The best way to do that is through a randomised controlled trial. In this type of trial, a comparable group of women are sorted into two random groups, one group is given breast screening, the other are not, and the end results are compared.

I think this is better, mainly because it reduces bias best. Bias – skewing of your results – can happen when, for example, the healthiest people (and least likely to die of breast cancer) attend most for screening; it will look as though screening made them live longer, but in fact, they would have lived longer anyway.

Many case control studies will try and account for this by controlling mathematically for these types of bias. But it may not be easy to do this fairly.

We can see this with this example, which I have taken from Karsten Jorgensen’s work on this. The Malmo trial (in Sweden) contained over 21,000 women in each of the screening and non screening groups, aged over 45, and followed for a mean of 8.8 years. The data were assessed as a RCT, and the difference in mortality between the two groups was negligible. But the same data was then assessed as though it was a look-back, case control study, and found an odds ratio of 0.42.

The Malmö mammographic screening trial assessed as a randomised trial (Janzon and Andersson, 1991)

Invited (n=21 088) Controls (n=21 195)
Proportion 50% 50%
Breast cancer deaths 63 (31 were non- participants) 66
Mortality rate 0.299% 0.311%
Relative risk 0.96 (95% CI 0.68–1.35)


The Malmö trial assessed using a case-control design Janzon and Andersson, 1991)

Living controls (random sample) Women dead from breast cancer Total
Participation in screening
 Yes 229 36 265
 No 71 24 95
Total 300 360
Crude odds ratio 0.46
Adjusted (matching for age) odds ratio 0.42 (95% CI 0.22–0.78)
So in the case control design of the same Mamlo data, it seemed that you were much less likely to die of breast cancer if you had been screening. But we know when we analyse it as an RCT – as the trial was actually run –  that same large benefit does not appear.
I think this is very important because it demonstrates how we can be misled by supposed benefits from case control studies, but the RCT results are much less impressive in terms of a mortality difference.
This kind of problem isn’t new, but we seem to keep doing it. For example, a couple of years ago this study found that women over 80 who were diagnosed with early breast cancer were more likely to have had mammography. Given what we know about overdiagnosis (being diagnosed with a cancer at breast screening that was not going to interfere with your lifespan) or lead time bias (being diagnosed with a breast cancer earlier because of screening, but not in a way that would allow for better treatment) this wouldn’t be surprising.
In fact, the WHO have said  that we should not be using case control studies to assess the effect of screening; “observational studies based on individual screening history, no matter how well designed and conducted, should not be regarded as providing evidence of an effect of screening”.
We seem to be in a bit of a loop when it comes to breast cancer screening reporting. The bottom line is that RCTs are more reliable than case control studies, and it is either brave or daft to claim the difficulties with the evidence are resolved on lower quality data.

5 Responses to “Breast screening: case control vs RCT – the problems”

  1. Anne Marie Cunningham (@amcunningham) September 8, 2012 at 10:37 pm #

    Thanks Margaret. I still don’t understand what they did in this case-control analysis and I can’t access the original. There is definitely a typo in that table though as 36+24 =60 not 360.

    Are we agreed that the only factor is selection bias though? That was the main thing we were discussing yesterday.

  2. Margaret McCartney
    margaretmccartney September 8, 2012 at 10:54 pm #

    it’s total stringing along – so 300 + 36+ 24 = 360 – confusingly written

    yes, indeed, my fault – I was presuming that it was the same problem as the second study down – ie more mammography= more cancer – my fault for having not read the full paper first. sorry!

    yes, I agree it would be better if the full workings were placed out, and not behind book covers (you can search inside to find it – which I had to do as I can’t find the book) but the problem is that this has hardly been done outside Mamlo – I am not aware of any other RCTs that then went back and did a case control using the same end data – it would be possible to do (but not easy to blind the researchers)

  3. Anne Marie Cunningham (@amcunningham) September 9, 2012 at 12:50 am #

    Why would you need to blind the researchers? The outcome in the cases is death from breast cancer, which can be established from registries and the exposure is participation in screening. Is there much risk of classification error?

    Thanks again

  4. Margaret McCartney
    margaretmccartney September 9, 2012 at 1:49 pm #

    think more that researchers should be blinded to the result of the RCT, so that if using same data with case control method they don’t know what the other results were….

  5. Karsten Juhl Jørgensen September 12, 2012 at 7:34 am #

    Thank you, Margaret, for explaining this issue so clearly. A newpaper article rarely provide thorough explanations, so some additional detail is warranted.

    This is a difficult topic – how can a 50% reduction in the
    risk of dying from breast cancer be all down to bias? I think some
    additional background numbers may make it seem less odd.
    The authors found that 2 051 of 3 650 (56%) of controls had been screened, whereas the corresponding figure for cases was 146 of 427 (36%). The difference in attendance was thus 20 percentage points, which is much easier to understand
    could be due to self-selection bias. These numbers were the basis for the calculated “effect” of screening.

    There is a couple of additional peculiar issues in this paper.

    1. The opening sentence is a statement that the randomised trials showed a 25% reduction in breast cancer mortality. The reference provided is the 2002 WHO/IARC report on breast screening. The authors then go on to show that case-control studies have very, very consistently found a 50% effect, including their own new Australian analysis (49% for the meta-analysis and 52% for Australia, to be precise). They make no comment as to how the effect of breast screening in a real life setting is twice as large as that found in the rigorously controlles randomised trials. Differing attendance cannot be the answer, as there were no correlation between attendance rates in the trials and the estimated effect.
    The Canadian trials, which only included those who accepted an invitation to
    screening (essentially 100% attendance), could not demonstrate any effect.

    2. The case-control studies are so consistent in their estimate of effect
    that they vary by only a few percentage points if we exclude two outliers
    (I2 for heterogeneity was 0.0%!) . For comparison, the randomised trials
    varied between a 35% effect and no effect at all. Yet again, the authors
    make no remarks about this remarkable inconsistency between their findings
    and studies of much stronger design.

    What they demonstrate with their study is not a huge and consistent effect
    of breast screening. It is a huge and systematic error in the case-control
    design. However, we already knew about this systematic error, because we
    know from the randomised trials that when such studies are analysed using the
    case-control design, they can show an “impressive”
    58% effect, even when there is no effect at all, as demonstrated above by Margaret. A consistent 50% effect in such studies is then hardly impressive. This is why the very same 2002 IARC/WHO report that the authors of the new article start by quoting denounces case-control studies for the evaluation of the effect of breast screening, no matter how elaborate the design.