Notes on the Whole System Demonstrator

Whole system demonstrator cluster RCT

The introduction says that the “monitoring of blood glucose” is an example of of useful monitoring that can be done by patients. In fact, Cochrane is pretty clear about how useful this is for people with type 2 diabetes not using insulin (the majority of people with diabetes), saying that “there is no evidence that self monitoring of blood glucose affects patient satisfaction,, general well being, or health related quality of life”. Similarly, with self monitoring of blood pressure, there is evidence that this can lead to small changes in blood pressure only, and there is no data on mortality or morbidity improvements for this intervention.

The WSD study examined patients with chronic obstructive airways disease, heart failure and diabetes. This is a wide range of conditions and the effect of telehealth on each these conditions were not analysed separately.

The randomisation was not done in the usual way but in terms of practices either assigned to the telehealth or the control group. Why?  “Initial discussions with sites indicated that individual randomisation of patients would probably not be acceptable to stakeholders. Therefore, we used a pragmatic approach to randomise general practice”. However there are large problems with this approach, as we have seen in breast screening trials for example. Cluster randomisation is beset with tendency to bias and I am not sure that the reasons given, for such a large and expensive trial, are good enough.

What was the intervention being tested? This to me is the biggest problem with this trial. Take note “Choices of telehealth devices and monitoring systems varied among the three trial sites, and there was no attempt to standardise these technologies across sites. We included a broad class of technologies, and the study was not designed or powered to examine differences between specific devices or monitoring systems.” In other words, the intervention varied hugely. This is crucial: what was the trial showing the effect of? The only constants were glucometer for diabetes (effect described above) weighing scales for heart failure (a commonly used device already, well outwith ‘telehealth’ trials) and an oximeter for COPD (which is of uncertain benefit in monitoring patients compared with usual care – this is important, because it might lead to either false reassurance or false alarms.)

Indeed, what is telehealth anyway? I don’t think that telephones or email communications between doctors and patients count as telehealth. The DoH had spent millions on the ‘black box’ monitoring devices well before the results of the WSD trial were published. These are “base units” which send data about health from the patients’ home. , either as questionnaires or clinical readings.  The outcomes analysed were for mortality and emergency admissions.

The graphs are here

How sure can we be that this apparent small benefit to intervention patients was real? The study was powered to detect a 17.5% change in admissions to hospital, however ” the magnitude of the group difference in admission proportion was relatively small (10.8%, 95% confidence interval 3.7% to 18.1%), and smaller than the size that the planned study design was able to detect (17.5%), raising questions about the clinical relevance of the results”. As for death rates “Intervention patients were significantly less likely to die within 12 months than controls. We also observed small differences in the mean number of emergency admissions per head between the intervention and control groups (crude rate 0.54 v0.68; difference 0.14). These changes were significant in the unadjusted comparisons and when we adjusted for a predictive risk score, but not when we adjusted for baseline characteristics”. In other words, the more the researchers tried to reduce the bias, the less sure the benefit appeared to be.

The study also found that there was an increased emergency admissions in the control group; “There could have been a propensity to select controls with a higher risk of short term admission and intervention patients with a lower risk.” It’s difficult to know, certainly we cannot exclude bias even with the case mix adjustments the researchers made.

So a comparable intervention wasn’t made, there are design issues in relation to the study design that increase the risk of bias, and the interventions being offered lack evidence themselves of benefit. The scale of savings is small. Neither has the WSD compared normal use of technology in practice (telephone, skype for remote access patients etc) against the expensive intervention being tested. For here is the conflict of interest. The DoH had bought these machines before the trial was completed, with publicity attached to it’s investment to “support the UK life sciences industry.” I’d personally like to see a bit more investment in the people who work as home carers, providing the unglamorous but essential help to wash, feed and assist people directly.




16 Responses to “Notes on the Whole System Demonstrator”

  1. Mandy Hall November 16, 2012 at 11:14 am #


    Interesting research. This whole area is a fast evolving world. The government appears to have the same approach to telehealth as the last administration, ohh shiny! Computers will solve the problems! Ohh it will make it all quicker! It will make us look modern and shiny and groovy! (Modernising the NHS by the 97 Blair Government was based on IT) .

    But – (and I am speaking here as a Information Management geek and ex PG student – and hopefully soon to be again) if they view organisations as sets of information and communication systems then if they change the IT and way people communicate then it has a direct effect on the organisation. This isn’t being monitored, evaluated for at all (as far as I can see).

    Small example – Lab links – previously GP’s commonly sat around in their common areas having lunch, coffee breaks etc, talking, chatting, swapping information about patients etc while also doing their paperwork. Lablinks put the results on screen – so now they have to sacrifice some of that important ‘social’ time to staying behind a screen to go through their results. But what of the informal information that still needs to be exchanged? Is that lost? Or formalised and perhaps losing some of its significance ?

  2. Margaret McCartney
    margaretmccartney November 16, 2012 at 11:19 am #

    you have your finger on it. 10 minutes of coffee time is when we share information, peer review our questions and concerns about clinical issues, discuss ongoing care, share views on what best plans are, speak to nurses and other staff about concerns or problems. This kind of stuff doesn’t get done when we are all in our rooms staring at computer screens. I don’t think this benefits patients – and I worry it increases stress in doctors that leads to less good care for patients overall.

  3. steve November 16, 2012 at 12:33 pm #

    Hi Margaret – as you might imagine telehealth is a bit of a hot topic in my neck of the woods. But like yourself I remain far from convinced. I also agree that using telephone consults or e.mailing/texting patients is something we have all done for years and is NOT telehealth – is just good-standard care.
    The issue of COPD monitoring with pulse oximetry is a good example. What benefits there are in doing home O2 monitoring are at best marginal. It gets even worse when we get calls from community staff because Mr Blogs’ oxygen staturation is 91% – “Yes, we know – he has COPD and that’s pretty good by his standards”.
    So the Government will spend millions on an unproven technique for improving the care of COPD patients in the community yet at the same time they won’t spend a farthing on something which HAS been shown to substantially improve the well-being of COPD patients in the community – pulmanory rehabilitation. So why do they do this? Well pulmanory rehab isn’t sexy; pulmanory rehab isn’t manufacturd by companies that put millions into political party coffers; and you don’t have to pay national insurance and superannuation for a machine. I think that’s about the sum of it or am I too cynical?

  4. Dr George Margelis November 17, 2012 at 1:14 am #

    Congratulations on a well thought and presented piece on the real story behind the WSD and the associated hype. I am a big fan of telehealth, and believe that if it is properly implemented by clinicians it can be very useful. However the discussion has been hijacked by non clinician groups, including the various companies who make the various devices and covered by what Steve Jobs was described as having, a “reality distortion field”.
    What is needed is solid research on effective models of care that utilise the technology to good effect. To do that clinicians need to be given scope to experiment, but the WSD did not allow that sort of innovation. Whilst initially such innovation may not be amenable to the standard rules of evidence based medicine, I think that if the clinical community if given the opportunity will do a much better job of finding solutions than the current IT industry is.

  5. jobo November 17, 2012 at 4:49 pm #

    Sad to see you peddling the Type 2 study as evidence that Type 2s don’t benefit from testing their BGs. You’ll note that in that study, there was zero talk of education for patients regarding how to actually make use of their blood sugar results. Of course blood testing is meaningless and may cause anxiety if patients aren’t taught how to act on their results and modify their behaviour accordingly.

    As a Type 1, I was never given information from doctors about how to make use of my blood test results. It was only by being in contact with other patients online that I learnt how to use my blood test results in order to modify my behaviour (diet, exercise and insulin dosing, correction dosing etc.). If things are that bad for T1s (my situation is not rare) then how bad are they for T2s who don’t get given adequate information about diet, carbs and their relation to blood test results.

    This study is constantly being used as a smokescreen by doctors in order to cut costs. Come join a patients forum and find out about how plenty of motivated and self-educated patients with Type 2 are using blood test strips to good effect in order to be part of the 5-6% (Hba1c) club. As a T1 I find it sad that these patients are literally having to beg their doctors and fight their PCTs for the tool that helps them manage their diabetes, with some pensioners forced to spend their state pension on blood test strips in order to maintain their own excellent self-care regimes.

    Doctors would rather not spend the time educating patients about how to use blood test strips – it’s far easier (and cheaper) to pat patients on the head and tell them not to worry their pretty little heads about self-management and self-testing. Meanwhile patients unknowingly descend into further complications and ill health.

  6. Margaret McCartney
    margaretmccartney November 17, 2012 at 10:37 pm #

    I don’t think for a moment that people with type 2 diabetes don’t get benefit from full knowledge as well as self control, that’s quite different from telehealth. The ability of patients to decide for themselves what they want to know and how has been keen to many local groups’ success with deciding for themselves what and how they want to know. None of this is telehealth. You have many good points about the lack of funding for people with long term conditions, but this doesn’t prove that telehealth works.

  7. Ellie Hothersall November 17, 2012 at 11:37 pm #

    While not involved with this study, I was involved with a separate telegraph study trying to get off the ground at the same kind of time, so was at some of the WSD dissemination events/conferences. I was at one in Manchester which I remember very clearly, as the morning sessions were all updates on recruitment,etc. Then the afternoon sessions were all workshops on how to get telehealth mainstream, long term funding after the trial published its results.

    This was one main reason for walking away from the project, as the outcomes were very obviously politically determined long before recruitment finished, and as such there wasn’t much hope of our smaller, much less well-gunned work being able to outshine it.

    Thanks very much for this useful summary, I will signpost people here.

  8. Harry Longman November 18, 2012 at 6:04 pm #

    I’ve been doing innovation all my life, and as an engineer that is a useful trait. But I’m even more committed to evidence than innovation, and what you describe is very disturbing as poor quality evidence is being used to drive multi £million spending (just the same for NHS111).

    Yet I discovered an innovation by GPs a couple of years ago which had such astonishing effects that having brought the inventors together, I formed Patient Access to make it widely known. Time after time we are repeating eye-popping case studies where the waiting time to see a GP falls off a cliff, from 5,6 or 7 days to same day. We can demonstrate measurable improvements in continuity, that elusive and fragile yet precious doctor-patient relationship. And it is all based on a whole system where the GP phones the patient as the first response to all requests. Anyone can do it, with no new technology. The change can be total in the space of four weeks. Patients have rapid access to their own GP.

    What really gave this legs was when I discovered that patients of the early inventor and pioneer practices (31 in a wide variety of contexts) were 20% less likely to attend A&E. That is all patients, not just the few with LTCs and telehealth devices.

    So excited was I by this that I took it to the DH with some of the GPs involved. They didn’t even give us a biscuit.

    Whether you call a GP phone call to a patient telehealth or not is immaterial, but the wider system effects could be game changing for the NHS. While there is growing interest and a trickle of practices making the change (3 now in Scotland, soon to be 4), there is currently no proper academic research into the method and its effects. This is urgent. Can you help?

  9. Chris Johnson November 20, 2012 at 10:33 am #

    I never much liked the word “telehealth” because it puts the focus on technology and not on the patient-professional relationship. I prefer “connected health”.

    Whatever you call it, the basic proposition is to bring more data forward and to help the patients understand the data themselves so they can take a more active part in preventing and treating conditions in closer collaboration with professionals. An expectation is that the future professional patient relationship is better informed with data and, yes this implies a different way of working.

    There seems to be sufficient positive indicators from this and other studies to justify continuing to explore new methods – reduction in hospital admissions and fewer deaths. I am optimistic that connected health will postpone the moment at which home carers have to intervene, although the timescales in the study were too short to prove this one way or the other. It would be curious when the availability of more data and broader understanding of what it means should lead to worse outcomes.


  10. steve November 20, 2012 at 4:32 pm #

    And what about the possibility that more data simply leads to more (possibly unnecessary) intervention?

  11. Chris Johnson November 21, 2012 at 11:53 am #

    Good point Steve. Margaret regularly warns of situations where data initiates unnecessary interventions and some people become anxious when they start monitoring their vital signs. Anxiety is one of the possible reasons advanced for the initial uptick in activity experienced in the study.

    Still I do not want to revisit the times before x-ray, scans, blood tests, or the multitude of monitors in a modern hospital. Connected health will enable many people to be treated from home instead of the hospital which many will prefer. One of the surprising results from US studies has been the major savings in transportation of patients from home to hospital for routine measurements and consultation.

    The cases chosen by NHS for telehealth experiments may not be the best for testing the new approach, although they are clearly “low-hanging fruit” from a cost-perspective and conditions for which home and portable monitors are available. Mental health, medication compliance, sleep apnea are promising areas too and there are many others where a closer data based relationship between professional and client should be productive.

    I think we should be careful about pouring scepticism on telehealth experiments. There are too many likely benefits.

  12. steve November 21, 2012 at 4:21 pm #

    Hi Chris – I agree with what you say but my concerns are that:-

    (1) too many people put too much emphasis on data and not enough on the individuals concerned. In order to do telehealth properly and effectively we will need to do a full assessment for every individual we would like to use telehealth data collection on. Such an assessment will take a long time – perhaps 1 to 2 hours per patient. The people who know these patients best (their GP, community nurse and/or carer) will all have to be involved and none will have the time to do it. I speak as a GP and can assure you that GP, in Scotland at least, is now at breaking point. There has been an enormous shift from secondary care into primary care which has not been backed up by a shift in resources and we simply cannot take on any more work. If I am to become involved in telehealth data collection I want (a) over-whelming evidence that it works in terms of reducing admissions AND improving health/quality of life; and (b) I want guarantees it will be adequately resourced indefinitely. So far I have seen little evidence of the former and no sign of the latter.

    (2) That brings me to my second concern – this is primarily seen by NHS managers as a way to save money – that is what is driving the process NOT improving health and well-being. I think that vision is mis-guided.

  13. Margaret McCartney
    margaretmccartney November 21, 2012 at 11:35 pm #

    hang on
    technology can be good, or bad, useful or not useful.
    Telehealth means a lot of different things.
    At the moment, the DOH want to use expensive boxes purchased before the trial was finished to re organise healthcare.
    I don’t think they have the evidence to support this, and I don’t think they have adequately examined the potential harms.
    It is this, fundamentally, that I object to – I don’t object – how could I? – to patients using technology appropriately and wish to use aspects of evidence based monitoring.
    Telehealth as described by the DOH is going to be used to reorganise care. I don’t think we have the evidence that says this is helpful, cost effective, or what the most vulnerable people want.
    For example, heart failure may well be managed by monitoring – but the evidence is so far for a set of scales and a phone line – not for an expensive widget.

  14. Chris Johnson November 22, 2012 at 11:00 am #

    There is a positive report from Cochrane (CD007228) on how telemonitoring and structured telephone support can improve things for CHF patients. Many studies show that different telehealth applications can make a difference. It makes sense to look at the area case by case and try implementations when there is some evidence that they will work – and there is a growing body of evidence in favour from around the world.

    @Steve. Not sure why you think a 1-2 hr assessment is going to be needed?

  15. steve November 22, 2012 at 5:08 pm #

    Hi Chris – because you need to be sure who is going to benefit; how are they going to benefit; what perameters you are going to monitor; how is the data being recorded and collated; what is going to happen with the data; who is going to respond to the data (and how); you need to liaise with all involved – patient/client, carer/family, community nurses, allied health professionals, GPs, social services (we’re all integrated up here dontyaknow); etc. I think it’s called informed consent!

  16. Chris Johnson November 28, 2012 at 5:05 pm #

    Thanks for the clarification, Steve.
    For me connected health is much more about patient centric behaviour – so who are you to judge what is appropriate?

    Doubtless the average GP or Consultant is smarter than the average patient, but when you have a chronic condition there is a big incentive to learn quickly albeit in a narrow area. I trust my GP as long as he or she is giving me useful advice and service, but when I feel some guy in say India is serving me better then I might go to him or her and buy my drugs on the internet.

    I think we need to wake up to the 21st century and not bury our heads in the sand. “Telehealth” goes a lot further than Skyping your GP and the potential negative consequences of failing to engage. They go further than Margaret is describing.

    The idea that I need a 2 hour consultation with my GP to get permission to send my blood pressure to someone who gives a damn is frankly absurd.