They Said What?

Home » mHealth

Category Archives: mHealth

Vivify Brings Incompetence to Life

The population health industry never ceases to delight us with its creativity.  Vendors come up with ways of demonstrating their incompetence that are so creative we are compelled to use screenshots to back up our observations.  Otherwise no one would believe us.

Consider Vivify.  They reported on a study of in-home post-discharge telemonitoring led by a:

vivifyprincipalinvestigatorlessname

Not being able to spell the name of his own occupation is the good news.  The bad news is, the “principle investigator” also can’t write, can’t do simple arithmetic, and – most importantly for someone who claims to be a “principle investigator” — can’t investigate.  (Those shortcomings aside, this is a very impressive study.  For instance, the font is among the most legible we’ve ever seen.)

The Writing

There is some redundancy in the writing, but, giving Vivify the benefit of the doubt, perhaps the extra verbiage reflects the principle investigator’s concern that someone might miss the nuances or subtleties in his exposition.  Examples:

  • Vivify’s home monitoring system is “simple and easy”;
  • The patient receives a “weight scale”;
  • They had an “ROI of $2.44 return for every dollar invested”, and…
  • “With appropriate connectivity, patients could engage in real-time interactive videoconferencing.”

Needless to say, these product attributes are very intriguing, so intriguing that you may want to learn more about the company. They are only too happy to oblige, making sure we catch yet another nuance:

vivify about us

The Arithmetic

The study claims the average patient’s cost declined $11,706, for a 2.44-to-1 ROI.  Doing the math, that means Vivify’s post-discharge in-home self-care tele-monitoring costs…let me just get my calculator out here…$4797/patient?   At that price, why rely on self-monitoring?  Why not just move a nurse in?

vivifyROIandsavings

(Note for the literal-minded:  the ROI language is slightly different here than the passage we quoted, which appears elsewhere in the case study.)

The Principle Investigation

In general, Vivify targets patients with “specific chronic illnesses,” including pneumonia.  (Vivify, I don’t know how to break this to you gently, but: pneumonia isn’t a chronic illness, specific or otherwise.  No one ever says: “I was diagnosed with chronic pneumonia a few years ago, but my doctor says we’re staying on top of it.”)

vivify penumonia

However, for this investigation, only CHF was targeted: a cohort of 44 recently discharged CHF patients with an average age of 66.  This raises the question: How did the principle investigator scrounge up a cohort of 44 discharged CHF patients with an average age of only 66?  More than half of CHF discharges are over 75.  It’s statistically impossible to randomly select 44 CHF discharges with an average age of 66.   And – isn’t this a lucky coincidence – the study claimed a large (65%!) reduction in readmission rates but readmission rates are already much lower for younger patients.   Once again, not a word of explanation.

Because Vivify’s apparent level of misunderstanding of basic arithmetic and study design boggled even our minds (which is difficult to do, given that we mostly blog about wellness), we decided to give them a chance to explain directly that we might have missed something. Further, because these explanations would have taken them 15 minutes if indeed we were missing something obvious, we offered them $1000 to answer them, money they decided to leave on the table.  (Anyone have questions for me? Send me $1000 and I will happily spend 15 minutes answering them.)

This email to Vivify is available upon request.

We don’t even know what the 65% reduction is compared to.  Usually – and call us sticklers for details here – when someone claims a 65% reduction in something vs. something else, they tell us what the “something else” is.  Are they saying 35% were readmitted?  Or 66-year-olds are readmitted 65% less than 75-year-olds?

Savings Claims

My freshman roommate was like the bad seed in the old Richie Rich comics.  Among other things, he would have a snifter of cognac before bed, whereas I had never tasted cognac and thought a “snifter” was for storing tobacco.  We didn’t get along and at one point I accused him of being decadent.

“Decadent, Al?  Let me tell you about decadent.  I spent last summer at a summer camp – everyone was there, Caroline Kennedy, everyone – where we played tennis on the Riviera for a month and then went skiing in the Alps.”  I had to admit that was indeed decadent.

“Al,” he replied.  “I haven’t even gotten to the decadent part yet.”

Likewise, we haven’t even gotten to the best example of arithmetic-gone-wild:  the savings claim.  Remember that $11,706 savings claim above?  Well, read that passage again–it turns out that represents a “90% decrease in the cost of care.”  Apparently, the patients cost $12,937 when they were in the hospital, but after they went home, they only cost $1231.  (We have no idea how that squares with the other finding, that the Vivify system itself costs $4797, based on the ROI of 2.44, or, as they put it, “an ROI of $2.44 return for every dollar invested.”.)

 

Quizzify 3

We can spell “guaranteed savings” and other words too.

The irony is that other vendors in this space really do save money and really do measure validly.  It’s one thing to make up outcomes in wellness. That’s a core part of the industry value proposition. But, unlike wellness vendors, tele-monitoring vendors other than Vivify typically know the basics: what they are doing, how to measure outcomes, how to save money–and how to spell.

Propeller Health’s Success Claims Just Don’t Fly

Propeller Health

Short Summary of Intervention:

“The Leading Mobile Platform for Respiratory Health Management,” according to the company website, Propeller Health helps patients manage their asthma by using an electronic sensor to help patients record their inhaler use and track their medications.

Links to Materials Being Reviewed

This abstract of these interim findings for a study financed by the California Healthcare Foundation, which is also an investor in Propeller Health, also links to others.

Other treatments include Forbes’ David Shaywitz Two Examples Highlight Challenges of Demonstrating the Benefit of Digital Health Technology/ and The Health Care Blog’s Meet Propeller Health, Digital Health’s Poster Child for Invalid Savings Reporting (Al Lewis and Vik Khanna)

Summary of key figures and outcomes:

  • 40 people studied for four months
  • No change in Asthma Control Test Scores
  • 25% less rescue drug use
  • 62% reduction in inpatient utilization
  • $688/asthmatic in savings from reduced ER and Inpatient usage (vs. $282 “improvement” for control group), before the costs of the equipment is added back in
  • 67% of asthmatics presenting in ER admitted to the hospital (vs. 20% for country as a whole)

Questions for Propeller Health:

Why shouldn’t a disinterested observer question the linkage between California Healthcare Foundation financing the study and also investing in your company?

ANS: Refused to answer

Doesn’t the large treatment effect violate Dr. John Ioannidis’ conclusion that large treatment effects are usually wrong, especially because both groups received the same treatment?

ANS: Refused to answer

Wouldn’t one expect that such large improvements in avoided hospitalizations be caused by equally large improvements in the official Asthma Control Test Scores, instead of no improvement in those test scores?

ANS: Refused to answer

Should the Asthma Control Test Score algorithm be revisited due to its failure to predict this massive improvement?

ANS: Refused to answer

In disease management, typically a reduction in inpatient utilization is achieved by an improvement in adherence generating more drug use.   How were you able to achieve the opposite?

ANS: Refused to answer

Why did you write “Patients on Propeller save more than $600/year” on your website when the net reduction in spending (vs. the control group) was only $406?

measurably better respiratory health

ANS: Refused to answer

How do you reconcile your claim of $688 in savings per asthmatic in ER and inpatient utilization with the fact that commercial health plans only spend about $400 per asthmatic on asthma-coded ER and inpatient utilization?

ANS: Refused to answer

Your principal investor calls the $206 reduction in costs for the control group an “improvement” in costs. How could it be called an improvement when there was no intervention in the control group and therefore any change is due to regression to the mean and the Hawthorne effect? Wouldn’t it be more ethical to acknowledge that?

ANS: Refused to answer

Why did you decide to overlook the 67% inpatient admission rate of patients who present at the ER, and why is it several times the expected rate of 20% or less?

ANS: Refused to answer

Your own principal investigator, Rajan K. Merchant, is on record as saying that this data shouldn’t have been released by “Proppeller” because the study isn’t complete yet. “Anything prior is pure speculation,” he observed. How do you respond to your principal investigator’s opinion that you should not have released this data?

ANS: Refused to answer


Responses, addenda, apologies, retractions, changes etc.

Addendum: Email exchange

Email from Propeller:

The trial [is] still underway and we look forward to the principal investigators submitting the findings for peer review later this year.

Our Response:

Thank you.  So then another question is: why did you release this information now if indeed the trial is still underway and the principal investigator says the information shouldn’t have been released?   (Note:  It is OK to apologize if it was a mistake.  There is a spot on the website where we will note apologies.)

ANS: Refused to answer or apologize

%d bloggers like this: