They Said What?

Home » Uncategorized » It’s Time to Retire the Infamous “3.27-to-1 ROI” from the “Harvard Study” (Part 1)

It’s Time to Retire the Infamous “3.27-to-1 ROI” from the “Harvard Study” (Part 1)

Do you know whether heartburn pills are safe for long-term use?

Quizzify knows. Click to learn more.

“You know that laundry detergent that gets bloodstains out of t-shirts?  I’d say if you’re routinely coming home with bloodstains in your t-shirts, laundry probably isn’t your biggest problem.”

Recognize that line?  That was Jerry Seinfeld’s signature joke, the one that more than any other propelled his early fame.  Having used it for a decade, he retired it shortly after Seinfeld went on the air.

Now it’s time to retire the wellness industry’s signature joke: the so-called Harvard Study and its 3.27-to-1 ROI. Among other things, as we’ll see, a large chunk of the data in that study also predates Seinfeld, a show which recently celebrated its 25th anniversary.  You know the old line: “How can you tell a lawyer is lying?” Answer: “His lips move.”  Well, the way you can tell a wellness vendor is lying is that they still cite that study, knowing it to be completely invalid.  (If they actually think that 3.27-to-1 is valid, in a way that’s even worse.)

Some Background

The article, which today wouldn’t pass peer review by the normally cautious Health Affairs, was rushed into publication during the healthcare reform debate as a favor to David Cutler, a healthcare advisor to President Obama and an author on the study.  Along with the results from the Safeway wellness program — which turned out not to exist (and I’m referring not just to the results but literally to the program itself) — this meta-analysis provided cover for the 30%-to-50% clawback provision (for non-smokers and smokers respectively)  in what became ironically known as the Safeway Amendment to the Affordable Care Act.

In turn, this clawback was needed to secure the support of the Business Roundtable (BRT) for the Affordable Care Act. The BRT was and is obsessed with this provision. Not because they are especially concerned about employee health — the guy who ran their wellness lobbying also ran a casino company, exposing thousands of employees to second-hand smoke.  Rather, because this clawback provision creates billions in forfeitures that go back into the coffers of its member organizations.  Bravo Wellness accidentally spilled that secret.  The worse the program, the more an organization can save via forfeitures. For instance, Wellsteps saved millions for the Boise School District even though healthcare spending/person jumped–simply by creating a program so unappealing that employees preferred to obtain insurance through their spouses. (Either that or they outright lied–you make the call.)

How Prof. Baicker Invalidated Her Own Study

The title was definitive and the conclusion was amazingly precise. The title was: “Workplace Wellness Can Generate Savings.”  Not “might” or “possibly could on a good day” but “can.” And the conclusion itself left little doubt.  The ROI wasn’t “approximately 3-to-1” or even “approximately 3.3-to-1” but rather a statistically precise 3.27-to-1.  You don’t make such a definitive pronouncement without being ready to defend it.

Or maybe you do. Here’s how Professor Baicker’s story changed on various occasions once it became clear that her result was a major outlier, and that wellness loses money:

  1. It’s too early to tell whether these programs pay off, and employers should experiment on their employees;
  2. She’s not interested in wellness any more;
  3. People aren’t reading the paper right. They aren’t paying enough attention to the “nuances.” Shame on the readers!
  4. “There are few studies with reliable data on the costs and the benefits”

This may be unique in healthcare history: writing a paper and then almost immediately retracting its key finding, claiming you have no interest in the topic even as your fans deify you, and then blaming readers for believing the conclusion.

But the biggest head-scratcher is the 4th item.  Here is someone who just published a conclusion with two-significant-digit precision now saying (accurately, as we’ll see) that, oh, by the way, the studies she analyzed are mostly garbage.  You know the old saying: “In wellness, you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.”  She was kind enough to save us the trouble.

When RAND’s Soeren Mattke saw this stuff along with her refusal to defend it, he went ballistic.  If you know RAND’s wellness uberguru Soeren Mattke, you know he is a very even-tempered, even-handed guy.  it takes a lot to annoy him…and yet she did.  Here is his smackdown of her work.

Studies Comprising the Meta-Analysis

If you’ve been following the Baicker saga for a while, none of this is news.  Here’s what is news:  thanks to a long plane ride with both an internet connection and an assortment of in-flight films I had already seen, I was able to dig into the actual studies.  My conclusion: combining these third-rate studies in third-rate journals into a Health Affairs meta-analysis was like combining a hodegpodge of individual subprime mortgages written for first-time homebuyers with 5% downpayments and no credit history into an AAA-rated collateralized mortgage obligation.

In terms of timely relevance, of these 22 studies:

Keep in mind too that the average time between when a study begins and publication is about four years, meaning much of this data was collected before some TheySaidWhat? readers were born.  Wellness-sensitive medical events (in both exposed and non-exposed populations of all ages, for reasons having nothing to do with wellness) have fallen by about 70% since most of this data was collected, meaning even if these studies were valid at the time (they weren’t), they have no relevance today. Likewise, dietary recommendations are now the opposite, people can no longer smoke in their offices etc.  There is nothing else in healthcare where data on interventions from 20 and 30 years ago is considered relevant.

In terms of publication bias, consider the journals:

Piling investigator bias on top of author bias, 14 of these studies were authored or co-authored by members of the Koop Award Committee, a committee that can’t spot obviously fabricated outcomes even after I point them out, people whose entire livelihoods depend on making up savings figures, and who endorse admitted liars.

The Actual Data

Every study that used a comparison group employed some variation of the demonstrably invalid participants-vs-non-participants methodology.  Not one study was plausibility-tested.   Almost every one of them showed savings far in excess of what could be saved if all wellness-sensitive medical events were wiped out.

Quite literally no one thinks savings can be achieved within 18 months, and yet the majority of studies with a comparison group were 18 months or less.  All showed massive savings.  One study lasted only 6 months, but showed roughly 20% savings nonetheless.


 

You might ask, how could all that money be saved in such a short period of time? For that answer, you’ll have to wait a couple of days. For Part 2, we will dig into a few of the studies themselves to see where the magic happens…and we do mean “magic”, since nothing else can explain those preternatural results.


6 Comments

  1. Mitch Collins says:

    Great post

    Like

    • whynobodybelievesthenumbers says:

      thanks. After this posting, any wellness vendor that still cites this 3.27-to-1 ROI is demonstrating its ignorance and/or dishonesty.

      Like

  2. Eric Schmidt says:

    Since I started reading your blog (on the recommendation of a friend) I am completely turned around, completely. This is the worst of the worst–a Harvard professor sullying Harvard’s name for political reasons. I hope she gets in a lot of trouble.

    Like

  3. Robert Dawkins PhD says:

    The latest number is 3.6-to-1

    Like

  4. ryanjkrog says:

    This study is actually a required reading for my degree program.

    Like

In the immortal words of the great philosopher Pat Benatar, hit me with your best shot.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Recent Comments

whynobodybelievesthe… on The reward for showing your we…
Marissa on The reward for showing your we…
whynobodybelievesthe… on The reward for showing your we…
whynobodybelievesthe… on The reward for showing your we…
Sam Lippe on The reward for showing your we…
%d bloggers like this: