“You know that laundry detergent that gets bloodstains out of t-shirts?  I’d say if you’re routinely coming home with bloodstains in your t-shirts, laundry probably isn’t your biggest problem.”

Recognize that line?  That was Jerry Seinfeld’s signature joke, the one that more than any other propelled his early fame.  Having used it for a decade, he retired it shortly after Seinfeld went on the air.

Now it’s time to retire the wellness industry’s signature joke: the so-called Harvard Study and its 3.27-to-1 ROI. Among other things, as we’ll see, a large chunk of the data in that study also predates Seinfeld, a show which recently celebrated its 25th anniversary.  You know the old line: “How can you tell a lawyer is lying?” Answer: “His lips move.”  Well, the way you can tell a wellness vendor is lying is that they still cite that study, knowing it to be completely invalid.  (If they actually think that 3.27-to-1 is valid, in a way that’s even worse.)

Some Background

The article, which today wouldn’t pass peer review by the normally cautious Health Affairs, was rushed into publication during the healthcare reform debate as a favor to David Cutler, a healthcare advisor to President Obama and an author on the study.  Along with the results from the Safeway wellness program — which turned out not to exist (and I’m referring not just to the results but literally to the program itself) — this meta-analysis provided cover for the 30%-to-50% clawback provision (for non-smokers and smokers respectively)  in what became ironically known as the Safeway Amendment to the Affordable Care Act.

In turn, this clawback was needed to secure the support of the Business Roundtable (BRT) for the Affordable Care Act. The BRT was and is obsessed with this provision. Not because they are especially concerned about employee health — the guy who ran their wellness lobbying also ran a casino company, exposing thousands of employees to second-hand smoke.  Rather, because this clawback provision creates billions in forfeitures that go back into the coffers of its member organizations.  Bravo Wellness accidentally spilled that secret.  The worse the program, the more an organization can save via forfeitures. For instance, Wellsteps saved millions for the Boise School District even though healthcare spending/person jumped–simply by creating a program so unappealing that employees preferred to obtain insurance through their spouses. (Either that or they outright lied–you make the call.)

How Prof. Baicker Invalidated Her Own Study

The title was definitive and the conclusion was amazingly precise. The title was: “Workplace Wellness Can Generate Savings.”  Not “might” or “possibly could on a good day” but “can.” And the conclusion itself left little doubt.  The ROI wasn’t “approximately 3-to-1” or even “approximately 3.3-to-1” but rather a statistically precise 3.27-to-1.  You don’t make such a definitive pronouncement without being ready to defend it.

Or maybe you do. Here’s how Professor Baicker’s story changed on various occasions once it became clear that her result was a major outlier, and that wellness loses money:

1. It’s too early to tell whether these programs pay off, and employers should experiment on their employees;
2. She’s not interested in wellness any more;
3. People aren’t reading the paper right. They aren’t paying enough attention to the “nuances.” Shame on the readers!
4. “There are few studies with reliable data on the costs and the benefits”

This may be unique in healthcare history: writing a paper and then almost immediately retracting its key finding, claiming you have no interest in the topic even as your fans deify you, and then blaming readers for believing the conclusion.

But the biggest head-scratcher is the 4th item.  Here is someone who just published a conclusion with two-significant-digit precision now saying (accurately, as we’ll see) that, oh, by the way, the studies she analyzed are mostly garbage.  You know the old saying: “In wellness, you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.”  She was kind enough to save us the trouble.

When RAND’s Soeren Mattke saw this stuff along with her refusal to defend it, he went ballistic.  If you know RAND’s wellness uberguru Soeren Mattke, you know he is a very even-tempered, even-handed guy.  it takes a lot to annoy him…and yet she did.  Here is his smackdown of her work.

Studies Comprising the Meta-Analysis

If you’ve been following the Baicker saga for a while, none of this is news.  Here’s what is news:  thanks to a long plane ride with both an internet connection and an assortment of in-flight films I had already seen, I was able to dig into the actual studies.  My conclusion: combining these third-rate studies in third-rate journals into a Health Affairs meta-analysis was like combining a hodegpodge of individual subprime mortgages written for first-time homebuyers with 5% downpayments and no credit history into an AAA-rated collateralized mortgage obligation.

In terms of timely relevance, of these 22 studies:

• Only 1 is less than 10 years old (that was Highmark, and they fired their vendor, ShapeUp)
• Only 2 are from the 21st century
• 5 predate Seinfeld

Keep in mind too that the average time between when a study begins and publication is about four years, meaning much of this data was collected before some TheySaidWhat? readers were born.  Wellness-sensitive medical events (in both exposed and non-exposed populations of all ages, for reasons having nothing to do with wellness) have fallen by about 70% since most of this data was collected, meaning even if these studies were valid at the time (they weren’t), they have no relevance today. Likewise, dietary recommendations are now the opposite, people can no longer smoke in their offices etc.  There is nothing else in healthcare where data on interventions from 20 and 30 years ago is considered relevant.

In terms of publication bias, consider the journals:

• 3 no longer exist
• 11 are wellness promotional journals, like the Journal of Occupational and Environmental Medicine and the American Journal of Health Promotion, both of which have thoroughly discredited themselves as legitimate journals and neither of which has ever published a case study showing negative returns, despite the editor of the latter saying that “90% to 95% of programs lose money” and “RCTs show a negative ROI“

Piling investigator bias on top of author bias, 14 of these studies were authored or co-authored by members of the Koop Award Committee, a committee that can’t spot obviously fabricated outcomes even after I point them out, people whose entire livelihoods depend on making up savings figures, and who endorse admitted liars.

The Actual Data

Every study that used a comparison group employed some variation of the demonstrably invalid participants-vs-non-participants methodology.  Not one study was plausibility-tested.   Almost every one of them showed savings far in excess of what could be saved if all wellness-sensitive medical events were wiped out.

Quite literally no one thinks savings can be achieved within 18 months, and yet the majority of studies with a comparison group were 18 months or less.  All showed massive savings.  One study lasted only 6 months, but showed roughly 20% savings nonetheless.

You might ask, how could all that money be saved in such a short period of time? For that answer, you’ll have to wait a couple of days. For Part 2, we will dig into a few of the studies themselves to see where the magic happens…and we do mean “magic”, since nothing else can explain those preternatural results.