This is the third in a series deconstructing the Health Enhancement Research Organization’s (HERO) attempt to replace the basic outcomes measurement concepts presented to the human resources community in Why Nobody Believes the Numbers with a crowdsourced consensus version of math.  The first installment covered Pages 1 to 10 of their outcomes measurement report, where HERO shockingly admitted wellness hurts morale and corporate reputations.  The second installment jumped ahead to page 15, where HERO shockingly admitted wellness loses money. This report covers pages 11-13.  Next week we shall be covering Page 14. 

4 out of 5 wellness vendors don’t get this

Spoiler Alert:  The wellness industry believes that math is a popularity contest.   (We have a million-dollar reward if they can show that’s true.  More on that later.)

All the luminaries in the wellness industry got together to crowdsource arithmetic, and put their consensus (a word they use 50 times) in an 88-page report.  Unfortunately, math is not a consensus-based discipline, like democracy.  It is not even an evidence-based discipline, like science.  It is a proof-based discipline.  A methodology that doesn’t work in hypothetical mathematical circumstances is proven wrong no matter how many votes it gets.

The pages in question list 7 “methodologies” for measuring outcomes.  To begin with, consider the absurdity of having 7 different ways to measure.  Imagine if you asked your stockbroker how much money you made last year, and were told:  “Well, that all depends.  You could measure that seven different ways.  And by the way, six of those ways will overstate your earnings.”   Math either works or it doesn’t.  There is only one right answer.

Methodology #1: “Cost Trend Compared with Industry Peers”

This methodology “may require consulting expertise.”

As a sidebar, one of the many ironies of this HERO report is that most of these methodologies emphasize the need for actuarial or consulting “inputs” or “analytic expertise”…and yet no mention was made on Page 10 of the cost of this expertise when all the elements of cost were listed.  While not mentioned as a cost element, consulting firms are very expensive  And even if consulting were free, we generally recommend hiring only consultants to do outcomes report analysis who are certified in Critical Outcomes Report Analysis by the Validation Institute.

By contrast, Staywell and Mercer offer an example of what happens when you as a buyer use non-certified “consulting expertise” to evaluate a vendor.  Here’s what happens:  the vendor wins.  Needless to say, Staywell showing savings 100x  greater than what Staywell itself said was possible simply by reducing a few employees’ risks raises a lot of questions.  But despite repeated requests and offers of honoraria to answer these questions, Mercer wouldn’t answer and the only response Staywell gave us was to accuse us of bullying them.  Staywell and Mercer held firm to the Ignorati strategy of not commenting—even though Mercer was representing the buyer (British Petroleum), not the vendor.  Oh, yes—both Staywell and Mercer are represented on the HERO Steering Committee.

To HERO’s credit, they do admit the obvious for Methodology #1: If all your peers are using the same vendors, who recommend the same worthless annual checkups, the same overscreening/overdiagnosis, the same lowfat(!) diets, and the same consultants to evaluate all the phony savings attributable to these checkups, diets, and biggest-loser contests, obviously you’ll get the same results.  And since trend is going down everywhere (including Medicare and Medcaid, which have no wellness), everyone gets to “show savings.”

Methodology #2:  “Inflection on expected cost trend.”

Mercer has been a big proponent of this methodology, as in the previous Staywell example.  At one point they used “projected trend” to find mathematically impossible savings for the state of Georgia’s program even though the FBI(!) later found the program vendor, APS, hadn’t done anything. In North Carolina, they projected a trend that allowed them to show massive savings in the state’s patient-centered medical home largely generated, as luck would have it, by a cohort that wasn’t even eligible for the state’s patient-centered medical home.

Comparing to an “expected” trend is one of the most effective sleight-of-hand techniques in the wellness industry arsenal.  Every single published study in a wellness promotional journal comparing results to “expected trend” has found savings.   And have you ever hired a consultant or vendor to compare your results to “expected trend” who hasn’t found “savings”?  We didn’t think so.

QED.

Methodology #3: “Chronic vs. non-chronic cost trend.”

The funny things about this methodology are twofold.

First, the HERO Committee already knows this methodology is invalid because it was disproven in Why Nobody Believes the Numbers (and I offered an unclaimed $10,000 reward for finding a mistake in the proof).   We know that people on the Committee have read my book because at least one of them – Ron Goetzel – used to copy selected pages from it until the publisher, John Wiley & Sons, made him stop.  Methodology #3 was the fallacy on which the entire disease management industry was based.   I myself made a lot of money measuring outcomes this way, until I myself proved I was wrong.  At that point, integrity being more important to me than money, I changed course abruptly, as memorably captured by Vince Kuraitis’ headline: Founding Father of Disease Management Astonishingly Declares: “My Kid Is Ugly“.  (Naturally the benefits consulting industry filled the vacuum created by my withdrawal from this market, and plied their clients with worthless outsourced programs that more than coincidentally generated a lot of consulting fees.)

If you had perfect information and knew who had chronic disease (before the employees themselves did) and everyone stayed put in either the non-chronic or chronic categories, you could indeed use non-chronic trend as a benchmark, mathematically (though the epidemiology is still very squirrelly).  The numbers would add up, at least in a hypothetical case.

But we can’t identify anywhere near 100% of the employees who have chronic disease.  Absent that perfect information, any fifth grader could understand the proof that this methodology is fabricated, as follows.  Assume that 10 people with a chronic disease cost $10,000 apiece both in the baseline and in the study period.  Their costs are therefore flat.  The program did not reduce costs between periods.

Now add in 10 people with undetected chronic disease as the “non-chronic benchmark.”  Maybe they are ignoring their disease, maybe they don’t know they have it, maybe they are misdiagnosed, maybe the screen was wrong (vendor finger-pricks are very unreliable).  Assume these 10 people cost $5000 in the baseline…but they have events in the study period so their costs become $10,000.

That makes the “non-chronic trend” 100%!  Suddenly, the program vendor looks much better because they kept the costs of the chronically ill cohort constant even though the “benchmark” inflation was 100%.

At Quizzify, wellness results always add up.

Second, Why Nobody Believes the Numbers has already shown how to make this methodology valid mathematically (though the epidemiology applied to that math might still be squirrelly, and there could still be random error in non-hypothetical populations).  You simply apply a “dummy year analysis” to the above example.  So do exactly what is described above, but for a year-pairing before the program.   Then you’ll know what the regression-to-the-mean bias is, and apply that bias to the study years.  So If in fact the “non-chronic trend” is always 100% due to the people with unrecognized chronic disease, you would take this trend out of the benchmark non-chronic population before applying that trend to the chronic population.  In this case, as in every case, the bias is eliminated.  This is called the Dummy Year Adjustment.  (Chapter 1 of Why Nobody Believes the Numbers offers several examples of the DYA.)

Proofs are best understood to be proofs if accompanied by rewards, since only an idiot would monetarily back a proof that wasn’t a proof.   So here’s what we propose for this one:  I’ll up my $10,000 reward to $1,000,000.  A panel of Harvard mathematicians can decide who is mathematically right.  The HERO Committee escrows a $100,000 nuisance fee for wasting my time and paying for the panel if they are wrong.  (We’ll pay if we lose.) They present Methodology #3.  We lose the $1,000,000 if the panel votes that this HERO methodology is valid without our “Dummy Year Adjustment.”

My challenge: Either collect your $1,000,000, or publicly apologize for proposing a methodology which you know to be made up.  Or is offering you a million dollars “bullying,” a word defined very non-traditionally in this field?   Our bad.

Yes, we know this sounds like a big risk but you might remember the old joke:

Science teacher:  “If I drop this silver dollar into this vat of acid, will it dissolve?”

Student: “No, because if it would, you wouldn’t do it.”

Methodologies 4 and 5:  The Comparison of Participants to Non-Participants

Besides not making any intuitive sense that active motivated engaged participants are somehow equivalent to inactive unmotivated non-participants, Ron Goetzel already admitted this methodology is invalid.  Health Fitness Corporation, accidentally proved that on the slide below.

Note that they “matched” the participant (blue) and reference (red) groups in the 2004 “baseline year” but didn’t start the “treatment” until 2006.  However, in 2005, they already achieved 9% savings vs. the “reference group” even without a program.  This “mistake” was in plain view, and was pointed out to them many times, politely at first.  Page 85 of Why Nobody Believes the Numbers showed it, but as the screenshot below shows, I was too polite to mention names or even to call it a lie, figuring that as soon as Health Fitness Corporation or Ron Goetzel saw it, they/he would be honest enough to withdraw it.

Not knowing the players well, I naively attributed the fact that HFC used this display to a rookie mistake, rather than dishonesty.  That was plausible because rookie mistakes are the rule rather than the exception in this field.  (As we say in Surviving Workplace Wellness, the good news about wellness vendors is that NASA employees don’t need to worry about their job security because these people aren’t rocket scientists.)

On the advice of colleagues more familiar with the integrity of the wellness industry true believers, I also tried a test of the rookie-mistake hypothesis: I strategically placed the page with this display next to the page that I knew Ron Goetzel would be reading (and copying), a page whereon I complimented him on his abilities.  I might the the world’s only bully who publicly compliments his victims and offers to pay them money:

That way, I would know that if Mr. Goetzel and his Koop Committee and their sponsors HFC didn’t remove this display, it was due to a deliberate intentiion to mislead people, not an oversight or rookie mistake.

Sure enough, that display continued to be used for years.  Finally, a few months ago, faced with the bright light of being “bullied” in Health Affairs, HFC withdrew the slide.  Ron “the Pretzel” Goetzel earned his moniker, twisting and turning his way around how to spin the fact that this “mistake” was ignored for so long despite all the times it had been pointed out.  He ending up declaring the slide “was unfortunately mislabeled.”  He gave no hint as to who did the unfortunate mislabeling, despite being repeatedly asked.  We suspect the North Koreans.  The whole story is here.

Summary and Next Steps

The first five of these methodologies in Pages 13-14 have several things in common:

• They all contradict the 6^th methodology;
• They contradict the statement on page 15 that the only significant savings is in reducing admissions. Of course, self-contradiction is embedded in Wellness Ignorati DNA.  To paraphrase the immortal words of the great philosopher Ned Flanders, the Wellness Ignorati  “believe all the stuff in wellness is true.  Even the stuff that contradicts the other stuff.”
• They call for megadoses of consulting and analytic expertise, contradicting the list on Page 10 that omits the cost of outside expertise.

Speaking of Methodology #6, our next installment will cover it.  It’s called event-rate based plausibiltiy testing.  I would know a little something about that methodology, since I invented it.  I am flattered that the Wellness Ignorati, seven years later, are finally embracing it.  I am even more flattered that they aren’t attributing authorship to me.  No surprise.  That’s how the Wellness Ignorati got their name – by ignoring inconvenient facts.  Ignoring facts means they cross their fingers that their customers don’t have internet access.  Customers who do can simply google on “plausibiltiy test” and “disease management” and see whose name pops up.