Home » Posts tagged 'NBER'
Tag Archives: NBER
I would invite everyone to join tomorrow (Tuesday’s) webinar by Ron Goetzel. He will be attempting to undermine the National Bureau of Economic Research’s (NBER) outstanding University of Illinois study, which showed — surprise — that conventional wellness programs don’t come close to changing behavior, let alone saving money. I would love to attend, but I, of course, am not invited to his events any more than he is invited to mine. Oh, wait a sec, I invite him to all my events and alert him to all my postings on linkedin so that he can correct any errors I’ve made. Sorry, my memory failed me there for a second.
Speaking of failed memories, he is being joined on this webinar by Jessica Grossmeier. If that name rings a bill, it’s because she claimed her company, Staywell, saved $17,000 per risk factor reduced — about $3000/pound shed — for British Petroleum, having forgotten that she herself claimed it is only possible to save $105/avoided risk factor. See “British Petroleum Wellness Program is Spewing Invalidity.”
Despite this being the Gold Standard of randomized control trials, he will be accusing the NBER of many errors. (A cynic might note that being accused of making errors in a wellness study by Ron Goetzel is like being accused of cheating on your taxes by Paul Manafort. ) He will argue that:
- The study only covered the first year — he won’t mention that the authors also said the first year suggests nothing “is trending towards savings” in future years either;
- The study contradicts — you guessed it — Kate Baicker’s infamous 3.27-to-1 ROI, without mentioning that the NBER’s principal investigator, as coincidence would have it, reports to Kate Baicker, so it’s pretty unlikely he would diss her unless the data left him no choice;
- The study contradicts all the other findings out there — except for all the other studies testing the par-vs-non-par study design against a benchmark, all of which showed results quite literally identical to the University of Illinois result, in that the wellness program accomplished zero;*
- The participants outperformed the non-participants;
- They haven’t reported on the screening yet;
- It wasn’t a good program. To hear Ron tell it (literally hear him tell it — you can listen to the tape), anytime a program fails, it’s because it wasn’t done correctly. “100 employers [have] programs with really smart ingredients…but thousands of others still don’t do wellness right,” are his exact words in print. He is refusing to name any of them, other than the old Johnson & Johnson analysis. (J&J is a wellness vendor. Investigator bias, anyone?)
What else will he argue? Tough to say. One thing for certain: he won’t mention my name — any more than Bravo did when they wrongly predicted that the EEOC rules would be replaced in January while I predicted the opposite. Instead he uses a new vernacular for my postings: “Industry chatter.”
He probably picked up this idea from Bravo, which uses the phrase “industry noise” to describe me.
Where’s Waldo-meets-Ron Goetzel: Spot the errors and you may win a big prize
So let’s make this interesting. Whoever comes up with the best smackdown of the webinar’s obvious fallacies (and omissions) automatically gets entered in the contest to win the Martha’s Vineyard vacation, with the house, car and private (well, semi-private) beach. It is otherwise open only to people who have won various Quizzify trivia contests, but being able to identify five or ten pieces of “chatter” or “noise” in this self-anointed “expert webinar” clearly counts as being health-literate. To compete, send me an email with an attachment. I’ll pick a couple of finalists and put them on linkedin. (If you don’t want your name used — and Ron does bite back, so I don’t blame you — I will post on my own.)
*The result is also quite consistent with Ron’s observation that there is basically no change in behavior leading to risk reduction. If we are splitting hairs here, Ron found a 1-2% reduction, not 0%. Of course, that took three years.
There is a lot more to this study than meets the eye.
Some tourist attractions feature an “A” tour for newbies and then a “behind-the-scenes” tour for those of us who truly need lives. For instance, I confess to having taken Disney’s Magic Kingdom underground tour, exploring, among other things, the tunnels through which employees travel so as not to be seen out of costume in the wrong “Land.”
Likewise, there have been many reviews of the recent wellness study conducted by the National Bureau of Economic Research (NBER), the first-ever randomized control study of a wellness program. This, however, is the first review to go beyond the “A” tour of the headlines.
By way of background, the headline is that the mainstream wellness program the investigators examined at the University of Illinois did not noticeably move the needle on employee health. They didn’t address return-on-investment (ROI), because there obviously was none. Achieving a positive ROI would require moving the health risk needle—not just by a little, but rather by enough to significantly improve the health of many employees. Then, since wellness-related events, such as heart attacks, would not otherwise have befallen these employees immediately, this improvement would have to be sustained over several years before there was a statistical chance of some events being avoided.
Finally, the magnitude of this improvement would have to be great enough to violate the rules of arithmetic, because it is not mathematically possible to avoid enough medical events to break even on wellness. For instance, it actually costs about $1 million to avoid a heart attack through a screening program.
This finding, therefore, represents an existential threat to conventional wellness programs.
It all boils down to: why would an associate professor (Damon Jones) publicly humiliate his own dean (Katherine Baicker — yes, the very same Katherine Baicker who always seems to be on the wrong side of every wellness debate) …unless he is absolutely sure he is right?
She can’t fire him now because that would get picked up by the lay media. Perhaps she should have paid him $130,000 not to disclose the results.
OK, this time I’m not the one causing the kerfluffle in the wellness industry, though I will confess to being a force multiplier.
Not since 2014, when the very unstable morons at the Incidental Economist made fun of the very stable geniuses who give out the Koop Award and also unequivocally concluded wellness loses money — combined with continued fallout from the Penn State debacle and the Nebraska scandal — has the wellness industry had such a bad year. And it’s only February.
Let’s review what’s happened so far in 2018. First, a federal judge ruled that voluntary wellness programs need to be — get ready — voluntary. The EEOC’s responded with the legalese equivalent of: “Fine, be that way.”
Next, WillisTowersWatson did something that might get them in hot water with the very stable wellness industry leaders: they were honest. They published a study revealing that employees hate wellness even more — way more — than they hate waiting for the cable guy to show up.
Finally, the very unstable National Bureau of Economic Research conducted a controlled study finding basically no impact whatsoever of a wellness program. More importantly, they specifically invalidated the “pre-post” methodology. Even more importantly, they specifically invalidated 78% of the studies used in Kate Baicker’s “Harvard Study” meta-analysis.
Here is an interesting piece of trivia. The lead researcher is an assistant professor at the Harris School of Public Policy. Why is this interesting trivia? Because Katherine Baicker — the Typhoid Mary of Wellness, whose THC-infused 3.27-to-1 ROI is the basis for essentially every subsequent genius wellness outcomes claim — is now the dean of that very same Harris School. I’m just guessing here, but I’d say it’s gotta be a trifle embarrassing when your own subordinate publicly disproves your own study. I mean, it’s one thing for me, RAND, Bloomberg, and anyone else with five minutes, internet access and a calculator to do it, but…your very subordinate?
On the other hand, the researcher, Damon Jones, just demonstrated not just amazing competence, but amazing integrity as well. In other words, he has no future in wellness.
The Wellness Empire Strikes Back
How does the wellness industry respond to these smoking guns threatening their entire revenue stream? Apparently, there is little cause for concern on their planet.
Let’s start with America’s Health Insurance Plans (AHIP), the health insurance industry lobbying group. Here is AHIP’s oxymoronic Wellness Smartbrief (January 26), on the NBER research. Yes, it summarizes the same wellness-emasculating study as the one above, though you could never guess it from the headline:
Continuing, AHIP said:
Offering incentives for completing wellness activities might be more cost-effective than offering incentives for wellness screening, a recent study of a comprehensive program found.
Perhaps AHIP has been infiltrated by Russian trolls, because here’s what the NBER article actually said about “completing wellness activities”:
We…do not find any effect of treatment on the number of visits to campus gym facilities or on the probability of participating in a popular annual community running event, two health behaviors that are relatively simple for a motivated employee to change over the course of one year.
Wellness programs might attract mostly employees who are already fitness-conscious, but the potential to attract healthy employees whose medical spending is already low could nonetheless be a boon to employers, the researchers found.
And on the subject of “the potential to attract healthy employees” being a “boon to employers,” the authors actually said:
We further find that selection into wellness programs is associated with both lower average spending and healthier behaviors prior to the beginning of the study. Thus, one motivation for a firm to adopt a wellness program is its potential to screen for workers with low medical spending. Considering only health care costs, reducing the share of non-participating (high-spending) employees by just 4.5 percentage points would suffice to cover the costs of our wellness program intervention.
In other words, you can apply some workplace eugenics to your company by using wellness to weed out obese employees, employees with chronic or congenital diseases, and so on. Good for you!
Soon, if AHIP and others have their way, there will be no need for guesswork in eugenics: employer wellness programs will be able to screen these employees out based on their actual DNA.
AHIP’s take on AARP v. EEOC
And now, AHIP’s take on this landmark case, their ace reporters scooping everyone with this February 2 headline on the December 20th court ruling:
Here are more typical headlines on that court ruling, headlines that came out the same month that the court ruling came out. Perhaps AHIP used the interim six weeks to focus-group various verbs until they settled on…tweak???
AHIP: It’s not just the headlines
One prominent healthcare executive recently attended an AHIP conference and reports:
I just returned from one of the dumbest meetings I’ve ever attended in Washington. Report of a new “study” by AHIP. Turns out people don’t mind health costs all that much, they just want more benefits. And everything is hunky-dory with their health plans, people like them so much. They love wellness benefits and crave more. Prescription drug prices have been nicely controlled thanks to the competitive marketplace (no, I am not making this up or exaggerating for drama). For every $1 employers spend on benefits workers get $4 in value. Priorities for SHRM rep: Fitbits for all employees, solving the outrage that only 20% of her employees got an annual physical. 85 cents of every dollar spent on health care goes to chronic disease.
Over these same two hours, I’d estimate about a thousand employees were misinformed, harmed or harassed by wellness vendors, roughly equal numbers of employees got useless annual checkups, employers spent about $200-million on healthcare and 40 people died in hospitals from preventable errors. But I’m being such a Debbie Downer! I’m going home to read Why Nobody Believes the Numbers to remove myself from this alternative universe.
Enter the Health Enhancement Research Organization (HERO)
HERO’s Prevaricator-in-Chief, Paul Terry, is demonstrating his usual leadership abilities in this crisis, of course. After all, HERO is the wellness industry trade association and these three items — the NBER invalidating their product, employees hating their product, and a federal judge forbidding them to force employees to use their product — represent existential threats to his “pry, poke and prod” members.
Teddy Roosevelt said, “complaining about a problem without posing a solution is called whining.” It’s a quote that also reminds me why I’ve not thought of angry bloggers who target health promotion [vendors] as bullies. Though they relish trolling for bad apples, their scolding is toothless, more the stuff of chronic whiners.
I suspect he is talking about me here as the “chronic whiner” who is “scolding” them. Or perhaps he is referring to the “angry bloggers” at the Los Angeles Times, the New York Times, Slate, or STATNews, since those “toothless” publications seem to be scolding wellness vendors more than I ever have. For instance, I’ve never called wellness vendors’ offering a “scam” or a “sham.” I simply quote these very stable wellness geniuses verbatim, as above or below, or last week.
Being quoted verbatim, not angry bloggers, is their worst nightmare. (One thing I would concede, though, is that “Paul Terry and the Angry Bloggers” would be a great name for a rock band.)
Yep, looks like the implosion of his industry all my fault. Otherwise, I’m not quite sure who is the “angry blogger” he is referring to, other than to note that Mr. Terry himself seems to blog a tad angrily himself, both above, and here…
Why I choose to ignore the blogger critics: We’re fortunate to work in a profession with a scant number of vociferous critics. My take is that there is one thing these few angry loners [Editor’s note: the complete “scant list” of the 220 “few angry loners” who have been “vociferous critics” can be found here] want more desperately than attention: that’s to be taken seriously. What they fail to comprehend is that as they’ve gotten ever more farfetched and vitriolic in search of the former, they’ve cinched their inability to attain the latter.
Baiting people with misinformation and offensive insults (but just a tad under highly offensive) is a pesky ploy that trolls hope will eventually land a bite that confers credibility where there is none. Even reading such drivel is a form of taking the bait; responding is swallowing it whole. Some say dishonesty should not go unchallenged and I respect their view; nevertheless, I’m convinced responding to bloggers who show disdain for our field is an utter waste of time. I’ve rarely been persuaded to respond to bloggers, and each time I did it affirmed my worry that, more than a waste, it’s counter-productive.
…and especially here, a seemingly incongruous decision to “act out” by someone who claims to be “choosing to ignore the blogger critics.”
Having read years of my “drivel” alongside Mr. Terry’s posting explaining why you shouldn’t “swallow this bait,” perhaps readers might opine here: which of us, exactly, is the “chronic whiner”?
Coincidentally, when I run live health-and-wellness trivia contests, the first of our 3 rules is: No Whining. Seems to me that he would have just violated it. Indeed the only rule HERO hasn’t violated so far is #3 below. Not that I want to put ideas in their head.
Not to be confused with those immortal words often attributed to the great philosopher Yogi Berra, a big joke among economists is: “An economist is someone who upon learning how something works in practice, wonders how it will work in theory.”
That joke morphed into reality last month — though it was a controlled study testing a theory, rather than the theory itself. The “theory” would be that inactive unmotivated non-participants can be used as a control for active motivated participants. Ironically, this study design has never been proposed as legitimate, even in theory. Wellness “researchers” like it because it always show savings, even when nothing happens. For example, even when there was no program for the participants to participate in.
Obviously if this were a legitimate design, the FDA would approve it for clinical trials, saving a ton of time and money vs. having to do controlled trials.
To wit, the National Bureau of Economic Research (NBER) just published a study showing that the participants-vs-non-participants (“par-vs.-nonpar”) study design, used extensively by the very stable geniuses in the wellness industry to do their alt-research to fabricate their alt-findings, is completely invalid.
Highlights of NBER study for the wellness industry: bad news first
First the bad news. Fasten your seat belts and be shocked, shocked to learn that the researchers could identify:
- No noticeable change in health behaviors due to the wellness program
- No noticeable change in health outcomes due to the wellness program
- Clear self-selection bias among participants opting into the wellness program
Lest anyone think we are taking this out of context, here are their exact words:
We do not find any significant effects of treatment on total medical expenditures, employee productivity, health behaviors, or self-reported health measures in the first year following random assignment. We further investigate the effect of our intervention on medical expenditures in greater detail, but fail to find significant effects on different quantiles of the spending distribution or on any major subcategory of medical expenditures (pharmaceutical, office, or hospital). We also do not find any effect of treatment on the number of visits to campus gym facilities or on the probability of participating in a popular annual community running event, two health behaviors that are relatively simple for a motivated employee to change over the course of one year.
This of course merely confirms what we observe in outcomes reports published by wellness vendors, including the two most recent proud recipients of the Deplorables Awards. Actually, in the case of Wellsteps there was indeed a noticeable change in health outcomes among program participants — they got worse.
Also, the authors — no doubt anticipating the objection from the very stable geniuses at the Health Enhancement Research Organization — specifically note that nothing in Year One’s results presage any step-function improvement in Year Two. So the specious “Wait ’til next year” argument is off the table.
You might be thinking, “Another nail in the wellness industry coffin.” True, except that there almost isn’t room for any more nails. Soon the coffin will have enough nails to create its own gravitational field.
Next, the good news
Here are the two pieces of good news for the wellness industry. One finding was:
Our 95% confidence intervals rule out 78 percent of previous estimates on medical spending and absenteeism.
That means that it is possible that 22% of previous estimates may conceivably not be completely invalid. This is not to say that 22% are valid, just that they aren’t automatically invalid. That is great news for the wellness industry, where clearing the bar for not being automatically invalid is cause for celebration. As Dave Chase says, the bar for wellness is so low a snake could jump over it.
However, the 78%-totally-invalid figure specifically invalidates Katherine Baicker, author of the so-called “Harvard Study.” Depending on whether you are a wellness vendor or an oppressed employee, she is either the Johnny Appleseed or the Typhoid Mary of wellness. Her famous 3.27-to-1 ROI was tallied entirely from par-vs.-nonpar studies, exactly the methodology that the NBER just invalidated, citing exactly the studies she cited. (The three examples in my study referenced above were also part of her meta-analysis.)
So perhaps she might now make a formal statement regarding par-vs-non-par as a study design? Either a retraction or a defense. Just something that clarifies her previous statements, which seem to be neither retractions nor defenses but rather more like excuses:
- It’s too early to tell (um, after 30 years of workplace wellness?)
- She has no interest in wellness any more
- People aren’t reading her paper right (we’re only reading the headline, the data, the findings and the conclusion, apparently)
- “There are few studies with reliable data on the costs and the benefits” (um, then how were you able to reach a conclusion with two significant digits?)
The irony is that Kate Baicker has otherwise done outstanding research. Her study on Oregon Medicaid is a classic. In Oregon at the time, Medicaid eligibility was determined by lottery amongst applicants. That meant that — quite the opposite of wellness control groups — the control group of people not picked in the lottery had equal motivation to seek insurance coverage as people who were picked. After following both groups going forward, her finding was that obtaining insurance to access basic medical care did not change outcomes. (Having insurance did bring peace of mind, though.)
And yet somehow in “Workplace Wellness Can Generate Savings,” she was quite comfortable reaching a conclusion that was completely inconsistent with her Oregon finding, not to mention the Law of Diminishing Returns: throwing additional unrequested, generally unwanted, and largely misdirected medical interventions and advice at employees who already have insurance — and recall that most insured Americans are drowning in medical care — could dramatically improve their outcomes enough to calculate not just a massive ROI, but an ROI precise to two significant digits.
What she will hopefully learn through the NBER study is something that I learned 11 years ago: when the data proves you wrong, fess up. Then people like me have to find someone else to blog about. Fortunately, in wellness, that is not a heavy lift.
The second piece of good news for the wellness industry
To quote the study:
…wellness incentives may shift costs onto unhealthy or lower-income employees if these groups are less likely to participate in wellness programs. Furthermore, wellness programs may act as a screening device by encouraging employees who benefit most from these programs to join or remain at the firm…
To be clear, this calculation does not imply that adoption of workplace wellness programs is socially beneficial. But, it does provide a profit-maximizing rationale for firms to adopt wellness programs, even in the absence of any direct effects on health, productivity, or medical spending. [emphasis theirs]
In other words, employers can use wellness programs to subtly discriminate against unhealthy — read, older and poorer — workers. Many of the very stable geniuses in the wellness industry will be happy to hear this. The exceptionally stable genius who will be most thrilled to hear this is Michael O’Donnell. Mr. O’Donnell is the former Prevaricator-in-Chief of the wellness industry trade publication and a current member of the Koop Award cabal. These are excerpts from one of his editorials:
First, he says prospective new hires should be subjected to an intrusive physical exam [editor’s note: notwithstanding the fact that this is totally illegal], and hired only if they are in good shape. OK, not every single prospective new hire needs to be in good shape — only those applying for “blue collar jobs or jobs that require excessive walking, standing, or even sitting.” Hence he would waive the physical exam requirement for mattress-tester, prostitute, or Koop Award Committee member, because those jobs require only excessive lying.
Second, he would fine people for not meeting “outcomes standards.” In an accompanying document, he defines those “outcomes standards.” He specifies fining people who have high BMIs, blood pressure, glucose, or cholesterol.
In other words, Mr. O’Donnell wants to charge for insurance by the pound, as that accompanying document says. Actually, by BMI, which of course is of dubious value as a measure of weight, let alone health.
Here is his actuarial formula:
Although having read his very stable arithmetic elsewhere in this same document, I’d worry about the accreditation status of any actuarial school, or for that matter any school of any kind within the 50 states, that would accept him:
Thirty-one states have no laws that prohibit employers from using smoking status as the reason for not hiring… In the remaining 29 states…smoking status cannot be used as the reason for not hiring.
I’m not waiting around for a retraction from this genius either.