How the health of Boise’s teachers deteriorated while Wellsteps was winning a Koop Award
We’ve always made the observation that in wellness, you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself. Wellsteps attempts to prevent people from doing this by making the data almost impossible to read in the first place.
And a good thing, because watch what happens if you do figure out how to read it. (For newbies to this thread, this is the Boise School District results. We have already covered Wellsteps’ inability to understand math and their inability to understand alcohol and smoking in two previous postings. This one is about Wellsteps’ inability to understand health. You’d think that would cover all the bases…and yet we have two more posts coming next week.)
Start with the BMI, which of course is a dubious measure to begin with. No variances are given, so we will use the average figures. 683 people reduced their BMI, while 1641 employees got larger. The changes are trivial in any event, but do move in the wrong direction. And according to the p-values, the changes are significant.
Systolic blood pressure is similar. 1992 people got worse, while only 360 improved. (Diastolic did better, as did glucose.)
Likewise, cholesterol, where only 216 people improved while 1434 got worse and 691 stayed almost the same.
in all these cases, you might say, “yes, but the highest-risk people improved.” Of course, they did. The highest-risk cohort almost always shows improvement and the lowest risk cohort almost always deteriorates. This is called the “natural flow of risk.” Taking credit for improvements while ignoring people who deteriorate is indeed the wellness industry business model. Compare Wellsteps “improvements” on the employees with the most health risks to Dee Edington’s “natural flow of risk” (also a bit inscrutable on its own). The “natural flow” shows about half the high-risk people improving on their own (1640 to “medium” and 678 all the way to “low”), with no program at all.
The impact on the population as a whole
In another attempt to demystify the Wellsteps results and assuming a variance of zero (since none was provided), I simply summed all the improved biometrics and compared that total with the total that deteriorated. It turns out that 6397 deteriorated while only 5293 improved.
In other words, the physical health of the Boise School District suffered during the Wellsteps wellness program.
And no wonder–Wellsteps was screening the stuffing out of these people, far in excess of guidelines, and encouraging unneeded annual checkups. This conclusion is reached without even taking into account the likelihood of employees lying on HRAs, cheating during the weigh-ins, or Wellsteps putting their fingers on the scale in the measurements above (“adjusted for age and sex”), as they are wont to do.
Saying that health actually deteriorated during a program would be controversial, but there is no need to take our word for this conclusion. You can ask the employees themselves. Come to think of it, you don’t even have to ask them. Wellsteps asked them…and they agreed: their health got worse. Keeping with the inscrutability theme, you have to squint real hard to see it on the last line, but it’s there: a statistically significant decline in self-reported health status over the two-year period:
Wellsteps termed this deterioration “an improvement” but then again, Boise employees could forfeit a total of $830 by not playing along with their wellness scheme…and Wellsteps calls that figure a “small incentive.” So in addition to having their own version of math, they have their own vocabulary.
However, there is one instance in which I would agree with Wellsteps, and that is they certainly have the following figured out:
Share this:
Wellsteps: 20% of Boise’s teachers drink too much
Yikes! Wellsteps just announced–and the Koop Award Committee just agreed–that 20% of Boise’s teachers have a drinking problem. Boise parents, before you start worrying, the good news is that it may not specifically be the teachers with the drinking problems. It’s technically 20% of all school employees, so you can relax. Maybe the problem is more prevalent in non-teaching employees, such as, oh, I don’t know, school bus drivers.
Is Wellsteps right or wrong about the drinking problem in Boise? Read on and then you can decide for yourself. Having read their alcohol analysis ourselves, we’re doubling down on our original conclusion that Wellsteps deserves a Koop Award like Vladimir Putin deserves a Nobel Peace Prize. Wellsteps is way off base, meaning Boise’s schoolchildren are safe.
Our original conclusion was reached simply by looking at the “smoking gun” slide in our previous posting, which they had tried to suppress (but we took a screenshot). We’ll expand on that slide another day. For now, just ignore that particular smoking gun and instead consider the rest of the smoking arsenal they just wrote up in their self-congratulatory press release.
Speaking of smoking, today in addition to reviewing the Wellsteps analysis of Boise’s drinking, we’ll also include their smoking write-up.
Alcohol Consumption in the Boise School District
There was a saying in the old Soviet Union: “We pretend to work. They pretend to pay us.” For employees completing the Wellsteps’ Health Risk Assessment (HRA), the corollary would be: “We pretend to tell the truth. They pretend to believe us.” Of 3284 employees, 671 admitted to drinking. That means 80% of the Boise School District employees are teetotalers, as compared to 30% of the country as a whole. Or else — a shocker — 50% of the HRA respondents are lying.
The Koop Committee did not question this result.
But wait. There’s more.
Now let’s consider the 20% of respondents who admitted to drinking at all. How much do they claim to drink? A paltry 1.31 drinks a day, or about 10 ounces of alcohol a week. By contrast, the top 20% of drinkers in the US not employed by the Boise School District (“Boise”) consume roughly 43 ounces of alcohol a week. Let’s overlay the Boise School District consumption (in blue) against US consumption, on the following chart. It shows 10 ounces for Boise vs. 15 ounces for the second-to-top decile for the US as a whole, and vs. a whopping 73 ounces for the top decile.
The Koop Committee did not question this result.
But wait. There’s even more.
How much do you have to drink in order for Wellsteps to accuse you of having a drinking problem? Answer: any amount. According to Wellsteps, drinking alcohol at all counts as a “worst health behavior.” Shame on us for not realizing the evils of a glass of wine!
This screenshot from their writeup captures both the average respondent’s self-reported consumption of alcohol, at the very top, with the “worst health behaviors” in alcohol consumption at the very bottom. Note that the two rows of figures are identical. In other words, everyone who drinks has a “high level of…alcohol use” no matter how much they drink. The average consumption (meaning at least half are even lower) to count as a “worst behavior” and “high level of alcohol use” is the aforementioned 10 ounces a week.
The Koop Committee did not question this result.
Smoking
Squirrelly HRA findings aren’t confined to drinkers. Of the 3284 school district employees, apparently only 77 smoke (see chart above), placing the Boise School District’s alleged smoking rate at roughly 2.5%, or about 80% lower than the national or Idaho average. And those 2.5% are very light smokers. Whereas the average Idaho smoker burns through 16 cigarettes a day, Boise School District smokers abstain from smoking 10 days a month. So just like alcohol, hardly anyone smokes…and even the smoking employees hardly smoke.
With this amazingly healthy population, it’s a wonder Wellsteps could make any improvements at all. But as we will see later this week, they made enough improvements to save massive sums of money, defying all odds and all rules of arithmetic, which are strictly enforced.
The Koop Committee did not question this result.
Share this:
We called it! We predicted the combination of invalidity and cronyism would win Wellsteps a Koop Award!
We cannot, cannot make this stuff up.
Wellsteps, which could take lessons in integrity from the presidential candidates, was obviously fabricating the outcomes for its Boise School District. How do we know this? Simple. Costs can’t rise and fall at the same time, even using wellness industry math. And yet Wellsteps claimed they did.
As soon as we saw how obviously, hilariously invalid their result was, we predicted that Wellsteps would win a Koop Award for the Boise School District. We based this prediction on the combination of data fabrication, cronyism, nonsense, and cluelessness which are the DNA of both that award and of Wellsteps’ phony outcomes. Our only mistake was thinking they would win in 2015, but you’ll see at the end, we said that if they didn’t win in 2015, it was because they were late, and would win in 2016, which is what they just did.
Note when you compare Wellsteps Stumbles Onward: Costs Rise and Fall at the Same Time to their current press release, you’ll see there is something missing from the latter. They removed the “smoking gun,” which invalidates the entire program. In both documents, they said costs absolutely declined across the whole population, including non-participants…
…but on a per capita basis the costs of both participants and non-participants increased, at least in their initial writeup. This slide below has now been conveniently disappeared from their press release. I suspect this is not an accident. Here it is:
Participants’ cost rose just a little while non-participants’ cost rose a lot. This separation is due to the proven fallacy of the participants-vs-non-participants methodology. Even so, the line graph says the whole enchilada at Boise declined, not just the participants.
The only way per capita costs could increase AND total costs decline is if the program is so bad that employees prefer to join their spouse’s health plan, or if the number of employees declines. But even the most dishonest wellness vendor wouldn’t credit either of those changes to their wellness program, and no member of the Koop Award Committee could “overlook” that impossibility.
Or would they?
Be sure to read the second installment, where we dive even deeper into the Wellsteps doodoo.
Share this:
Wake the neighbors. Phone the kids. Goetzel admits wellness loses money (again).
If Ron Goetzel keeps telling the truth, at some point we need to believe him.
The July 20 Chicago Tribune quotes Mr. Goetzel as saying that wellness can reduce risk factors by 1% to 2%. Yikes! He’s right. Mr. Goetzel is finally telling the truth. Or, in the immortal words of the great philosopher Rick Perry, even a stopped clock is right once a day.
More importantly, the debate on savings is over (again). We have learned, also in the immortal words of the great philosopher Rick Perry, whether or not who is right.
Let’s see what happens if Mr. Goetzel is right…
That 1% to 2% reduction, by the way, are outcomes from the allegedly best wellness programs, the ones that have won his Koop Awards. Here is a list of winners, along with the risk factor reduction. The risk factor reduction doesn’t count failures, meaning dropouts and non-participants . Not counting people who fail is a wellness industry tradition. Imagine if they counted that way in education. Every school would have a 100% graduation rate. But let’s cut Mr. Goetzel some slack and agree to assume that his 1% to 2% reduction in risk factors is correct.
Let’s apply this assumption to the HERO guidebook chapter that says wellness-sensitive medical events constitute about 5% of total events. (This proportion is higher if you include disease management-sensitive events. This 5% estimate includes only events from conditions people don’t know they might possibly have until they get screened and find out how sick they are, a revelation which will certainly increase their productivity and focus on the job.)
Since hospitalizations comprise roughly 40% of cost, that means 2% of all costs can be addressed with pry, poke and prod programs. Reducing that 2% by the upper bound of Ron’s risk-reduction estimate of 2% yields a grand total of 0.04% reduction in costs.
What is that in dollars? If an average covered person costs an employer $6000, Mr. Goetzel says wellness could save $2.40. This figure excludes the extra doctor visits, drugs and follow-on tests that might be ordered after the initial screen.
So what is the ROI for wellness, assuming Mr. Goetzel’s savings figures are correct? Mr. Goetzel says that a typical wellness program costs $150/employee/year. (That, by the way, is a large multiple of what Quizzify costs, so Quizzify can get immediate savings simply by not being a wellness program.)
And the envelope please…
Using Ron Goetzel’s very own assumptions for both the benefits and the costs, the ROI is: $2.40/$150, or 0.016-to-1. This is not a misprint: for every dollar spent, a company loses almost 99 cents. No wonder, as he says, most programs fail. Yikes! He’s right again. Looks like Mr. Perry better check his clock.
Share this:
If the CDC held the copyright on stupidity, Mercer would have just violated it.
With the possible exception of data fabrication by either HERO or the Koop Award Committee, there is no topic we’ve covered more often than the taxpayer-financed hilarity published by the Centers for Disease Control and Prevention (CDC), whose premise is that laughter is the best medicine.
Specifically:
- The “arresting fact” that 7 out of 10 deaths are caused by chronic disease. That’s called civilization, and 2/3 of the world’s countries would envy us for this statistic.
- Their new mantra that 86% of spending is caused by chronic disease, a big bump up from 75% a few years ago, a figure that apparently wasn’t generating quite enough ridicule for their liking. (Of course the flip side is also true — that bump in chronic disease also means acute disease and birth events are being wiped out.) Yet anyone can see from this readily available government data that admissions coded to chronic diseases (or events directly stemming from chronic diseases) are the exception rather than the rule. The figure that matters is that in the working population perhaps 2% of cost — meaning about 6% of admissions — is due to wellness-sensitive medical events. So in the unlikely event that you can prevent 10% of that, you’ve saved 0.2% of your healthcare spending, before counting costs. If you are spending $500/person/month, that 0.2% would be $1–which curiously squares almost exactly with HERO’s estimate, before they accidentally admitted wellness loses much more money than they had originally thought.
Enter Mercer. One would think that for what they charge they could afford either to (1) hire a smart person to look at their stuff before they publish it, or else (2) connect to the internet. Instead they purchased a photocopier.
Share this:
This Is a Little Embarrassing But…
…Apparently a “Pulse” in which I tell people not to read my books Why Nobody Believes the Numbers, Cracking Health Costs, and Surviving Workplace Wellness is on its way to becoming my most popular. (Judging from our sales figures for Surviving, the marketplace seems to have reached that same conclusion on its own.)
However, it is the case that Unaccountable: What Hospitals Won’t Tell You and How Transparency Can Revolutionize Health Care presents what amounts to the business case for Quizzify, and ultimately Quizzify is how we keep the lights on to write They Said What? so far be it from me to discourage you from buying anything that makes you more likely to sign up for Quizzify.
Share this:
Is Katherine “3.27-to-1” Baicker Coming in from the Dark Side?
In the workplace wellness epidemic, Harvard School of Public Health Professor Katherine Baicker is Patient Zero. However, she may soon come up with the cure for the disease she spread. This would be like when the inventor of the PSA test said it shouldn’t be used.
By way of background, her original dalliance with wellness in Health Affairs–timed and designed to get the Business Roundtable’s “Safeway Amendment” allowing a 30% to 50% clawback of insurance premiums into the Affordable Care Act–introduced the infamous “3.27-to-1 ROI” into wellness vendor vernacular. That was almost seven years ago, but wellness vendors continue to cite this figure as gospel even though RAND demolished it and Professor Baicker herself walked it back four times.
For good reason, as it turns out — this Pulse posting says it all. The 10-to-30-year-old studies comprising her meta-analysis are even more amusing in retrospect. Lots of overcreening, overreliance on now-discredited BMIs, and of course recommendations for low-fat diets–even for employees with metabolic syndrome. Most published in third-tier journals, by the Wellness Ignorati and their friends.
And almost every one of them compared active motivated participants to non-participants. 6 years ago we suspected that design was invalid. Today we know for certain, thanks to Health Fitness Corporation and Aetna. Two more elegant studies to prove that point could not be imagined. No “investigator bias” or “publication bias” there, since both were hoping for the opposite finding.
However, mindful of her reputation, Professor Baicker recently announced (in as many words) that she is going to redeem herself.
Wellness Workdays
Her protestation that she is no longer interested in wellness appears to have taken a backseat to her desire to remove this one blot–granted, a blot of Lady MacBeth proportions–on an otherwise excellent curriculum vitae.
She is going to do a study for BJ’s Wholesale Club, in conjunction with an outfit called Wellness Workdays. Wellness Workdays is a classic wellness vendor. That is to say, they won’t be winning a Nobel Prize anytime soon, or even a spelling bee. Let’s start by examining their analytic and clinical prowess.
To start with, their “White Paper” doesn’t just quote the infamous 3.27-to-1. They’ve upped the ante to 6.00 to 1, maintaining the two significant digits while almost doubling the savings. How? They’ve added the 3.27-to-1 for healthcare savings to the 2.73-to-1 for absenteeism reduction from that same 2010 study. Those two separate conclusions were reached from almost totally different studies. Anyone can tell that from reading the original. Anyone, that is, except Wellness Workdays.
Their analytic qualifications are matched only by their clinical qualifications. One member of their medical advisory board is Chief of Allergy and Clinical Immunology at the Indian River Medical Center in Vero Beach, Florida. While this expertise is not exactly central to the mission of the pry,poke, and prod industry, in all fairness it should be noted that the Indian River Medical Center runs one of the best allergy programs in all of Vero Beach.
Another is an OB-GYN in Colorado. Perhaps this advisor will develop a protocol for employees who want to be screened and induced at the same time. A third consults to orthopedists at “Lennox Hill Hospital,” a role that probably doesn’t require too much heavy lifting, because there is no hospital by that name.
This guy is also an expert on steroids and other performance-enhancing products, and has “published rseveral esearch studies.”
So they can’t spell, can’t proofread, can’t understand study design, and can’t cobble together a qualified advisory board. In other words, to paraphrase the immortal words of those great philosophers Gilbert & Sullivan, they are the very model of a modern clueless wellness vendor.
Katherine Baicker’s proposed study design
We are confident she is going to get it right this time and “discover” that wellness loses money. By selecting a vendor of the caliber of Wellness Workdays, she isn’t leaving anything to chance.
Having learned her lesson — and I’ve been pretty conscientious about forwarding helpful study design materials to guide her — she will certainly tally wellness-sensitive medical events across the entire population. More importantly, here’s what she is not going to do with this study:
- “Match” volunteer participants to — you guessed it — non-participants. This nonsense that keeps the wellness industry afloat, and Professor Baicker no doubt sees right through it;
- List the “unobservable differences” between participants and non-participants as a “limitation.” It’s not a limitation. It’s a baldfaced lie. It is now known that this face-invalid study design is truly invalid. She wouldn’t lie, right? That would damage the reputation of the entire Harvard School of Public Health in what will certainly be a high-profile study;
- Show a high ROI even though the change in health risk factors is trivial. This of course is the stock-in-trade of the Koop Award committee, but a real academic researcher would know better;
- Compare the costs to “trend” and say “costs were projected to rise by [this amount] but they only rose by [this lower amount], and therefore a huge pile of money was saved”;
- Attribute all cost differences to the program, whereas a real researcher would look only at utilization differences that could actually be attributed to urging employees to eat more broccoli. (By contrast, one of the studies in her previous meta-analysis credited the wellness program with a reduction in cat scratch fever.)
How do we know she isn’t going to make up phony outcomes again? She “tipped her hand” with the Oregon Medicaid lottery study. A terrific natural experiment, her study emphasized the value of a “lottery control,” meaning every subject had the same intent-to-treat. Exactly the opposite of the wellness participants-vs-non-participants study design.
Further, who wants to be known as the Typhoid Mary of workplace wellness? The Oregon study was an excellent one, so naturally it showed exactly the opposite of what wellness studies show. Specifically, facilitating access to care doesn’t reduce the cost of care or improve physical health status. And that was for people — newly minted Medicaid recipients — who didn’t have any insurance to begin with. Wellness, of course, takes employees who already have plenty of access to care and drown them in even more, unwanted and largely unneeded, screenings and checkups.
As for BJ’s Wholesale Club, I suspect they got suckered into this. Who volunteers to become the next Pepsico, a case study of how wellness programs fail? BJ’s obviously isn’t studying their own competitors: Target has one of the best programs in the country — precisely because they are far too smart to use a vendor like Wellness Workdays.
In any case, we look forward to her research study. Or perhaps, since this is in conjunction with Wellness Workdays, to rseveral esearch studies.
They Said What? 