Home » Posts tagged 'Validation Institute'
Tag Archives: Validation Institute
Questions to ask Livongo before signing with them
In 2014, the Validation Institute, then a joint property of Intel and GE, tasked me, based on Why Nobody Believes the Numbers, to develop the most sophisticated outcomes measurement course ever devised. That was not a heavy lift, because at the time no other such course existed. They named it Advanced Critical Outcomes Report Analysis, or “Advanced CORA” for short.
Five years later, the course, now called CORA Pro, remains the Gold Standard in outcomes measurement expertise. CORA Pros — and there are only 8 of us — have developed a talent not just for sniffing out questionable claims (anyone can “challenge the data”), but more importantly for identifying contradictory, implausible and even impossible claims that somehow hadn’t been noticed before. Even a simple press release can become a case study in impossibility.
CORA certification itself is very useful, and takes you halfway there. But there’s something special about CORA Pro. Once you are certified in CORA Pro, you never look at a vendor outcomes report the same way again. For instance, consider Livongo’s recent glowing press release about its outcomes. Non-CORA Pros, like whoever these people are, basically just parrot a vendor’s results uncritically with a glowing headline. They sound like Flounder.
By contrast, a CORA Pro would review Livongo’s claims thoughtfully, and ask probing questions, first about the outcomes:
- If your goal is to reduce admissions for diabetes and reduce insulin usage across the population, why didn’t you measure admissions for diabetes, or insulin usage?
- Why does this report show the opposite of your report from 2018? This report showed huge reductions in outpatient and no significant change in inpatient, whereas the 2018 report showed huge reductions in inpatient with no significant change in outpatient?
Then a CORA Pro would ask whether Livongo may be harming employees:
- Why is your goal Hb A1c of 6.5% in conflict with the American College of Physicians and other expert bodies not funded by the diabetes industry, which advocate 7.0% to 8.0%? Perhaps they mean reaching 6.5% with diet-and-exercise, but if insulin use is going up, which is likely because they didn’t claim it was declining, clearly some employees aren’t getting the subtle nuance that they shouldn’t aim for 6.5% with medication.
- Why does Livongo push employees to check glucose multiple times a day when Choosing Wisely says most should do the opposite, to prevent harm?
Just askin’… Because that’s what CORA Pros do.
The full list of questions can be found here.
To learn about CORA certification, click here.
Share this:
A vendor’s guide to snookering self-insured employers
Dear Wellness, Diabetes, Clinic, Price Transparency, and Medication Therapy Management Vendors,
While most of you already know the majority of these tricks, there might be a few you haven’t deployed yet. So take good notes.
Sincerely,
Al Lewis
PS If you are an employer, just pass this along to your vendors…and watch your savings skyrocket. Or use “An Employer’s Guide to NOT being snookered” to see your savings become realistic.
Best practices for every vendor
Compare participants to non-participants. Using non-participants as a control for participants allows you to show massive savings without doing anything. This is not an overstatement. Here is a program — which naturally won an award for its brilliance from Ron Goetzel and his friends before I observed that they were a fraud according to their own data– that did just that. They separated participants from non-participants but didn’t bother to implement a program for two years—by which point the participants had already improved by 20% vs. the non-participants — without even having a program to participate in. (Note on this slide that the control and study group were set up in 2004 but the program didn’t start until 2006, when the cost separation had already reached the aforementioned 20%.)
Two other observational trials support this conclusion. Most recently, the National Bureau of Economic Research ran a controlled trial to test exactly this hypothesis. Sure enough, like the three observational trials, they found that virtually the entire outcome in wellness can be explained by that popular study design itself, rather than the intervention.
In any participation-based program, ignore dropouts. Assume that employees who drop out do so randomly, not because they are discouraged by their lack of progress or interest.
Draw a line upwards and then claim credit for the “savings” between the actual upward spending and the “trend” you drew. As Optum’s Seth Serxner stated so succinctly: “We can conclude that the choice of trend has a large impact on estimates of financial savings.”
Start with the ridiculously high utilizers, high-risk people, or people taking lots of drugs. Let the group regress to the mean, and then claim that as savings.
Never admit, like Wellsteps did, that you are familiar with regression to the mean, since most employers are not aware of it. The higher the costs/risks of the original users, the more savings you can claim. Here are two verbatim claims:
- A heavy equipment manufacturer found high use of the ER was a becoming a cost concern, so it send mailings that showed appropriate care settings to the homes of members with two or more visits to the ER in the past year. As a result, ER visits were down 59 percent those who got the mailing.
- A pharmaceutical company saw a spike in ER claims was coming from repeated use by the same people, so two mailers were sent: one to households with one ER visit in the past year; another for those with two or more visits. Following the mailings, there was a 63 percent drop in ER visits.
Pretend not to notice that low utilizers can show an increase in utilization — or especially that low-risk people can increase in risk. Focus the mark (I mean, the customer) on the high-risk people who decline in risk. Never draw graphs to scale, or your customer might notice that 2/3 of their employees are low-risk in the first place.
It doesn’t matter what your intervention is. Claim credit for the entire difference in trend. For instance, in this example, Community Care of North Carolina claimed credit for a huge reduction in PMPM costs for babies for their medical home program…but babies weren’t even included in the program. (Neonatal expenses didn’t decline either.)
Or do what Safeway did, launching the wellness craze: change to a high-deductible plan, and transfer a large chunk of costs to employees. Don’t even bother to institute a wellness program, but attribute all the savings (from the transferred deductible spending) to wellness anyway, so that you get invited to the White House. And after that blows up on you, demonstrate that your very stable genius investment in wellness was not a fluke by investing your company’s money in Theranos.
Special Instructions for transparency tool vendors
Assume that every employee who uses your tool is looking to save their bosses some money, rather than (for instance) to find the closest MRI…and that none of them would have used a lower-cost venue absent your tool.
If only 10% of employees use your transparency tool, and only 10% of events are shoppable, nonetheless take credit for the entire difference in trend across the board, and ignore the literature showing online price-comparison tools don’t work.
If people who haven’t met their deductible shop more than people who have, attribute the former’s lower cost to use of the tool, rather than to the fact that by definition people who don’t meet their deductible spend less than people who blow through it.
Special instructions for wellness and diabetes vendors
If you are a wellness or diabetes prevention/management vendor, never ever let employers know that every year since statistics have been kept, fewer than 1 in 1000 employees/dependents end up in the hospital with diabetes. (And another 1 in 1000 with a heart attack.) Always tell them how many employees are at risk and how many “newly discovered conditions” they have, and how they will all end up in the hospital, even though hospitalizations for heart attacks and diabetes in the employer-insured population have been declining for years.
Wellness vendors should always put the trivial percentage reduction in risk (for participants only, of course – and ignoring dropouts) on one page and the massive savings on another page. Most employers won’t bother to do the math to notice, for example, that Interactive Health claimed $50,000 in savings for every employee who reduced one risk factor, while the state of Nebraska won an award for claiming to save $20,000+ for every risk factor reduced, as did Staywell for British Petroleum.
If you didn’t reduce risk factors, present your outcomes in a format no one can make heads or tails of, like this one, from Wellsteps. If Wellsteps was able to snooker an entire committee of self-anointed outcomes experts to win an award for program excellence, surely you can snooker a few customers.
Claiming people lose weight is a big part of your outcome reporting, so make sure to do the following:
- Never count nonparticipants, and ignore dropouts.
- Don’t do any long-term follow-up to see who regained the weight (most participants)
- Give them time to binge before the initial weigh-in
Special instructions for diabetes vendors
In addition to measuring on active participants only, raise the bar for Hb A1c so that only people with high Hb A1c’s can be included. That belt-and-suspenders approach will ensure that you can’t fail to show savings, even if (as is likely the case) you don’t change anyone’s behavior other than the employees who were going to change anyway, which you might as well count.
Next — most diabetes vendors and a few wellness vendors have already figured this out — you can charge much more if you can submit claims, rather than just be an admin expense line item. You see, most employers focus much more on the 10% admin expense than they do the 90% medical expense, which they consider to be beyond their control. Your claims expense – which would draw attention to itself as an admin cost — won’t get noticed in the 90% of medical losses, sort of like the dirt from the tunnel sprinkled around the Stalag in The Great Escape.
Special instructions for medication therapy management vendors
Only mention “gaps in care” that you close, not the ones that open up. And, as noted in the chart below, always use percentages. So in this chart (provided by one of the major PBMs), they claimed that twice as many gaps were closed (37%) vs opened (18%), and yet, as is almost always the case with MTM vendors, nothing happened to the total number of gaps, which remained at exactly 820:
Tally all the employees who were on large numbers of meds and now take fewer. But don’t mention all the employers who were on fewer meds and now take more.
What to do if you’re asked why you aren’t validated by the Validation Institute
Here are the most popular answers to that question:
- No one has asked us to. (Quizzify didn’t need to be asked.)
- We hired our own outside actuarial firm to validate us, and they concluded we save a lot of money.
- Sure, we’ll get validated as soon as you sign the contract with us.
Share this:
Four More Vendors Caught…Doing Something Right
The popular perception is that They Said What tries to catch vendors doing something wrong. Nope – they generally self-immolate and we just take screenshots. Or as we say, in this industry, “you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.”
Yet no matter how screamingly obvious the data, a journalist, essayist, or blogger loses credibility if they always say the same thing. That’s why we try very hard to catch vendors doing something right.
For us to do that, vendors have to give us the opportunity – by actually doing something right. To avoid being judgmental, we like to see them independently validated for, in as many words, “doing something right.” Validated not just by anyone, but by the Care Innovations Validation Institute. Care Innovations, a wholly owned subsidiary of Intel Corporation, launched the Validation Institute in 2014 to provide companies with 3rd party validation of their outcome claims. (Disclosure: while I am neither a Validation Institute employee nor an advisor, my book Why Nobody Believes the Numbers provides the methodological basis for some of their validations, and they sometimes retain me as an outside expert/validator.)
The Validation Institute is the Gold Standard of validation. Everything is vetted carefully and has never been challenged. Validation can be done two ways: on the basis of valid contractual representations or on the basis of actual outcomes. The vast majority of validated organizations have the former, because their outcomes to date are insufficient for the latter. An example of contractual language validation would be Quizzify’s savings guarantee. (Disclosure: I founded Quizzify. The validation for Quizzify was obviously conducted by another of the Validation Institute’s team of 3rd party, independent validators).
While many companies guarantee or show savings, it turns out the language used in that guarantee or demonstration of savings determines whether savings can quite literally happen on their own due to faulty study design or whether they truly reflect underlying improvements. Quizzify, for example, couldn’t get outcomes validation because it hasn’t been around long enough to apply these valid contractual representations/guarantees to its own outcomes.
By contrast, the four organizations below are among the few whose validation is specifically outcomes-based – meaning these companies took the next step and what they say they did, is what they actually did.
Even so, if you read the validation language carefully, you’ll see it never exceeds what the outcomes data allows.
Alphabetically, we look at each of these four in some detail, describing the outcomes that were achieved. Just to reiterate, these four companies are among the very few in population health that can lay claim to outcomes improvement, measured validly. Why? Because any company that could get Validation Institute validation, would. (Quizzify sought it as soon as we had language that could be validated…and are very pleased with the attention it has brought us and the doors it has opened.)
Evolent Health: Focused on the Most “Impactable” Patients
Evolent is the first value-based care company with a complex care management program to show savings. Typically, companies compare the “pre” cost to the “post” cost, but anytime you target a chronic group that is high-need, high-cost, the “post” will always look better compared to the “pre.” Statisticians dryly call this regression to the mean and many vendors claim credit for the decline in cost when it had nothing to do with their interventions.
Instead, Evolent showed savings the hard way – by actually achieving them. They measured how much the cost of high-risk chronic patients declined on their own and then only took credit for the additional reduction. Suppose you have a magic potion to flip 100 coins from heads to tails. If only 50 flip, your potion is worthless; the probability of landing on tails is already 50-50. If 60 coins flip, the Validation Institute would give you credit for 10. Evolent showed that their program lowered utilization and costs beyond the reductions that would have happened anyway.
In a field known for the time lapses between a patient’s need for care management and its delivery, Evolent’s more advanced predictive modeling (covering more datasets than a carrier would typically use) expeditiously determine those at highest risk of having an “impactable” event. Further, whereas most such programs focus on just checking off boxes, Evolent intervenes across the spectrum of clinical, behavioral, social, nutritional and environmental domains.
Having reviewed many of these programs, I’ve been shocked by how long it takes them to find and enroll patients, how little they do for the patient, and how little they know about what they’re doing. Evolent is the opposite. This is their business, not a sideline – they take it seriously and it shows.
Healthways Well-Being: It’s Not Just about the Cholesterol
For the uninitiated, the philosophy of well-being is to address gaps not just in employee physical health but – as importantly – in their emotional, financial, occupational and social health. In many cases, those latter issues are the root cause of high healthcare spending and low productivity. Addressing those issues should help a given population – from the healthy to the sick – perform noticeably better while possibly spending less on healthcare.
Before you even heard of “well-being,” Healthways was measuring it, more than a decade ago. Since 2008, Healthways has partnered with Gallup to definitively measure well-being via the Gallup-Healthways Well-Being Index, the most proven, seasoned and comprehensive measure of well-being in populations in the world. Quite literally, if there is any component of this industry which has penetrated the public consciousness in a positive way, it’s the Gallup-Healthways Well-Being Index, whose publication often reaches the lay media.
Healthways can use surveys, down to the community level, to benchmark similar surveys for companies, departments and employees, so that organizations can focus their improvement efforts where they are needed most. The Validation Institute has confirmed Healthways’ findings that in fact performance (holding constant as many other variables as possible) correlates far more closely with indicators of well-being than with biometrics alone. This data collection, insight and benchmarking allows targeted interventions to complement or replace conventional wellness…and get closer to the root cause of underperformance.
Rarely is the root cause of poor health “I-don’t-care-itis,” as one wellness vendor calls it. Often it’s a different personal issue. Sometimes the root cause is department-specific. This data can be used to identify managerial or process flow issues far beyond the scope of – and far more powerfully than — conventional wellness.
Quantum Health: the Story Tells Itself
While I make more general comments about the other vendors on this list, I don’t need to for Quantum Health. They were the first and are still the only company validated for total savings across an entire organization.
Instead I will share a story that shows how their incentives for members to call in – combined with their non-siloed approach to those calls – create a confluence of time and place that change behaviors and likely outcomes.
Once, when I visited them, an employee of a new customer called, asking if diabetic shoes were a covered benefit. In most, if not all, carriers, the employee answering that query would be evaluated based on accuracy of the answer, number of rings, politeness and how many calls they handled that hour. So the person would say “yes” or “no” and then get off the phone. At Quantum Health, the agent answered the query but was prompted by the supporting software (and by training) to recognize that question as a red flag. Here was an employee whose diabetes was already so advanced he was asking about shoes…and yet he was nowhere in the diabetes registry. A typical carrier wouldn’t find out about this person until after the inpatient claim for his inevitable crash was filed, warehoused, prioritized and queued for telephonic outreach. And then, assuming the carrier had the correct phone number, and this patient answered the call and was receptive, rehabilitation could begin.
And yet there he was – right on the phone – asking for help. So the agent probed a little further and then transferred him to a nurse in the same pod, who engaged him right away, almost certainly avoiding or forestalling a future high-cost medical event.
US Preventive Medicine: Finding the Formula
The editor of the American Journal of Health Promotion, Michael O’Donnell, famously admitted that up to 95% of wellness programs fail. U.S. Preventive Medicine is squarely in his other 5%. As quite literally the purest wellness program validated by the Validation Institute, USPM has – alone in the wellness industry – found the formula for a significant and sustained reduction of wellness-sensitive medical events (hospitalizations and ER visits).
The Validation Institute analysis showed that USPM generated a sustained average 41% reduction of hospitalizations and ER visits across several chronic conditions (Diabetes, Asthma, Coronary Artery Disease, Hypertension, Chronic Obstructive Pulmonary Disease and Congestive Heart Failure) over a four-year timeframe, significantly outperforming the averages as tallied by the Healthcare Cost and Utilization Project (HCUP). USPM provides a unique data-driven, high-tech and high-touch combination of conventional and unconventional interventions to enhance engagement and translate that engagement into actual behavior change.
The Validation Institute has publicly urged all wellness vendors to collect real data, apply their value event rate-based template (the only methodology that They Said What and HERO agree on, as also described in Health Affairs), to see if they can match USPM’s performance…and so far, none have come close.
Michael O’Donnell might have been optimistic in his assessment—the failure rate seems much higher. But it’s not 100% — USPM is the exception that proves wellness can indeed be done successfully…if all the components fit together.
Share this:
We Caught a Wellness Vendor Doing Something Right: US Preventive Medicine
For a year now, we’ve been outing wellness vendors whose endless stream of rookie mistakes in outcomes calculations (that somehow always seem to overstate savings) has provided a correspondingly endless stream of mirth and merriment to our expanding cadre of visitors, whose numbers now exceed 50,000 in total.
During this period, like Diogenes, we’ve been searching for an honest, competent wellness vendor, one that we could highlight to show that we are not simply mean-spirited anti-wellnites who for some unknown reason were intent on denying employees the opportunity to become healthy.
Diogenes had it easy compared to us.
Quite literally we have sifted through hundreds of wellness vendors. Some are just a few fries short of a Happy Meal. They get highlighted in our popular On the Even Lighter Side compilation. Some have very squirelly data. They win Golden Squirrel Awards. Others, well, we’re not calling anybody scoundrels but let’s just say they could never be confused with Mother Theresa. They get Smoking Guns, featuring questions that buyers can ask them based on their own claims that can’t be answered.
None of these vendors or any other wellness vendor, it is worth noting, have managed to receive validation from the Intel-GE Care Innovations Validation Institute, despite the obvious advantages of a respected third-party’s endorsement…and the offer by They Said What? to remove all unflattering references to them if they achieved such validation, as many non-wellness companies have,
Finally, however, our quest for an honest and competent wellness vendor is rewarded. US Preventive Medicine has both achieved validation for its contractual language and has compiled an enviable record–across its entire book of business–of event reduction. The full press release can be viewed here.
Further, they measure risk factors on the entire population, which is valid, not just the highs-to-lows.
We applaud their willingness to attempt validation but especially the results they’ve achieved and the contracts that codify those results, assuring that their customers will also receive validation should they choose to apply for it.
They Said What? 