There is an old joke: “How can you tell if a vendor is lying about ROI?”
Answer: “They’re claiming an ROI.”
That’s not entirely accurate. Some vendors really do achieve savings. This is particularly true with vendors that just reduce the cost of something you need to buy anyway, like Quizzify for ER visits, where in most states 50% reductions in ER Bills are routine for employees using Quizzify2Go. Or Diathrive for diabetes supplies. Or any number of vendors for drugs, my personal choice being Drexi/AMPS, home of the $2.76 90-pill Ambien supply.
But, for behavior-change vendors, you need to be the judge to translate vendorspeak into English. Examples:
- “We retained independent actuaries to validate our savings” translates as “We paid off some actuaries to fabricate our savings.”
- “We reduced the risk level for many of the highest-risk employees” omits “…but they would have come down anyway due to regression to the mean.”
- “We achieved tremendous savings vs. trend” needs the asterisk: “because how we choose the trend determines the savings.” (This one, by the way, was a real quote from a well-known consultant.)
- “We compared participants to matched non-participants, and found that…” Hard stop. Invalid.
Fortunately, in the next 3 weeks, you’ll have four opportunities to learn how to distinguish valid from invalid measurement.
On August 31 at 2 PM EDT, I’ll be joining Virta Health on a webinar, kicking off the discussion with this very topic: Outcomes Measurement for Dummies…and Smarties. And you can learn why Virta is so valid that I am putting up $100,000 as a “Challenge” for anyone who thinks another vendor’s outcomes are better. Sign up here.
If you really want to dig deep, join Health Benefits Nation in Orlando, hosted by The Validation Institute. This September 14-16 conference in Orlando covers the gamut, but specifically, there is a two hour session on valid measurement starting at 1 PM on the 14th.
Besides digging deep, the Validation Institute is my forum for, uh, naming appellations and kicking posteriors. And of course, I invite people to sue me if they don’t like what I have to say about them.
We will also have case studies where you need to spot the lies, rather than have me tee the lies up for you. Because that’s what happens in the real world. Lies told by “independent actuaries” aren’t going to invalidate themselves.
Actually, they usually do, but only if you look hard enough…and that’s what this session is all about.
One more vendor is about to join the pantheon of validity, Medencentive. I’ve been over their numbers up, down and sideways, as have many others. They are a health literacy vendor. Not like Quizzify (though they are a Quizzify customer) because they go deep on literacy with people who have specific conditions already, whereas Quizzify is educating on all the points where employees and healthcare meet, as the four examples on the homepage show. They will be hosting a webinar on September 13 at noon EDT. You can sign up here.
Finally, if anyone is anywhere near Houston on the 21st of September and is an early riser, register for The Healthcare Digital Dilemma, where I and several of the leading lights of the field — Josh Berlin, David Carmouche and others — to discuss this “dilemma.” You want to do more digital health…and yet you know most of it doesn’t work. We’ll be discussing how to find the solutions that do.
If you were one of the almost 8000 people to read the post Virta Health becomes the first diabetes vendor to save money, you may be wondering what exactly they do that no other diabetes vendor is able to do, to get me to put $100,000 at risk for the first time for a vendor other than Quizzify.
Part of the reason is that Quizzify and Virta are the peanut butter-and-chocolate of diabetes.
Virta reverses diabetes, while Quizzify helps pre-diabetics and diabetics avoid hidden sugars. That’s no easy task because 74% of processed foods contain them. This even includes some products that claim to contain “NO ADDED SUGAR,” but in fact are bursting with added sugars. So that I want to encourage our clients to adopt Virta, and vice-versa.
The other part of the reason is that their savings are measured validly. While plenty of vendors claim to save money and some vendors measure validly, very few are in the intersection set.
Virta and I and the Validation Institute are doing a webinar on valid measurement generally, and you can attend that webinar both to learn about valid measurement and to learn about Virta’s own results using valid measurement.
It’s not news that diabetes vendors don’t really do anything. The larger the vendor, the less they do. One even earns an “F” from the Better Business Bureau.
Here’s what is news: Virta saves money! Really. I’ve been over their numbers up, down and sideways because I was so skeptical. And who can blame me? Generally, to paraphrase the immortal words of the great philosopher the Queen of Hearts, I can invalidate six impossible diabetes vendor claims before breakfast.*
So when the Validation Institute contacted me to say that Virta wanted me on a validation call, I said: “No, they don’t.” They dutifully reported back to the Virta folks, who said: “No, really.”
Which VI dutifully reported back to me. I replied: “Fine.”
“Fine” is one of those words whose meaning depends on the intonation. It could mean better-than-good, like in numismatics. Whereas in a marital argument, “Fine” means: “I know I’m right, but I just don’t feel like getting into it.”
My intonation meant: “Sure, if Virta wants to have their validation request eviscerated, I’ll join the call.”
However, I am pleased to report that “fine” in this case really did mean “better-than-good,” as in: “Virta has the most valid and impressive outcomes in the diabetes field.”
Quite the opposite. Virta did two studies, both using methodologies that met the highest level of VI validity. As a result, in addition to the $50,000 Credibility Guarantee offered by the VI in support of their results, I am offering a $100,000 personal guarantee in support of the following two statements:
- Virta Health has proven it can deliver more cost savings (measured PMPM) than any other digital health point solution commercially available today, using a valid measurement methodology. This holds true regardless of condition category, including but not limited to: diabetes management, diabetes prevention, weight loss, mental health, musculoskeletal disorders, heart health, substance abuse, women’s health, fertility care, and cancer care.
- No diabetes solution—other than Virta Health—has demonstrated positive net financial savings in-year, using a valid methodology.
Whoever can disapprove either statement to the judges (described as selected below) will receive $100,000.
Terms and conditions are listed at the bottom of this blog.
While it’s a layup to bet the farm to challenge vendor claims due to their inherentlly sketchy nature, my standards for putting my own money at risk in support of a vendor’s claims are extremely high. Quizzify2Go (ER visit cost reduction) and Sera Prognostics (prediction/prevention of premature birth) are the only entities I’d risk my hard-earned dollars on, and neither is disease-focused. Virta is and will likely remain the lead dog in savings amongst all disease-focused vendors.
Here are the Virta studies that gave me the confidence to offer this challenge with my own money at stake:.
- Parallel Assignment: Indiana University
A parallel study is one where would-be participants are randomly assigned to control or the study group. The randomized control trial (RCT) is one such methodology. In the case of drugs, the control group gets a placebo, so they don’t know whether they are getting the drug or not. This is called a “blinded” study. In many drug studies, even the investigators don’t know. This is called “double-blinded.”
Neither is possible in population health because you would know whether you are in a wellness/diabetes program or not. So studies must be unblinded.
Even unblinded RCTs are rarely undertaken in population health because (in addition to employers not hitherto having access to claims data) such studies need Institutional Review Board approval as an investigation before proceeding, as ERISA plans are otherwise required to offer the same benefits to every employee. (One easy way around this is to offer the intervention to all comers, but promote actively to some worksites but not others.)
Virta minimized that threat at Indiana University, because the parallel assignment took place in different sites, to minimize the chance that (though the consent included the possibility of being in the control group) one diabetic employee might demand the intervention that the others are getting, once they see how helpful it is. Take a look at some of the results for those completing two years with Virta as compared to the parallel control, with both arms experiencing a similar lost-to-followup:
- HbA1c was reduced by 0.9 points on average
- Weight loss averaged more than 10%
- Prescription medications were cut in half, including an 81% reduction in daily dose of insulin
- More than $3000 savings in prescription drug cost reduction between years 1 and 2
- Wait-List (Lottery) Control
Another valid design, also used by Virta, is a Wait-List, or Lottery, Control.
The most famous natural experiment in population health using this control is the Oregon Medicaid study. Medicaid was expanded there to a higher income level, but slots were limited. People who wanted coverage had to enter a lottery. Medicaid eligibility was assessed only after names were drawn – so only for the lottery winners who completed the Medicaid application forms. That’s one of the reasons it was so important to assess effects of insurance by comparing the entire control group to the entire treatment group, rather than the subset of the treatment group deemed eligible or actually enrolling.
The researchers still assessed the effect of insurance coverage itself (not just winning the lottery) by using instrumental variables estimation, but relying on only the variation of lottery selection to identify those causal effects. (The two-year finding was that being covered by Medicaid as opposed to being uninsured didn’t appreciably change physical health status, but did quite dramatically reduce both depression and financial strain.)
In Virta’s case, the Veterans Health Administration (VHA) signed on as a client, but with a limited budget that could not accommodate all who qualified and wanted to participate. Therefore, those who were wait-listed became the natural control group. The VHA, whatever its other controversies, excels at data collection amongst veterans who stay within its system, and was able to compare the results of the actual participants to the would-be participants.
Virta’s approach delivered significant reductions in HbA1c (-0.69 points) and reduced diabetes medications fill by 34.5%. BMI, blood pressure, and even the number of outpatient visits were all reduced. Read more at the full study.
Terms and Conditions of Challenge
Selection of Judges
There will be five judges, selected as follows:
- Each side gets to appoint one, drawn from Brian Klepper’s listserve with almost 1000 people on it, from all walks of healthcare.
- Two others are appointed objectively. That will be whichever health services researchers/health economists are the most influential at the time the reward is claimed. “Most influential” will be measured by a formula: the highest ratio of Twitter followers/Twitter following, with a minimum of 15,000 followers.
- Those four judges will agree on the fifth.
Using the criteria below, judging will be based on validity of the measurement. Measurements deemed invalid, such as those described on the Validation Institute site, is a disqualifying factor, i.e., any challenge by a vendor that is not validated by the Validation Institute.
If the challenging party/vendor is deemed by the judges to have an equally valid metric as Client, the decision is made on the impact of the program in drug use reduction.
Each side submits up to 2,000 words and five graphs, supported by as many as 20 links; the material linked must pre-date this posting to discourage either side from creating linked material specifically for this contest.
Publicly available materials from the lay media or blogs may be used, as well as from any of the 10 academic journals with the highest “impact factors,” such as Health Affairs, published within the last five years.
Each party may separately cite previous invalidating mistakes made by the other party that might speak to the credibility of the other party. (There is no limit on those.)
The judges may rule solely on the basis of the written submissions. If not, the parties will convene online for a 2.5-hour virtual presentation featuring 10-minute opening statements, in which as many as 10 slides are allowed. Time limits are:
- 30-minute cross-examinations with follow-up questions and no limitations on subject matter;
- 60 minutes in which judges control the agenda and may ask questions of either party based on either the oral or the written submissions;
- Five-minute closing statements.
The entry process is:
- Challenger and Service Provider deposit into escrow the amount each is at risk for ($100k for the Challenger, and $100k to the Service Provider). Each party forwards $10,000 to the judges as well, as an estimate of their combined fees and/or contributions to their designated nonprofits.
- If the Challenger or Service Provider pulls out after publicly announcing an application, the fee is three times the amount deposited.
- The escrow is distributed to the winner and the judges’ fees paid by the winner are returned by the judges to the winner, while the judges keep the losers’ fees. (This challenger fee goes to the judges.)
The competition is open to any wellness, diabetes or disease management vendor outcomes claim made before April 15, 2023. This date may be updated by Service Provider from time to time.
*Alice laughed: “There’s no use trying,” she said. “One can’t believe impossible things.”
“I daresay you haven’t had much practice,” said the Queen. “When I was younger, I always did it for half an hour a day. Why, sometimes I’ve believed as many as six impossible things before breakfast.”