This is the April 2024 update. The offer is now open until April 15, 2025. In addition to being open to vendors covered in the seminal report highlighted below, it is now also open to employers using those vendors as well as benefits consultants who have recommended them.

Further and most importantly, the offer is now raised to $200,000, having had no takers at $100,000.

It’s not news that diabetes vendors don’t really do anything. The larger the vendor, the less they do. One even earns an “F” from the Better Business Bureau.

Here’s what is news: Virta saves money!  Really.  I’ve been over their numbers up, down and sideways because I was so skeptical. And who can blame me?  Generally, to paraphrase the immortal words of the great philosopher the Queen of Hearts, I can invalidate six impossible diabetes vendor claims before breakfast.*

So when the Validation Institute contacted me to say that Virta wanted me on a validation call, I said: “No, they don’t.”  They dutifully reported back to the Virta folks, who said: “No, really.”

Which VI dutifully reported back to me. I replied: “Fine.”

“Fine” is one of those words whose meaning depends on the intonation. It could mean better-than-good, like in numismatics. Whereas in a marital argument, “Fine” means: “I know I’m right, but I just don’t feel like getting into it.”

My intonation meant: “Sure, if Virta wants to have their validation request eviscerated, I’ll join the call.”

However, I am pleased to report that “fine” in this case really did mean “better-than-good,” as in: “Virta has the most valid and impressive outcomes in the diabetes field.”

Most vendor claims incorporate some or all of the following fallacies: regression to the mean, participation bias, or “savings vs. trend.” Virta had none of those.

Quite the opposite. Virta did two studies, both using methodologies that met the highest level of VI validity. As a result, in addition to the $50,000 Credibility Guarantee offered by the VI in support of their results, I am offering a $100,000 personal guarantee (note: they pay for this – terms on request) in support of the following two statements:

1. Virta Health has proven it can deliver more cost savings (measured PMPM) than any other digital health point solution commercially available today, using a valid measurement methodology. This holds true regardless of condition category, including but not limited to: diabetes management, diabetes prevention, weight loss, mental health, musculoskeletal disorders, heart health, substance abuse, women’s health, fertility care, and cancer care.
2. No diabetes solution featured in the Peterson Health Technology Institute (PHTI) report (which includes all the major vendors) —other than Virta Health—has demonstrated positive net financial savings in-year, using a valid methodology.

Whoever can disapprove either statement to the judges (described as selected below) will receive $100,000.

Terms and conditions are listed at the bottom of this blog.

While it’s a layup to bet the farm to challenge vendor claims  due to their inherently sketchy nature, my standards for putting my own money at risk in support of a vendor’s claims are extremely high. Quizzify2Go (ER visit cost reduction) and Sera Prognostics (prediction/prevention of premature birth) are the only entities I’d risk my hard-earned dollars on, and neither is disease-focused. Virta is and will likely remain the lead dog in savings amongst all disease-focused vendors.

The vendors eligible for this reward are any who are featured in the PHTI report.

Here are the Virta studies that gave me the confidence to offer this challenge with my own money at stake:.

1. Parallel Assignment: Indiana University

A parallel study is one where would-be participants are randomly assigned to control or the study group. The randomized control trial (RCT) is one such methodology.  In the case of drugs, the control group gets a placebo, so they don’t know whether they are getting the drug or not. This is called a “blinded” study. In many drug studies, even the investigators don’t know. This is called “double-blinded.”

Neither is possible in population health because you would know whether you are in a wellness/diabetes program or not. So studies must be unblinded.

Even unblinded RCTs are rarely undertaken in population health because (in addition to employers not hitherto having access to claims data) such studies need Institutional Review Board approval as an investigation before proceeding, as ERISA plans are otherwise required to offer the same benefits to every employee. (One easy way around this is to offer the intervention to all comers, but promote actively to some worksites but not others.)

Virta minimized that threat at Indiana University, because the parallel assignment took place in different sites, to minimize the chance that (though the consent included the possibility of being in the control group) one diabetic employee might demand the intervention that the others are getting, once they see how helpful it is. Take a look at some of the results for those completing two years with Virta as compared to the parallel control, with both arms experiencing a similar lost-to-followup:

• HbA1c was reduced by 0.9 points on average
• Weight loss averaged more than 10%
• Prescription medications were cut in half, including an 81% reduction in daily dose of insulin
• More than $3000 savings in prescription drug cost reduction between years 1 and 2

2. Wait-List (Lottery) Control

Another valid design, also used by Virta, is a Wait-List, or Lottery, Control.

The most famous natural experiment in population health using this control is the Oregon Medicaid study. Medicaid was expanded there to a higher income level, but slots were limited. People who wanted coverage had to enter a lottery. Medicaid eligibility was assessed only after names were drawn – so only for the lottery winners who completed the Medicaid application forms.  That’s one of the reasons it was so important to assess effects of insurance by comparing the entire control group to the entire treatment group, rather than the subset of the treatment group deemed eligible or actually enrolling.

The researchers still assessed the effect of insurance coverage itself (not just winning the lottery) by using instrumental variables estimation, but relying on only the variation of lottery selection to identify those causal effects. (The two-year finding was that being covered by Medicaid as opposed to being uninsured didn’t appreciably change physical health status, but did quite dramatically reduce both depression and financial strain.)

In Virta’s case, the Veterans Health Administration (VHA) signed on as a client, but with a limited budget that could not accommodate all who qualified and wanted to participate. Therefore, those who were wait-listed became the natural control group. The VHA, whatever its other controversies, excels at data collection amongst veterans who stay within its system, and was able to compare the results of the actual participants to the would-be participants.

Virta’s approach delivered significant reductions in HbA1c (-0.69 points) and reduced diabetes medications fill by 34.5%. BMI, blood pressure, and even the number of outpatient visits were all reduced. Read more at the full study.

Terms and Conditions of Challenge

Selection of Judges

There will be five judges, selected as follows:

• Each side gets to appoint one, drawn from Brian Klepper’s listserve with almost 1000 people on it, from all walks of healthcare.
• Two others are appointed objectively. That will be whichever health services researchers/health economists are the most influential at the time the reward is claimed. “Most influential” will be measured by a formula: the highest ratio of Twitter followers/Twitter following, with a minimum of 15,000 followers.
• Those four judges will agree on the fifth.

Using the criteria below, judging will be based on validity of the measurement. Measurements deemed invalid, such as those described on the Validation Institute site, is a disqualifying factor, i.e., any challenge by a vendor that is not validated by the Validation Institute.

If the challenging party/vendor is deemed by the judges to have an equally valid metric as Client, the decision is made on the impact of the program in drug use reduction.

Written submissions

Each side submits up to 2,000 words and five graphs, supported by as many as 20 links; the material linked must pre-date this posting to discourage either side from creating linked material specifically for this contest.

Publicly available materials from the lay media or blogs may be used, as well as from any of the 10 academic journals with the highest “impact factors,” such as Health Affairs, published within the last five years.

Each party may separately cite previous invalidating mistakes made by the other party that might speak to the credibility of the other party. (There is no limit on those.)

Oral arguments

The judges may rule solely on the basis of the written submissions. If not, the parties will convene online for a 2.5-hour virtual presentation featuring 10-minute opening statements, in which as many as 10 slides are allowed. Time limits are:

• 30-minute cross-examinations with follow-up questions and no limitations on subject matter;
• 60 minutes in which judges control the agenda and may ask questions of either party based on either the oral or the written submissions;
• Five-minute closing statements.

Entry process

The entry process is:

1. Challenger and Service Provider deposit into escrow the amount each is at risk for ($200k for the Challenger, and $200k to the Service Provider). Each party forwards $10,000 to the judges as well, as an estimate of their combined fees and/or contributions to their designated nonprofits.
2. If the Challenger or Service Provider pulls out after publicly announcing an application, the fee is three times the amount deposited.
3. The escrow is distributed to the winner and the judges’ fees paid by the winner are returned by the judges to the winner, while the judges keep the losers’ fees. (This challenger fee goes to the judges.)

Other

The competition is open to any wellness, diabetes or disease management vendor outcomes claim by any vendor, consultant or employer related to a vendor in the PHTI report made before April 15, 2025. This date may be updated by Service Provider from time to time.

*Alice laughed: “There’s no use trying,” she said. “One can’t believe impossible things.”

“I daresay you haven’t had much practice,” said the Queen. “When I was younger, I always did it for half an hour a day. Why, sometimes I’ve believed as many as six impossible things before breakfast.”