In a survey of health system leaders, 92% of organizations report that they evaluate third-party AI tools before deployment. However, the survey found wide variation in validation methods, ranging from formal vendor testing protocols to limited pilot programs.

The survey of 25 health system executives was conducted by the Center for Connected Medicine at UPMC and KLAS Research, which released a report on the results.

The report noted that leaders describe a variety of validation strategies. “Some organizations run formal test cases with vendors and require business sign-off. Others conduct limited pilots, proof-of-concept trials, comparative review against human output (e.g., dual coding), or sandbox testing. Some leaders note that validation can be difficult when the AI is embedded inside a vendor platform or when an organization cannot easily test the solution against the organization’s own data,” it said. 

Only 11 of 25 respondents say their organization has a data platform or other environment in which they can test and validate third-party AI solutions. Some respondents describe robust environments, including production clones, multiple pre-production EHR environments, cloud data warehouses, Microsoft Fabric and Azure, Snowflake, and sandbox workflows. Others say they do not have such an environment or are unsure whether one exists.

The report suggests that many organizations are still working to establish the infrastructure, governance frameworks and strategic foundations needed to support long-term success with AI.
 
"The healthcare industry has moved remarkably quickly from discussing the potential of AI to actively deploying solutions across the enterprise," said Rob Bart, M.D., chief medical information officer at UPMC, in a statement. "What's emerging from this research is a clear recognition that implementation is only the first step. Health systems are now focused on building the governance structures, testing capabilities and organizational strategies necessary to ensure AI delivers meaningful and measurable value.”

UPMC has developed a real-world data platform called Ahavi that can validate and improve third-party AI models before deployment at UPMC. Ahavi leverages de-identified patient data to test AI solutions “in silico” to see their effect on wide-ranging applications without disruption to patient care.

Among other key findings from the survey:

• More than 90% of health systems have deployed third-party AI solutions, demonstrating that AI has moved beyond experimentation and into mainstream use.
• Clinical documentation (52%) is the most commonly cited area for AI deployment, followed by revenue cycle, coding and billing (36%) applications.
• Less than half of respondents (44%) report having a dedicated data platform or environment for testing AI solutions, highlighting potential challenges in evaluating performance, safety and scalability.
• Most organizations’ AI strategies are evolving, with 63% indicating their approach to AI is developing or ad hoc rather than fully established or advanced.
• Leading barriers to AI adoption include limited resources, insufficient time and shortages of specialized talent, reflecting the realities of implementing emerging technologies in complex healthcare environments.
• Respondents reported no strong consensus on the metrics used to measure AI success, underscoring the need for greater alignment around how organizations evaluate return on investment, operational impact and clinical value.