Research conducted at UC San Diego Health on the types of editing that physicians across various specialties made to AI-generated drafts of patient messages provides some early evidence about physicians’ roles as humans-in-the-loop and offers some direction for AI optimization efforts.

In 2025 researchers used a year’s worth of AI-drafted replies to patient messages generated by a system deployed to over 1,000 physicians at UC San Diego Health. They analyzed physician edits to AI-generated draft responses integrated in the Epic EHR. They then created a 15-category taxonomy of edits using a large language model with iterative expert review. Response time was used as a proxy for clinical workload and examined in relation to edit categories with mixed-effects regression, adjusting for message complexity and temporal factors. Their findings were published in NEJM AI. 

The researchers found that all 15 edit categories were associated with longer response times. The most frequent edits involved scheduling or rescheduling appointments (38.5%), followed by providing lifestyle or non-pharmacologic guidance (18.2%) and providing empathy or emotional support (16.1%). The least frequent category was discontinuing or tapering prescription medications (2.4%). 

The largest marginal increases were observed for interpreting radiology results (+70.1%), clarifying or ruling out diagnosis (+63.9%), and interpreting laboratory results (+60.8%).

In the discussion session of their paper, the authors state that “the findings show that editing burden is not uniformly distributed: Edits requiring clinical judgment impose the greatest per-message time burden, while high-frequency administrative edits contributed the greatest cumulative workload across the health system.”

The UC San Diego team notes that to date time savings from these systems have not yet consistently materialized in practice; therefore, understanding where and why physicians modify AI-generated content is a first step to realizing their value for both clinicians and patients.