Assessing the Clinical Safety of an AI Assistant in Indonesian Chat-Based Telemedicine

Background: Clinical AI assistants are entering routine telemedicine workflows, but evidence on deployed systems remains limited outside high-income settings.

Objective: To estimate the proportion of AI-assisted consultations classified as clinically safe by an independent physician panel and describe domain-specific concerns.

Design: Retrospective, cross-sectional, three-reviewer evaluation.

Setting: Alodokter, a large Indonesian digital health platform that deployed the evaluated clinician-facing AI clinical assistant in January 2026.

Patients: A severity-stratified sample of 402 de-identified AI-assisted general-practice consultations completed April 19–21, 2026. A calibration exercise used 30 physician-only (without AI assistant) consultations that a separate Alodokter quality-control panel had unanimously classified as safe.

Measurements: Three Indonesian physicians independently scored a 14-item rubric. The primary outcome was the proportion classified Safe or Minor omission under the three-reviewer consensus rule; a minor omission carried no immediate risk of serious harm. The calibration cohort described reviewer threshold behavior.

Results: Using the three-reviewer consensus rule, 397 of 402 consultations were classified clinically safe (98.8%; 95% CI, 97.1%–99.5%): 388 (96.5%) were Safe, 9 (2.2%) had a Minor omission, and 5 (1.24%) were Unsafe. On the binary classification aligned with the primary endpoint, all three reviewers agreed on 351 consultations (87.3%); mean pairwise agreement was 91.5%, but Fleiss’ κ was 0.045 (95% CI, −0.016 to 0.120), reflecting the high prevalence of clinically safe ratings and differences in reviewers’ Unsafe thresholds.