Human Oversight Is Not Enough: Rethinking Accountability in AI-Assisted Clinical Decision-Making
Main Article Content
Authors
Shaanxi University of Chinese Medicine
Abstract
Governance of artificial intelligence (AI) in clinical care often places clinicians between AI outputs and care decisions, yet a human signature can coexist with weak practical control. This Perspective asks why participation may fail to produce accountable oversight, how duties should be distributed, and which institutional conditions make review meaningful and correctable. Drawing on human-factors research, sociotechnical governance, public-accountability theory, and recent oversight proposals, we argue that nominal involvement cannot establish accountability when clinicians lack relevant information, time, authority, or workable review routes. We define Accountable Human Oversight (AHO) as a normative institutional arrangement linking task-specific human review to capability, authority, traceability, and redress. These conditions overlap with prior frameworks, including a 2026 model. AHO’s narrower proposed contribution is to connect them to actor–forum answerability and accessible clinical reconsideration, rather than to claim new oversight dimensions. We outline risk-proportionate governance that assigns review ownership, evidence, triggers, and corrective actions across developers and model suppliers, healthcare institutions, clinicians, and regulators, while reserving case-level review for settings in which it can add value. AHO has not been empirically evaluated and is not evidence of improved patient outcomes. Future research should examine its feasibility, workload, equity, and effects.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright remains with the author(s). All articles are published under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits use, sharing, adaptation, distribution, and reproduction in any medium or format, provided that appropriate credit is given to the original author(s) and the source, a link to the license is provided, and any changes are indicated.
How to Cite
References
Abulibdeh, Rawan, Gift Osei Agyemang, Leo Anthony Celi, Rahul Gorijavolu, Nai Lee Kalema, Ricardo Kleinlein, Kaushik Madapati, Seyed Reza Salarikia, and Alaa Youssef. 2026. “Who’s Really in the Loop? Rethinking Oversight in AI-Assisted Health Care.” The Lancet 407, no. 10545: 2340–2344. https://doi.org/10.1016/S0140-6736(26)00204-7
Bainbridge, Lisanne. 1983. “Ironies of Automation.” Automatica 19, no. 6: 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
Bovens, Mark. 2007. “Analysing and Assessing Accountability: A Conceptual Framework.” European Law Journal 13, no. 4: 447–468. https://doi.org/10.1111/j.1468-0386.2007.00378.x
Cestonaro, Clara, Arianna Delicati, Beatrice Marcante, Luciana Caenazzo, and Pamela Tozzo. 2023. “Defining Medical Liability When Artificial Intelligence Is Applied on Diagnostic Algorithms: A Systematic Review.” Frontiers in Medicine 10: Article 1305756. https://doi.org/10.3389/fmed.2023.1305756
European Parliament and Council of the European Union. 2024. “Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence and Amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act).” Official Journal of the European Union L, 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Goh, Ethan, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Adam Rodman, and Jonathan H. Chen. 2024. “Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial.” JAMA Network Open 7, no. 10: Article e2440969. https://doi.org/10.1001/jamanetworkopen.2024.40969
Goddard, Kate, Abdul Roudsari, and Jeremy C. Wyatt. 2012. “Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators.” Journal of the American Medical Informatics Association 19, no. 1: 121–127. https://doi.org/10.1136/amiajnl-2011-000089
Labkoff, Steven, Bilikis Oladimeji, Joseph Kannry, Anthony Solomonides, Russell Leftwich, Eileen Koski, Amanda L. Joseph, Monica Lopez-Gonzalez, Lee A. Fleisher, Kimberly Nolen, Sayon Dutta, Deborah R. Levy, Amy Price, Paul J. Barr, Jonathan D. Hron, Baihan Lin, Gyana Srivastava, Nuria Pastor, Unai Sanchez Luque, Tien Thi Thuy Bui, Reva Singh, Tayler Williams, Mark G. Weiner, Tristan Naumann, Dean F. Sittig, Gretchen Purcell Jackson, and Yuri Quintana. 2024. “Toward a Responsible Future: Recommendations for AI-Enabled Clinical Decision Support.” Journal of the American Medical Informatics Association 31, no. 11: 2730–2739. https://doi.org/10.1093/jamia/ocae209
Lee, John D., and Katrina A. See. 2004. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors 46, no. 1: 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
Lekadir, Karim, Alejandro F. Frangi, Antonio R. Porras, and the FUTURE-AI Consortium. 2025. “FUTURE-AI: International Consensus Guideline for Trustworthy and Deployable Artificial Intelligence in Healthcare.” BMJ 388: Article e081554. https://doi.org/10.1136/bmj-2024-081554
Lyell, David, and Enrico Coiera. 2017. “Automation Bias and Verification Complexity: A Systematic Review.” Journal of the American Medical Informatics Association 24, no. 2: 423–431. https://doi.org/10.1093/jamia/ocw105
Lyell, David, Ying Wang, Enrico Coiera, and Farah Magrabi. 2023. “More Than Algorithms: An Analysis of Safety Events Involving ML-Enabled Medical Devices Reported to the FDA.” Journal of the American Medical Informatics Association 30, no. 7: 1227–1236. https://doi.org/10.1093/jamia/ocad065
Martinho, Domingos, Pedro Sobreiro, Andreia Domingues, Filipa Martinho, and Nuno Nogueira. 2026. “Ethical Responsibility in Medical AI: A Semi-Systematic Thematic Review and Multilevel Governance Model.” Healthcare 14, no. 3: Article 287. https://doi.org/10.3390/healthcare14030287
McCradden, Melissa D., Shalmali Joshi, James A. Anderson, and Alex John London. 2023. “A Normative Framework for Artificial Intelligence as a Sociotechnical System in Healthcare.” Patterns 4, no. 11: Article 100864. https://doi.org/10.1016/j.patter.2023.100864
Omeljančiuk, Aleksej, Eimantas Peičius, Aušra Urbonienė, and Gvidas Urbonas. 2026. “Beyond Algorithmic Oversight: Internal Morality of Medicine and Meaningful Human Control in AI-Assisted Care.” Healthcare 14, no. 12: Article 1638. https://doi.org/10.3390/healthcare14121638
Parasuraman, Raja, and Victor Riley. 1997. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors 39, no. 2: 230–253. https://doi.org/10.1518/001872097778543886
Parasuraman, Raja, Thomas B. Sheridan, and Christopher D. Wickens. 2000. “A Model for Types and Levels of Human Interaction with Automation.” IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 30, no. 3: 286–297. https://doi.org/10.1109/3468.844354
Sittig, Dean F., and Hardeep Singh. 2010. “A New Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems.” Quality and Safety in Health Care 19, suppl. 3: i68–i74. https://doi.org/10.1136/qshc.2010.042085
Toro-Tobon, David, Oscar Ponce Ponte, Victor M. Montori, and Juan P. Brito. 2026. “Clinician in the Loop: A Flawed Solution for AI Oversight.” BMJ 393: Article e089213. https://doi.org/10.1136/bmj-2025-089213
U.S. Food and Drug Administration. 2026. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. January 29. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software
van de Sande, Davy, Nicoleta Economou-Zavlanos, and Michel E. van Genderen. 2026. “Meaningful Oversight of Medical AI beyond Human in the Loop.” npj Digital Medicine 9, no. 1: Article 569. https://doi.org/10.1038/s41746-026-02971-1
Vasey, Baptiste, Myura Nagendran, Bruce Campbell, David A. Clifton, Gary S. Collins, Spiros Denaxas, Alastair K. Denniston, Livia Faes, Bart Geerts, Mudathir Ibrahim, Xiaoxuan Liu, Bilal A. Mateen, Piyush Mathur, Melissa D. McCradden, Lauren Morgan, Johan Ordish, Campbell Rogers, Suchi Saria, Daniel S. W. Ting, Peter Watkinson, Wim Weber, Peter Wheatstone, Peter McCulloch, and the DECIDE-AI Expert Group. 2022. “Reporting Guideline for the Early Stage Clinical Evaluation of Decision Support Systems Driven by Artificial Intelligence: DECIDE-AI.” BMJ 377: Article e070904. https://doi.org/10.1136/bmj-2022-070904
Wang, Amy, Sam Freeman, and Farah Magrabi. 2026. “Governance for Safe and Responsible AI in Healthcare Organisations: A Scoping Review of Frameworks.” npj Digital Medicine 9, no. 1: Article 516. https://doi.org/10.1038/s41746-026-02679-2
World Health Organization. 2021. Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. Geneva: World Health Organization. https://www.who.int/publications/i/item/9789240029200
Article Sidebar
Download
No new datasets were generated or analysed for this article. The scholarly and policy sources discussed are identified in the reference list.
This work is licensed under a Creative Commons Attribution 4.0 International License.