- Title Information
- Title
- A graphical toolkit for longitudinal dataset maintenance and predictive model training in healthcare
- Type of Resource (primo)
- text_resources
- Abstract
- BACKGROUND: Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and re-training.
OBJECTIVES: To present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end-users.
Methods: Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semi-structured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies.
RESULTS: Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of “good”. We reviewed persona-based case studies that illustrate functionalities of this novel prototype.
CONCLUSIONS: The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.
- Name
- Name Part
- Bai, Eric
- Role
- Role Term (marcrelator)
(authorityURI="http://id.loc.gov/vocabulary/relators", valueURI="http://id.loc.gov/vocabulary/relators/aut")
- Author
- Name
- Name Part
- Song, Sophia
- Role
- Role Term (marcrelator)
(authorityURI="http://id.loc.gov/vocabulary/relators", valueURI="http://id.loc.gov/vocabulary/relators/aut")
- Author
- Name
- Name Part
- Fraser, Hamish
- Role
- Role Term (marcrelator)
(authorityURI="http://id.loc.gov/vocabulary/relators", valueURI="http://id.loc.gov/vocabulary/relators/aut")
- Author
- Name
- Name Part
- Ranney, Megan
- Role
- Role Term (marcrelator)
(authorityURI="http://id.loc.gov/vocabulary/relators", valueURI="http://id.loc.gov/vocabulary/relators/aut")
- Author
- Name:
Corporate
- Name Part
- Brown University. Alpert Medical School. Scholarly Concentration Program. Non-Scholarly Concentrator
- Role
- Role Term:
Text
- research program
- Subject (fast)
(authorityURI="http://id.worldcat.org/fast", valueURI="http://id.worldcat.org/fast/01004795")
- Topic
- Machine learning
- Subject (fast)
(authorityURI="http://id.worldcat.org/fast", valueURI="http://id.worldcat.org/fast/00907154")
- Topic
- Electronic digital computers--Health aspects
- Subject (fast)
(authorityURI="http://id.worldcat.org/fast", valueURI="http://id.worldcat.org/fast/00864368")
- Topic
- Clinical medicine--Decision making--Data processing
- Subject (fast)
(authorityURI="http://id.worldcat.org/fast", valueURI="http://id.worldcat.org/fast/01163191")
- Topic
- User interfaces (Computer systems)
- Language
- Language Term:
Text (ISO639-2B)
- English
- Origin Information
- Date Created
(keyDate="yes", encoding="w3cdtf")
- 2021
- Note
(displayLabel="Scholarly concentration")
- Non-Scholarly Concentrator
- Access Condition:
use and reproduction
(href="https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode")
- Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)
- Access Condition:
logo
(href="https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png")
- Access Condition:
rights statement
(href="http://rightsstatements.org/vocab/InC/1.0/")
- In Copyright
- Identifier:
DOI
- 10.26300/bwjm-pn67