Deployment data extraction
One major identified barrier for the physics team was to develop a method for extracting data in real-time clinical practice. In aggregate, the below data extraction process required a median of 5 h (interquartile range [IQR] 4–5 h) per week of a medical physics resident’s time.
For the purposes of deployment, identification of new RT courses was required. One major challenge in practically identifying these courses was the labels used in the Aria oncology information system (OIS) (Varian Medical Systems, Palo Alto). During retrospective model development, this was simply queried to identify 8134 courses of radiotherapy completed from 2013 to 2016 [11]. In prospective development, identification of courses required queries through the scheduling system. The OIS designation at the time of SHIELD-RT designated new treatment appointments as three potential options: “new start” (new patient beginning new course), “old start” (patient with a prior OIS course starting new course) or “final treatment” (either final fraction of a multi-fraction treatment or start of a single fraction treatment) (Fig. 1). To identify courses during the first week of treatment, manual review was needed to verify “old starts” and for quality assurance to verify that single fraction treatments labeled as “final treatment” were indeed a new course of radiation therapy.
After identification of eligible treatment courses, RT data were extracted from the OIS, including details regarding the treatment course name, prescription, total dose, number of fractions, RT technique, and patient diagnosis based on International Classification of Diseases (ICD) codes.
Additional manual review was required to inspect draft (unsigned) prescriptions of sequential RT boosts and verify that they were an intended component of the treatment plan. This included subsequent radiation plans that were designed to deliver additional RT dose to a portion of the originally treated field within a single treatment course (e.g., a boost to a breast tumor bed following lumpectomy after primary whole breast treatment). Manual review of their inclusion was needed to accurately characterize a patient’s planned treatment course. Draft prescriptions typically represent planned treatment, but can also include boosts that are no longer intended (e.g., due to radiation planning constraints). These draft prescriptions are sometimes pended unsigned at the start of treatment initiation and therefore not automatically aggregated.
Machine learning deployment
Once patient RT data was identified, the process to generate ML predictions, randomize patients, and deploy clinical alerts was undertaken, requiring a median of 1.5 h per week (IQR 1–2 h) of the lead investigator’s time. From the OIS-generated patient list, the patient medical record number was used to query pre-treatment EHR data from the Duke enterprise data unified content explorer (DEDUCE) to provide additional input for the ML prediction [12]. DEDUCE aggregates data directly from the hospital and clinic operations via the Decision Support Repository (DSR), similarly to efforts utilizing data from institutional clinical data warehouses [13,14,15].
The combined OIS and EHR-queried data were then input into an aggregated R script to generate ML predictions. Patients identified as high risk (ML predicted 10% or greater risk of requiring acute care) were then entered into a REDCap database, which facilitated randomization, study documentation, and auditing [16]. Alerts were then manually placed in the OIS so that patients could be appropriately directed to supplemental visits, and the treating team was notified via manual emails. For auditing at a later time during the course of the study, the ML model was then run by two independent investigators and output verified.
The clinical workflow
During treatment, alerts in the OIS prompted radiation therapists to direct high-risk patients who were randomized to the intervention arm to examination rooms for weekly mandatory supplemental visits. As previously reported, 79.7% (444 of 557) of mandatory supplemental evaluations were completed, with a median of 0 missed visits per course (IQR 0–1). Anecdotally, these were largely associated with missed alerts or patients forgetting about their supplemental evaluations especially in the context of variable scheduled times. These visits required an additional median of 5 min (IQR 5–10 min) of clinician time per visit [4].