CPIA Module 24, Section 5: Real-World Project – Adverse Event Prediction
MODULE 24: ADVANCED ANALYTICS & PREDICTIVE MODELING

Section 5: Real-World Project – Adverse Event Prediction

A capstone project where we will walk through the process of building and validating a machine learning model designed to predict which patients are at the highest risk for an adverse drug event, turning theory into a tangible, life-saving tool.

SECTION 24.5

Real-World Project: Adverse Event Prediction

From Reactive Reporting to Proactive Patient Protection.

24.5.1 The “Why”: The Pharmacist’s Ultimate Goal

For your entire career, your primary function has been to prevent adverse drug events (ADEs). You do this one patient at a time, one order at a time. You scrutinize doses, check for interactions, review lab values, and counsel patients. This manual, expert-driven process is the bedrock of medication safety. But it has a fundamental limitation: it doesn’t scale. A pharmacist can only review so many charts in a day. In a 500-bed hospital, thousands of medication orders are processed daily, and it is impossible for a human to apply that deep, cognitive risk assessment to every single patient, every single day.

Traditionally, our response to ADEs has been reactive. We wait for an event to occur, we perform a root cause analysis, and we implement system-level changes to prevent that specific error from happening again. This is vital work, but it means a patient had to be harmed for us to learn. Predictive modeling offers a paradigm shift. It gives us the ability to move from being medication safety historians to becoming medication safety futurists. By training a machine learning model to recognize the subtle, complex patterns that precede an adverse event, we can systematically and automatically scan the entire patient population to find the needles in the haystack—the few patients who are at exceptionally high risk right now.

This capstone section is the culmination of everything you have learned in this module. We will walk through the end-to-end process of building a clinical prediction model. We will not just discuss theory; we will simulate the real-world decisions, challenges, and triumphs of taking a project from an idea (“Can we predict which patients will have a major bleed?”) to a functional, deployed tool that can be used by pharmacists to save lives. This project represents the pinnacle of pharmacy informatics: using advanced technology to amplify and scale the clinical expertise of pharmacists, allowing them to intervene before harm occurs and fundamentally changing the way we protect our patients.

Pharmacist Analogy: The Expert Anticoagulation Consult

Imagine you are the hospital’s anticoagulation expert. A physician calls you for a consult on a complex patient: an 82-year-old female with atrial fibrillation, a new DVT, a history of GI bleeding, end-stage renal disease, and a platelet count of 60,000. The question is simple: “Should I anticoagulate her, and if so, with what?”

Your brain does not just look up a guideline. It performs a complex, multi-variable risk assessment in real-time. This is your internal “prediction model.”

  • Feature Engineering: You are taking raw data points (age, diagnoses, labs) and turning them into meaningful features. Age > 75 is a risk factor for bleeding. The DVT is a strong risk factor for clotting. The history of GI bleed is a huge bleeding risk. ESRD dramatically increases the bleeding risk for most DOACs. Thrombocytopenia is another major bleeding risk.
  • Weighting: You don’t treat all these factors equally. In your expert opinion, the combination of a recent GI bleed history and severe thrombocytopenia is weighted much more heavily as a bleeding risk than her age alone.
  • Scoring & Thresholding: You mentally add up these weighted factors and arrive at a final “score” or gestalt. You conclude: “This patient’s risk of a catastrophic bleed is extremely high and likely outweighs the benefit of anticoagulation right now. My recommendation is to hold anticoagulation, place an IVC filter for the DVT, and re-evaluate her bleeding risk daily.” You have made a classification: you’ve predicted her to be in the “High-Risk Bleed” class.

The machine learning model we are about to build does the exact same thing. It simply replaces your single expert brain with a statistical algorithm that has learned from the outcomes of tens of thousands of previous patients. It learns the “weights” for each risk factor from the data itself. The goal is not to replace you, the expert pharmacist. The goal is to build a tool that can perform your expert-level risk assessment on every single patient in the hospital, every hour of the day, and then bring the handful of most complex and highest-risk cases to your attention. It is a tool for scaling expertise.

24.5.2 The Project Roadmap: Following the CRISP-DM Framework

A successful data science project is not a haphazard exploration; it’s a structured, disciplined process. We will use the CRoss-Industry Standard Process for Data Mining (CRISP-DM) as our roadmap. This is a battle-tested methodology that breaks a project down into six iterative phases, ensuring that we stay focused on the clinical goal and don’t get lost in the technical details.

Data Science Business Understanding Data Understanding Data Preparation Modeling Evaluation Deployment

Phase 1: Business & Clinical Understanding

This is the most important phase. A technically perfect model that solves the wrong problem is useless. We must start by deeply understanding the clinical need and defining a precise, measurable goal.

Defining the Problem

The vague goal of “predicting ADEs” is too broad. We need to narrow our focus. After meeting with the Medication Safety Officer and the Anticoagulation Stewardship committee, we define a specific target:

Project Goal: To develop and validate a machine learning model that, for all adult inpatients receiving therapeutic anticoagulation, predicts the risk of a clinically significant bleeding event within the next 24 hours.

Defining Success

How will we know if we’ve succeeded? Success is not just a high AUC. It must be defined operationally.

  • Technical Success: The final model must achieve a precision of at least 20% when identifying the top 5% of highest-risk patients. (This means that out of every 5 patients flagged by the model, at least 1 will be a true positive).
  • Operational Success: The model must be integrated into a daily workflow that generates a prioritized list of high-risk patients for the anticoagulation pharmacist. This workflow must be perceived as useful and non-disruptive by the pharmacists.
  • Clinical Success (Long-Term): Over a 6-month pilot period, the implementation of this tool should lead to a statistically significant increase in proactive, pharmacist-led bleeding mitigation strategies (e.g., dose adjustments, stopping NSAIDs) for high-risk patients.

Phase 2: Data Understanding

Now that we know our destination, we need to map out the data we’ll need to get there. This involves identifying potential predictive features and performing Exploratory Data Analysis (EDA) to understand their characteristics.

We will query our hospital’s EHR data warehouse (e.g., Epic’s Caboodle, Cerner’s HealtheIntent) to extract a dataset of all patient-days where a patient received therapeutic anticoagulation over the past 3 years.

Masterclass Table: Potential Predictor Variables (Features)
CategoryFeatureWhy It’s Potentially Predictive
Demographics Age Older age is a well-established, independent risk factor for bleeding.
Weight / BMI Extremes of weight can affect drug clearance and dosing, impacting risk.
Lab Values Serum Creatinine (SCr) / eGFR Renal dysfunction is one of the strongest predictors of bleeding, as many anticoagulants are renally cleared.
Hemoglobin (Hgb) / Hematocrit (Hct) Low baseline Hgb may indicate chronic bleeding or lower physiological reserve. A sharp drop can define a bleed event.
Platelet Count Thrombocytopenia is a direct and powerful risk factor for bleeding.
INR / aPTT Supratherapeutic levels are a direct cause of bleeding. High variability is also a risk factor.
Liver Function Tests (AST/ALT/Bili) Liver dysfunction impairs production of clotting factors, increasing bleed risk.
Medications Concurrent Antiplatelets (Aspirin, Clopidogrel) Synergistic effect significantly increases bleeding risk.
Concurrent NSAIDs Increases risk of GI bleeding through direct mucosal injury and platelet inhibition.
Polypharmacy Score A proxy for illness complexity and risk of drug interactions that may potentiate bleeding.
Diagnoses History of Bleeding (e.g., GI Bleed, ICH) One of the strongest predictors of a future bleed is a past bleed.
Comorbidity Index (e.g., Charlson, Elixhauser) A measure of the patient’s overall burden of chronic disease.

Phase 3: Data Preparation

This is the least glamorous but most important phase of the project, often consuming 80% of the total project time. Raw EHR data is messy, inconsistent, and not in a format that a machine learning model can understand. We must clean, transform, and engineer it into a high-quality analytical dataset.

Defining the Target Variable: What Is a “Bleed”?

This is a critical clinical informatics challenge. There is no single field in the EHR called `had_a_bleed`. We must create a composite definition using multiple data sources. Our definition will be:

A Clinically Significant Bleed Event is defined as a 24-hour period where a patient on therapeutic anticoagulation meets at least ONE of the following criteria:

  • Receipt of a new ICD-10 code for a significant bleed (e.g., Intracranial Hemorrhage, GI Bleed).
  • A drop in Hemoglobin of ≥ 2 g/dL.
  • Transfusion of ≥ 2 units of packed red blood cells (PRBCs).

Feature Engineering: From Raw Data to Model Food

We must now transform the raw data points from our table into numerical features the model can use.

  • Labs: We can’t just use the raw SCr value. We will use the most recent value within the last 24 hours. We will also create a binary flag: `is_renal_impaired` (1 if eGFR < 30, 0 otherwise). For Hgb, we'll use the value itself, and also a feature called `hgb_change_last_24h`.
  • Medications: We create binary flags: `on_aspirin` (1 if yes, 0 if no), `on_nsaid`, etc.
  • Diagnoses: We search the patient’s entire problem list for specific ICD-10 codes to create binary flags like `has_history_gi_bleed`.
  • Handling Missing Data: What if a patient didn’t have a platelet count drawn yesterday? We can’t have a blank value. We must use an imputation strategy, such as carrying forward the last known value or imputing the mean/median value for the entire population.

Phase 4 & 5: Modeling and Evaluation

With our clean, feature-rich dataset, we can finally train and evaluate our models. We will split our data into a training set (70%), a validation set (15%), and a testing set (15%).

Model Selection

We’ll start with a simple, interpretable model and then move to a more complex one.

  • Baseline Model: Logistic Regression. This classic statistical model is highly interpretable. We can look directly at the coefficients and see exactly how the model is “weighing” each risk factor. This is crucial for gaining clinical trust.
  • Advanced Model: Gradient Boosted Machine (XGBoost). This is a powerful, tree-based ensemble model that can capture complex, non-linear interactions between features. It will likely have higher predictive accuracy but is a “black box,” making it harder to explain its individual predictions.

Evaluation: It’s All About the Confusion Matrix

After training the model on the training set, we unleash it on our unseen test set to evaluate its performance. The results are summarized in a confusion matrix.

Model Confusion Matrix
Predicted: No Bleed
Predicted: Bleed
Actual: No Bleed
9,500
True Negatives (TN)
200
False Positives (FP)
(Alert Fatigue)
Actual: Bleed
250
False Negatives (FN)
(Missed Events)
50
True Positives (TP)
(Correct Alerts)
Interpreting the Results in a Clinical Context:
  • True Positives (TP): 50. Our model correctly identified 50 bleeding events. These are our successes.
  • False Positives (FP): 200. Our model flagged 200 patients who did not go on to bleed. This is the cost of using the model, leading to alert fatigue for the pharmacists who review these cases.
  • False Negatives (FN): 250. Our model missed 250 patients who did have a bleeding event. This represents the risk or harm of relying on the model. These are the most dangerous errors.

Key Metrics:

Precision (Positive Predictive Value): $$ \frac{TP}{TP + FP} = \frac{50}{50 + 200} = 20% $$ Of all the patients our model flagged, 20% of them were correct. This meets our technical success criterion.

Recall (Sensitivity): $$ \frac{TP}{TP + FN} = \frac{50}{50 + 250} = 16.7% $$ Our model only found 16.7% of all the true bleeding events. This highlights the trade-off: to reduce alert fatigue (low FP), we may have to accept missing some events (high FN).

Phase 6: Deployment & The Human in the Loop

A model sitting on a data scientist’s laptop has zero clinical value. Deployment is the process of integrating the model’s output into a real clinical workflow.

The Golden Rule: Augment, Don’t Automate

We do NOT create an automatic alert that fires to the physician saying “Patient at high risk of bleeding!” This would create massive alert fatigue and would be ignored. The model is not perfect, and it lacks clinical context. Instead, we use the model to augment the expert pharmacist.

The Deployed Workflow

1. Nightly Model Run

The model runs on all anticoagulated patients in the hospital.

→

2. Prioritized Worklist

Generates a report of the top 10 highest-risk patients, sorted by probability score.

→

3. Pharmacist Review

The anticoagulation pharmacist reviews this short, targeted list as their first task.

→

4. Clinical Action

Pharmacist performs a chart review and makes a recommendation if needed.

This “human-in-the-loop” system is the key to safe and effective deployment. The model does the heavy lifting of scanning the entire population, and the pharmacist applies their clinical judgment to the small subset of patients who need it most. This is the future of clinical decision support, transforming your role into a data-driven, proactive guardian of medication safety.