No products in the cart.
MODULE 24: ADVANCED ANALYTICS & PREDICTIVE MODELING
Section 2: The Model Development and Validation Process
From a clinical question to a deployed predictive tool: a step-by-step masterclass on the scientific methodology behind building and trusting machine learning models in healthcare.
SECTION 24.2
Model Development and Validation Process
Applying the Scientific Method to Artificial Intelligence.
24.2.1 The “Why”: Building Trust in the Black Box
In the previous section, we demystified the core types of machine learning. It can be tempting to see these algorithms as magical black boxes: you put data in, and an answer comes out. For a clinician, this is—and should be—a terrifying prospect. You would never administer a new drug without understanding its clinical trial data, its mechanism of action, its side effect profile, and the evidence supporting its use. You demand a rigorous, transparent, and validated process. A predictive model that impacts patient care must be held to the exact same standard.
The process of developing and validating a machine learning model is not an esoteric coding exercise; it is a formal scientific methodology designed to build trust, ensure safety, and prove efficacy. It’s the “clinical trial” for an algorithm. Just as a Phase III trial has distinct stages—patient recruitment, intervention, data collection, statistical analysis, peer review—so too does the model development lifecycle. Skipping a step, using the wrong patient population (data), or misinterpreting the results can lead to an “algorithmic adverse event” just as surely as a drug can cause a physical one.
As a pharmacy informatics analyst, your role in this process is paramount. You are the clinical guardian of the methodology. While a data scientist knows the mathematics of the algorithms, you know the clinical nuances of the data. You understand why a “missing” lab value might not be missing at all, but rather indicates a healthy patient who didn’t need the test. You understand the operational workflows that generate the data and the potential biases hidden within it. You are the expert who ensures the model is not just statistically valid, but clinically valid.
This section will provide you with a structured, step-by-step framework for model development. We will introduce the industry-standard methodology and translate each phase into the practical realities of a hospital pharmacy. By the end, you will understand how to take a vague clinical idea (“I wish we could predict which patients will get a C. diff infection”) and transform it into a robust, validated, and trustworthy clinical tool. This knowledge is what separates a simple data analyst from a true clinical data scientist.
Pharmacist Analogy: Developing a New Clinical Pharmacy Service
Imagine your hospital wants you to develop a new, protocol-driven “Pharmacist-to-Dose” vancomycin service. You wouldn’t just start writing orders on day one. You would follow a rigorous, systematic process to ensure the service is safe, effective, and evidence-based. This process is a perfect parallel to the model development lifecycle.
- 1. Business/Clinical Understanding: First, you define the problem and the goal. The Problem: Sub-therapeutic and supra-therapeutic vancomycin troughs are common, leading to treatment failures and nephrotoxicity. The Goal: Increase the percentage of patients with a therapeutic first trough level from 40% to 70%. (This is like defining your model’s target and success metrics).
- 2. Data Understanding & Preparation: You conduct a retrospective chart review of the last 100 patients who received vancomycin. You look at their charts (the data), analyze their dosing, their labs, and their outcomes. You notice the data is messy: weights are often missing, serum creatinine levels are timed inconsistently, etc. You develop a standard data collection form to ensure consistency going forward (data preparation).
- 3. Modeling (Protocol Development): Based on your chart review and published guidelines, you create a detailed dosing protocol (the model). This protocol is an algorithm of “if-then” statements: IF patient weight is X and CrCl is Y, THEN the loading dose is Z and the initial maintenance dose is Q.
- 4. Evaluation: You don’t roll out the protocol hospital-wide. You conduct a pilot study (a validation). You use your protocol on the next 50 patients and meticulously track their outcomes. You compare the results to a historical control group. Did you meet your goal of 70% therapeutic troughs? Are there any safety signals? (This is like using a test set to evaluate your model’s performance).
- 5. Deployment: After proving the protocol is safe and effective in the pilot study and making any necessary adjustments, you get P&T Committee approval. You then integrate the protocol into the pharmacy workflow, train the other pharmacists, and go live. (This is model deployment).
- 6. Monitoring: The work isn’t done. You create a dashboard to continuously monitor the performance of the vancomycin service over time. Are trough goals still being met? Has the patient population changed? (This is post-deployment monitoring).
Building a predictive model follows these exact same logical, safety-oriented steps. The “protocol” is just written in mathematical terms instead of a Word document, but the underlying scientific rigor is identical.
24.2.2 The CRISP-DM Framework: The Scientific Method for Data Mining
To formalize the model development process, the data science community developed the CRoss-Industry Standard Process for Data Mining (CRISP-DM). While the name sounds corporate, its structure is pure scientific method. It provides a roadmap that breaks down a complex project into six manageable, iterative phases. As a pharmacy analyst, understanding this framework is the key to successfully collaborating with technical teams and leading analytics projects.
The CRISP-DM Lifecycle
1. Business Understanding
2. Data Understanding
3. Data Preparation
4. Modeling
5. Evaluation
6. Deployment
Notice the arrows: the process is not linear but highly iterative. You constantly loop back to previous stages as you learn more.
Phase 1: Business & Clinical Understanding
This is the most important and most often neglected phase. Before a single line of code is written, you must precisely define the problem you are trying to solve and the goals you want to achieve. A statistically perfect model that solves the wrong problem is useless. Your role as the clinical expert is to lead this phase.
The Project Charter: Your Guiding Document
A formal project charter is the best practice for kicking off any analytics project. It forces you to answer the hard questions upfront. Key components include:
- Clinical Problem Statement: A concise description of the problem in clinical terms. (e.g., “Patients discharged on 10 or more medications are at high risk for medication errors and subsequent readmission.”)
- Project Objective: The specific, measurable goal. How will you define success? (e.g., “Develop a model to identify the top 10% of patients at highest risk for readmission due to polypharmacy, and target them for TOC pharmacist intervention, with a goal of reducing their readmission rate by 15%.”)
- Data Science Problem Type: Frame the objective as a machine learning task. (e.g., “This will be a binary classification model predicting 30-day readmission [Yes/No].”)
- Stakeholders: Who needs to be involved? (e.g., TOC Pharmacists, IT/Data Warehouse team, Quality Improvement department, physician champion).
- Success Criteria (Metrics): How will you measure success? (e.g., Clinical: Reduction in readmission rate. Model Performance: Achieve a recall of at least 0.80 for the ‘Readmit’ class. Operational: The model must run daily and provide a risk score before 9 AM.)
The #1 Pitfall: Solving an Un-actionable Problem
It’s easy to come up with interesting questions to predict. But you must ask the most important question of all: “If I had a perfect model that could predict this outcome, what would I do differently?” If the answer is “nothing,” then you should not build the model. A model is only valuable if its predictions can be integrated into a workflow to trigger a specific action or decision. Predicting which patients will get a common cold is not useful. Predicting which patients are likely to be non-adherent to their DOACs is extremely useful, because you can target them for an intervention.
Phase 2 & 3: Data Understanding & Preparation
This is where the real work of data science happens. It’s often said that 80% of the time in a data science project is spent on data preparation, and only 20% is spent on modeling. This is the phase where your clinical expertise is most critical to prevent a “garbage in, garbage out” scenario. The raw data from an EHR is a messy, complex, and often misleading reflection of clinical reality.
Data Understanding: The Chart Review at Scale
In this sub-phase, you act like a detective. You explore the raw data sources, assess their quality, and form initial hypotheses. You ask questions like:
- Where does this data live? Are the lab results in one table? Are the medication orders in another? Do I need to join five different tables to get the full picture for one patient?
- What do the fields actually mean? Does the `medication_stop_date` field mean the doctor discontinued it, or the patient’s admission ended? This requires deep institutional knowledge.
- How much data is missing? If you’re building a model to predict AKI and 50% of your patients are missing a baseline creatinine, your model will be fundamentally flawed.
- Are there outliers or strange values? A recorded weight of 5 lbs or 5000 lbs is clearly a data entry error. How will you handle these?
Data Preparation: The Art of Feature Engineering
Once you understand your data, you must transform it into a clean, structured format that a machine learning model can understand. This involves cleaning the data and, most importantly, feature engineering—the process of using your domain knowledge to create new, more powerful features from the raw data.
This is where you, the pharmacist, can provide enormous value. A data scientist might see a list of 20 medications. You see a complex regimen that includes a high-risk anticoagulant, a drug requiring therapeutic monitoring, and a significant drug-drug interaction. You can translate your clinical assessment into new features the model can use.
Masterclass Table: From Raw Data to Powerful Features
| Raw Data Available | Simple (Useless) Feature | Clinically-Informed Engineered Feature | Why It’s Better |
|---|---|---|---|
| A list of all dispensed medications for a patient. | `number_of_medications` |
|
The raw number of meds is less important than the type and risk of those meds. An 80-year-old on 12 vitamins is very different from an 80-year-old on 12 cardiovascular drugs. Your engineered features capture this clinical nuance. |
| A list of all lab tests and results. | The most recent `serum_creatinine` value. |
|
A single creatinine value is meaningless without context. The trend and rate of change are what signal a clinical problem like Acute Kidney Injury (AKI). Your engineered features capture this dynamic, time-series aspect of the data. |
| Admission and discharge timestamps. | `length_of_stay_in_days` |
|
The total length of stay is an outcome, but features related to the timing and context of the stay can be powerful predictors. Patients discharged late on a Friday may have more trouble accessing their pharmacy, a key risk factor your engineered feature captures. |
| Unstructured text from a physician’s discharge summary note. | (Model cannot use raw text) |
|
Using Natural Language Processing (NLP) techniques to search for keywords (e.g., “homeless,” “lives alone,” “poor historian,” “non-compliant”) allows you to extract incredibly valuable risk factors from unstructured text that would otherwise be invisible to the model. |
Phase 4, 5, & 6: Modeling, Evaluation, and Deployment
These final phases are where the statistical work happens and the model is brought to life. While the technical details are often handled by data scientists, your role as a clinical validator and workflow expert is crucial for success.
Phase 4: Modeling
This is the phase where the data scientist takes the clean, prepared data and trains various machine learning algorithms. They might try a logistic regression, a random forest, and a gradient-boosted model to see which one performs best on the data. A key concept here is the Train-Test Split.
The Train-Test Split: Preventing “Cheating”
To get an honest assessment of a model’s performance, you cannot evaluate it on the same data it was trained on. That would be like giving a student the answer key to an exam and then being impressed when they get a perfect score. To prevent this, the data is split before training begins:
- Training Set (typically 70-80% of the data): This is the data the model gets to “see” and learn the patterns from.
- Testing Set (typically 20-30% of the data): This data is kept in a locked box that the model never sees during training. It is used only at the very end to provide an unbiased evaluation of how well the model performs on new, unseen data.
A model that performs perfectly on the training data but fails miserably on the test data is said to be overfit. It has essentially memorized the training data instead of learning the generalizable patterns. This is a common and dangerous problem that the train-test split helps to diagnose.
Phase 5: Evaluation
Once a model is trained, it is unleashed on the test set. Now you must rigorously evaluate its performance using both statistical metrics and clinical judgment. We discussed key classification metrics (Precision, Recall, Accuracy) in the last section. Another critical tool is the Receiver Operating Characteristic (ROC) Curve.
An ROC curve plots the model’s True Positive Rate (Recall/Sensitivity) against its False Positive Rate at all possible classification thresholds. A perfect model would go straight up to the top-left corner (100% True Positives, 0% False Positives). A useless model that is no better than a coin flip would be the diagonal line. The Area Under the Curve (AUC) is a single number that summarizes the model’s performance across all thresholds. An AUC of 1.0 is a perfect model; an AUC of 0.5 is a useless one.
Your Clinical Role in Evaluation: Your job is to go beyond the AUC. You must perform a qualitative error analysis. Look at the specific cases the model got wrong.
- False Positives: Are there patterns in the patients the model incorrectly flagged as high-risk? Maybe it’s flagging all patients from a specific surgical unit that has good outcomes, revealing a bias in the data.
- False Negatives: This is even more important. Look at the high-risk patients the model missed. Why did it miss them? Did you fail to include a critical feature? Was there a data quality issue? This clinical investigation is essential for improving the model and ensuring its safety.
Phase 6: Deployment & Monitoring
A model is not a final report; it’s a living clinical tool. Deployment is the process of integrating the model into a clinical workflow where it can be used.
Deployment Options:
- Passive / Informational: The model generates a daily report or dashboard that clinicians can reference (e.g., the readmission risk list for TOC pharmacists).
- Active / Alerting: The model is integrated directly into the EHR and fires a real-time Best Practice Advisory (BPA) or alert to a clinician (e.g., an AKI prediction model alerting a provider to a high-risk patient). This is more powerful but carries a much higher risk of alert fatigue.
Post-Deployment Monitoring: Once a model is live, it must be continuously monitored. The world changes, clinical practice changes, and new drugs are introduced. A model trained on data from 5 years ago may no longer be accurate today. This phenomenon is known as model drift. You must have a plan to periodically retrain and re-validate the model on new data to ensure it remains accurate and safe over time.
