CPIA Module 7, Section 3: Testing & Validation Methods
MODULE 7: Building & Maintaining Clinical Decision Support (CDS) Rules

Section 7.3: Testing & Validation Methods

A critical lesson in quality assurance. You’ll learn how to design rigorous testing plans, create patient test cases, and validate that your rule fires accurately for the intended scenarios—and, just as importantly, remains silent when it should.

SECTION 7.3

Testing & Validation Methods

The Pharmacist’s Final Check: Ensuring Digital Safety Before Dispensing a Rule.

7.3.1 The “Why”: The Professional Obligation of Verification

In your entire professional life as a pharmacist, one principle has been drilled into your core identity more than any other: the final check. You would never allow a prescription to be dispensed without a pharmacist meticulously verifying its accuracy, appropriateness, and safety. This act of verification is not merely a task; it is the ethical cornerstone of our profession. It is the promise we make to our patients that the medication they receive is the correct one, at the correct dose, for the correct reason. This sacred duty does not disappear when you transition to informatics; it is amplified.

A Clinical Decision Support rule is, in essence, a prescription for a process. It is a set of instructions that will be dispensed to hundreds or thousands of clinicians and will impact the care of countless patients. Activating a new CDS rule in the live EHR environment without subjecting it to the most rigorous, exhaustive testing and validation process imaginable is the informatics equivalent of handing a patient a bottle of medication without ever looking at the label. It is a breach of our fundamental professional obligation. An untested or poorly tested rule is not a safety net; it is a hidden landmine waiting to cause catastrophic harm—either by firing when it shouldn’t (creating alert fatigue and causing clinicians to ignore all warnings) or by failing to fire when it should (allowing a preventable error to reach the patient).

This section is dedicated to the science and art of verification in the digital realm. You will learn that testing is not a superficial, “check-the-box” activity performed at the end of a build. It is a comprehensive, structured, and continuous process that begins the moment you write your first line of pseudo-code. We will explore the different phases of testing, from the microscopic examination of a single piece of logic to the holistic assessment of how your rule interacts with the entire clinical workflow. Mastering these methods is non-negotiable. It is how you earn the trust of the frontline clinicians who will use your tools, and it is how you fulfill your unwavering duty to ensure the safety of every patient your work touches.

Retail Pharmacist Analogy: The Sterile Compounding Verification

Imagine you are the Pharmacist-in-Charge of a pharmacy that performs sterile compounding. A physician has ordered a complex Total Parenteral Nutrition (TPN) solution for a critically ill neonate. The calculations are intricate, involving dozens of ingredients in minute quantities.

Your verification process is a multi-layered, systematic quality assurance protocol. It is not a single, quick glance.

  1. Blueprint Verification (Pseudo-Code Review): Before anything is mixed, you take the physician’s order and independently perform all the calculations on paper. You verify osmolarity, electrolyte concentrations, and caloric content. This is the peer review of your “pseudo-code.”
  2. Ingredient Verification (Unit Testing): You watch the technician pull the ingredients. You check the NDC and lot number of every single vial against your worksheet. You are testing the individual “data objects” to ensure they are correct.
  3. Process Verification (System Integration Testing): You observe the technician’s aseptic technique. You ensure they use the correct syringe, the correct needle, and follow the prescribed order of mixing to prevent precipitation. You are testing how the components interact with the workflow.
  4. Final Product Verification (User Acceptance Testing): After the TPN is compounded, you perform the final check. You hold the bag up to the light, looking for particulates or precipitation. You check the final volume. You confirm the label is perfect. This is your final sign-off before it goes to the patient.

This rigorous, multi-step validation is the only professionally acceptable way to ensure the safety of a high-risk compound. The process for validating a CDS rule is identical in principle. You must verify the logic on paper, test each data component in isolation, test how the rule interacts with the larger EHR workflow, and perform a final acceptance test from a clinician’s perspective before you “dispense” the rule to the entire organization.

7.3.2 The Testing Hierarchy: From Unit to User

Effective CDS validation is not a single event, but a staged campaign. We move from the most granular level of testing up to the most holistic, ensuring quality at each stage before proceeding to the next. This hierarchy is typically broken down into three distinct phases: Unit Testing, System Integration Testing, and User Acceptance Testing (UAT).

1. Unit Testing

Does the rule’s internal logic work correctly in isolation? Focus on the “IF-THEN” statement itself.

→

2. System Integration Testing

Does the rule’s action work correctly with other parts of the EHR? Focus on the “THEN” clause’s interaction with the system.

→

3. User Acceptance Testing

Is the rule clinically appropriate, useful, and understandable from the end-user’s perspective? Focus on the human-computer interaction.

Attempting to skip a stage in this hierarchy is a primary cause of failed CDS implementations. A rule that passes Unit Testing (the logic is correct) but fails System Integration Testing (the suggested order button is broken) is a technical failure. A rule that passes both but fails UAT (the alert text is confusing to doctors) is a clinical failure. All three are required for success.

7.3.3 Masterclass in Unit Testing: The Art of the Test Case

Unit testing is the scientific core of validation. It is the process of creating fictional patient scenarios, or “test cases,” to prove that your rule’s logic is flawless. Your goal is to prove two equally important things:

  1. The rule fires every single time it is supposed to.
  2. The rule remains silent every single time it is supposed to.

Failure to rigorously test the second point is a catastrophic error. A rule that is too “noisy” and fires inappropriately will be ignored by clinicians, rendering it useless and contributing to alert fatigue. A great informaticist spends as much time building test cases to prove a rule’s silence as they do to prove its voice.

To do this, we create three categories of test cases: Positive, Negative, and Edge.

Positive Test Cases: Proving the “Fire”

A positive test case is a patient profile specifically constructed to meet all the “IF” and “AND” criteria of your rule, while avoiding all “NOT” criteria. You are building the ideal patient for whom the alert was designed.

Masterclass Table: Positive Test Case for Levofloxacin Renal Dosing Rule
Test Case: PHA-LRD-01 (Positive)
Objective To verify that the levofloxacin renal dosing alert fires correctly for a patient with moderate renal impairment receiving an unadjusted standard dose.
Test Patient Profile
  • Name: Positive RenalTest, John
  • Age: 72 years
  • Sex: Male
  • Weight: 80 kg (documented today)
  • Allergies: No Known Allergies
  • Problem List: Hypertension, Type 2 Diabetes
  • Recent Labs (within 12 hours):
    • Serum Creatinine: 2.1 mg/dL
  • Calculated CrCl (Cockcroft-Gault): $$ \frac{(140 – 72) \times 80}{72 \times 2.1} \approx 35.9 \text{ mL/\min} $$
Action Performed Log in as a test physician. Open John Positive_RenalTest’s chart. Place a new medication order for Levofloxacin 750 mg IV Q24H. Sign the order.
Expected Outcome
  • An interruptive alert should appear immediately upon signing the order.
  • The alert text should read: “Patient’s estimated CrCl is 36 mL/min. For CrCl 20-49, consider adjusting dose to 500 mg Q48H per hospital guidelines.” (Note the dynamic data pull).
  • The alert should have a clickable button labeled “Change order to 500 mg Q48H.”
Rationale The patient meets all criteria:
  • Order is for Levofloxacin.
  • Calculated CrCl (36 mL/min) is between 20 and 49.
  • Lab data is recent.
  • Patient is not on dialysis.
  • The ordered dose (750mg Q24H) is NOT an approved renal dose, so the exclusion logic does not apply.

Negative Test Cases: Proving the “Silence”

A negative test case is a patient profile constructed to fail at least one of the rule’s core criteria or to meet one of its exclusion criteria. You must create multiple negative test cases to test each logical branch of your rule.

Masterclass Table: Negative Test Cases for Levofloxacin Rule
Test Case ID Objective Test Patient Profile / Action Expected Outcome
PHA-LRD-02 (Negative – Good Renal) Verify alert does NOT fire for a patient with normal renal function. Same profile as John Positive_RenalTest, but with Serum Creatinine: 0.8 mg/dL (CrCl ≈ 94 mL/min). Order Levofloxacin 750 mg Q24H. NO ALERT. The order should be signed without interruption.
PHA-LRD-03 (Negative – Already Dosed) Verify alert does NOT fire if the physician has already ordered the correct renal dose. Same profile as John Positive_RenalTest (CrCl ≈ 36 mL/min). Order Levofloxacin 500 mg IV Q48H. NO ALERT. The rule’s exclusion logic should recognize this as an appropriate renal dose and suppress the alert.
PHA-LRD-04 (Negative – Dialysis) Verify alert does NOT fire for a patient on hemodialysis. Patient profile with Problem List: ‘End Stage Renal Disease’ and a documented dialysis nursing flow-sheet entry. Order Levofloxacin 750 mg Q24H. NO ALERT. The rule should be suppressed by the dialysis exclusion criterion, deferring to a different, dialysis-specific dosing protocol.
PHA-LRD-05 (Negative – Different Drug) Verify alert does NOT fire for a different medication. Same profile as John Positive_RenalTest (CrCl ≈ 36 mL/min). Order Ciprofloxacin 400 mg IV Q12H. NO ALERT. This confirms the rule is specific to the intended medication/class.

Edge Cases: Probing the Boundaries

Edge cases are the subtle, tricky scenarios that live on the boundaries of your logic. Testing for them is what separates a good informaticist from a great one. These are the cases most likely to be missed and cause unexpected behavior.

Masterclass Table: Edge Cases for Levofloxacin Rule
Test Case ID Objective Test Patient Profile / Action Expected Outcome & Rationale
PHA-LRD-06 (Edge – Boundary Value) Verify rule behavior at the exact upper boundary of the CrCl range. Manipulate patient weight/creatinine to achieve a Calculated CrCl of exactly 49.5 mL/min. Order Levofloxacin 750 mg Q24H. Depends on system precision. Does the system round the CrCl? Does the logic use “less than 50” or “less than or equal to 49”? The expected outcome is whatever you defined in your pseudo-code, but this test verifies the system handles boundaries as you intended.
PHA-LRD-07 (Edge – Stale Data) Verify rule does NOT fire if the only available lab data is old. Same profile as John Positive_RenalTest, but the Serum Creatinine of 2.1 mg/dL is from a lab drawn 4 days ago. NO ALERT. The rule logic should check the timestamp of the lab (`Source_SerumCreatinine.Timestamp is within last 48 hours`) and fail this condition, preventing an alert based on potentially irrelevant data.
PHA-LRD-08 (Edge – Missing Data) Verify rule behavior when a critical piece of data is missing. Create a new patient with an age and creatinine, but with no documented weight. Order Levofloxacin 750 mg Q24H. Ideally, NO ALERT and an error logged. The CrCl calculation should fail gracefully without a weight. The rule should not fire. In the background, this event should be logged for informatics to review, as it may represent a workflow issue (nurses not documenting admission weights). A poorly built rule might crash or use a default value, leading to a false alert.

7.3.4 Beyond Logic: System Integration and User Acceptance Testing

Passing unit testing is a major milestone. You have proven that your rule’s logic is sound. But your job is not done. You must now move up the hierarchy to ensure the rule not only thinks correctly but also acts correctly within the complex ecosystem of the EHR and is understood by the humans who will interact with it.

System Integration Testing: Does the “THEN” Clause Work?

This phase focuses entirely on the action your rule is supposed to perform. You are no longer primarily testing the “IF” conditions; you are testing the “THEN” clause’s interaction with other parts of the system. This requires moving beyond the rule editor and actually simulating the full clinical workflow in a test environment.

  • Testing Suggested Orders: For our levofloxacin rule, we included a button to “Change order to 500 mg Q48H.” In integration testing, you run the positive test case again. When the alert fires, you click that button. You must then verify: Did the original 750mg Q24H order get correctly discontinued? Was a new, signed order for 500mg Q48H created? Did it route to the pharmacy verification queue correctly? A failure at any of these steps is an integration failure.
  • Testing Communications: If your rule is designed to send a task or a message (e.g., “Send task to Pharmacy for peri-op anticoag plan”), you must verify that the task is generated, contains the correct patient information, and is routed to the correct work queue.
  • Testing Performance: A rule that is logically correct but queries massive amounts of data inefficiently can slow down the EHR for everyone. Integration testing often includes performance analysis to ensure your new rule does not negatively impact system response times, especially during peak ordering hours.

User Acceptance Testing (UAT): The Final Clinical Check

This is the final, and arguably most important, phase. UAT is the process of having actual frontline clinicians (physicians, pharmacists, nurses) test your rule in a simulated environment and provide feedback. Your rule may be logically perfect and technically sound, but if clinicians find it confusing, annoying, or clinically irrelevant, it will fail in the real world. UAT is your chance to catch these “human factors” issues before go-live.

The Pharmacist’s UAT Playbook

When you are asked to participate in UAT for a new CDS rule, you are performing a critical peer review. Approach it with the same rigor as you would verifying a complex prescription.

  1. Receive the Test Script: The informaticist who built the rule should provide you with a test script. This document will outline the test patient’s scenario and the steps to perform (e.g., “Log in as Dr. Test. Open patient Jane UAT. Order Drug X.”).
  2. Follow the Script… At First: Execute the positive and negative test cases exactly as written. Does the alert fire when it should? Does it stay silent when it should? Document your results.
  3. Go Off-Script – Try to Break It: This is where your clinical expertise shines. Think like a busy resident at 3 AM. What are the weird, non-standard ways someone might place this order? Try ordering it from a different part of the chart. Try modifying the order after it’s been signed. Does the rule still behave as expected?
  4. Critique the Content: Read the alert text out loud. Is it clear, concise, and unambiguous? Is the recommendation immediately understandable? Does it provide enough information to act, but not so much that it’s overwhelming? Suggest alternative wording if you think it could be improved.
  5. Assess the Workflow: How many clicks does it take to accept the recommendation? Is the alert intrusive for the clinical severity? Would a non-interruptive reminder be better? Provide feedback on how the alert “feels” within the flow of patient care.
  6. Sign Off or Raise Concerns: At the end of your testing, you will be asked to “sign off.” Do not do this lightly. Your sign-off is your professional endorsement that you believe this rule is safe, effective, and ready for production. If you have any concerns, no matter how small, document them clearly. It is your duty to act as the final quality gate before the rule goes live.

7.3.5 The Unseen Task: Documenting Everything

If an action wasn’t documented, it didn’t happen. This core tenet of medical charting applies with equal force to CDS development. Rigorous documentation is not bureaucratic busywork; it is the auditable, legal record of your quality assurance process. It protects your patients, your institution, and your own professional integrity.

For every CDS rule you build, a corresponding set of documentation must be created and maintained. This “validation package” should be linked to the rule’s metadata and preserved for the entire lifecycle of the rule.

Masterclass Table: The CDS Validation Package
Document Content Purpose
Clinical Governance Approval Scanned copy or link to P&T / Committee minutes. Provides the official clinical authority and sign-off for the rule’s existence and its core logic. This is your “prescription” to build the rule.
Finalized Pseudo-Code The last, approved version of the plain-English logic. Serves as the master blueprint. If there is ever a dispute about how the rule is supposed to work, this document is the source of truth.
Test Plan / Test Cases The full set of positive, negative, and edge case patient profiles and scenarios you developed. Demonstrates the rigor of your validation strategy. It shows that you thoughtfully considered and tested the various ways the rule could succeed or fail.
Test Results Evidence Screenshots, log files, and/or signed attestation forms from UAT participants. This is the proof that the tests were actually performed and what their outcomes were. Screenshots of the alert firing correctly (and not firing for negative cases) are invaluable.
Deployment & Communication Record A record of the date/time the rule was activated in production and a copy of the communication sent to end-users about the new CDS. Completes the audit trail from concept to go-live.
Your Professional Safety Net

Imagine a scenario six months after your levofloxacin rule goes live. A patient experiences an adverse event, and an investigation is launched. A question arises: did the CDS rule fire correctly? Because you have a complete validation package, you can respond with absolute confidence. You can produce the P&T approval, the exact logic that was tested, the specific test cases you ran (including the one that matches this patient’s scenario), and the screenshot evidence showing the rule worked as designed during testing. This documentation is your professional safety net. It demonstrates that you followed a rigorous, evidence-based process and performed your due diligence as an informatics professional.