Section 7.4: Performance Monitoring and Audit Logs

7.4.1 The “Why”: The Pharmacist’s Duty to Monitor

As a pharmacist, your responsibility to the patient does not end when you dispense a medication. In many ways, it has just begun. The period after dispensing is when you perform one of your most critical functions: therapeutic monitoring. You follow up with the patient to ask: Is the medication working? Are you experiencing any side effects? Are you taking it correctly? You check lab values to assess efficacy and toxicity. This continuous loop of action, assessment, and adjustment is the hallmark of excellent pharmaceutical care. It ensures that the therapy which was appropriate at the time of dispensing remains safe and effective over its entire lifecycle.

Activating a new Clinical Decision Support (CDS) rule—the “go-live”—is the informatics equivalent of dispensing that first dose. And just like with a medication, your professional duty does not end there. In fact, a new and equally critical responsibility begins: performance monitoring. You must now ask of your rule the same questions you would ask of a drug therapy: Is it working as intended? Is it causing any unintended side effects (like alert fatigue)? Are clinicians “adherent” to its recommendations? The practice of analyzing CDS audit logs is the direct translation of your therapeutic monitoring skills into the informatics domain.

Ignoring this post-implementation surveillance is a form of professional negligence. A CDS rule that seemed perfect in the controlled environment of testing can have unforeseen and dangerous consequences in the chaotic reality of live clinical practice. It can under-fire, missing opportunities to prevent harm. It can over-fire, drowning clinicians in noise and eroding their trust in all alerts. It can be misinterpreted, leading to confusion. Without a robust monitoring program, your digital safety net can quickly decay into a digital nuisance or, worse, a source of iatrogenic error. This section will equip you with the data analysis skills to become a vigilant digital pharmacist, one who not only builds rules but also meticulously manages their long-term therapeutic outcomes within the EHR.

7.4.2 The Source of Truth: Anatomy of a CDS Audit Log

Every time a CDS rule is evaluated by the EHR, a digital footprint is created. This collection of footprints is the audit log. It is a vast, highly structured database that records every interaction—or potential interaction—between your rule and the end-users. Learning to read and interpret this log is the foundational skill for all performance monitoring. While the exact fields may vary by EHR vendor, all robust audit logs capture the same core concepts.

Each row in an audit log typically represents a single event: a rule evaluation, a rule firing, or a user’s interaction with an alert. It is a treasure trove of raw data waiting to be refined into clinical intelligence.

Masterclass Table: Core Fields in a CDS Audit Log

7.4.3 Key Performance Indicators (KPIs) for CDS Success

The raw audit log is overwhelming. To make sense of it, you must aggregate this transactional data into meaningful metrics. These Key Performance Indicators (KPIs) are the vital signs of your CDS rule’s health. You will monitor these KPIs over time to diagnose problems, measure the impact of changes, and report on the value of your work to hospital leadership.

KPI 1: The Fire Rate

This is the most basic measure of your rule’s activity. It answers the question: “How often is my rule’s clinical scenario encountered?”

$$ \text{Fire Rate} = \frac{\text{Total Times Rule Fired}}{\text{Total Opportunities for Firing}} $$

The “opportunity” (the denominator) can be defined in different ways depending on the rule. For a medication-specific rule, it might be the total number of orders for that medication. For a hospital-acquired pneumonia rule, it might be the total number of patients admitted with a pneumonia diagnosis. A high fire rate is not necessarily bad, nor is a low one good. It is simply a baseline measure of activity that helps diagnose other issues. For instance, if a rule for a rare drug-drug interaction fires 100 times a day, your logic is almost certainly too broad and needs to be refined.

KPI 2: The Acceptance Rate

This is the single most important measure of your rule’s clinical utility and effectiveness. It answers the question: “Do clinicians agree with and follow my recommendation?”

$$ \text{Acceptance Rate (%)} = \frac{\text{Number of Times Guidance Was Accepted}}{\text{Total Times Rule Fired}} \times 100 $$

A high acceptance rate (typically > 80-90% for well-designed rules) is the ultimate sign of success. It indicates that your rule is identifying clinically relevant scenarios, the recommendation is appropriate, and the alert is designed in a way that is easy for clinicians to act upon. A persistently low acceptance rate is a five-alarm fire; it means you are creating work and interruption for clinicians with little to no benefit, actively contributing to alert fatigue, and eroding trust in the entire CDS system. Investigating the cause of a low acceptance rate is a top priority for any informatics pharmacist.

KPI 3: Override Rate & Reason Analysis

This is the inverse of the acceptance rate, but it requires a deeper qualitative analysis. It answers the question: “When clinicians disagree with my rule, why?”

$$ \text{Override Rate (%)} = \frac{\text{Number of Times Guidance Was Overridden}}{\text{Total Times Rule Fired}} \times 100 $$

Simply knowing the override rate isn’t enough. The gold is in the override reasons. This is where clinicians tell you, in their own words, the specific clinical context where your rule’s logic falls short. A rigorous analysis involves categorizing these free-text or pre-selected reasons into themes. Do 10% of overrides for our levofloxacin alert mention “unstable renal function”? If so, that’s a powerful signal that we need to build a specific exclusion for AKI into our logic. Are users frequently selecting “Other” and typing “Patient preference”? That might indicate a need for patient counseling resources linked from the alert. This analysis is your primary tool for the iterative refinement of your CDS rules.

7.4.4 Masterclass: From Raw Data to Actionable Report

Let’s walk through a realistic scenario. One month after our `[PHA] Renal Dosing – Levofloxacin` rule goes live, we are tasked with presenting its initial performance data to the P&T committee. Our first step is to query the CDS audit log for all events related to our `Rule_ID`.

Step 1: The Raw Data Pull

We receive a raw data export from the analytics team. It’s a massive spreadsheet with thousands of rows. Here is a small, representative sample:

Step 2: Aggregate the Data & Calculate KPIs

Using spreadsheet functions or a data analysis tool, we aggregate the raw data. We find the following totals for the first month:

• Total Orders for Levofloxacin (Denominator/Opportunity): 1,250
• Total Times Rule Fired (`Evaluation_Outcome` = TRUE): 150
• Total ‘Accept’ Actions: 110
• Total ‘Override’ Actions: 40

Now, we calculate our primary KPIs:

• Fire Rate: (150 Fired / 1,250 Orders) = 12%
• Acceptance Rate: (110 Accepted / 150 Fired) = 73.3%
• Override Rate: (40 Overridden / 150 Fired) = 26.7%

Step 3: Analyze the Qualitative Data (Override Reasons)

We now perform a thematic analysis of the 40 `Override_Reason` free-text entries.

Step 4: Synthesize and Present the Story

The data tells a story. Your job is to narrate it clearly and concisely for the P&T committee. You don’t just present numbers; you present insights and recommendations.

This data-driven approach transforms you from a simple rule-builder into a strategic, evidence-based manager of the hospital’s clinical logic. You have used the audit log to identify a problem, diagnose its root cause, and propose a specific, targeted solution. This is the continuous quality improvement cycle that defines a high-performing informatics team.