No products in the cart.
MODULE 15: PROJECT MANAGEMENT FOR INFORMATICS INITIATIVES
Section 15.5: Post-Implementation Evaluation
How do you know if your project was truly a success? Learn how to define key performance indicators (KPIs) and conduct a post-implementation evaluation to measure the project’s return on investment (ROI), its impact on clinical outcomes, and to capture valuable “lessons learned” for future initiatives.
SECTION 15.5
Post-Implementation Evaluation
Measuring What Matters: Proving Value and Perfecting Practice.
15.5.1 The “Why”: Closing the Loop from Effort to Impact
In your clinical practice, the concept of “follow-up” is a cornerstone of responsible care. When you recommend a new therapy for a patient with hypertension, your work is not finished when they leave the pharmacy. The job is complete only when you have evidence that the intervention was successful: a follow-up blood pressure reading shows improvement, the patient reports no intolerable side effects, and they are successfully adhering to the regimen. This act of “closing the loop” is what separates a transactional dispenser from a clinical practitioner. It is the process of connecting your professional effort to a tangible patient outcome.
The Post-Implementation Evaluation (PIE) is the project management equivalent of this critical follow-up. A project is not truly successful simply because it “went live.” A successful go-live means the project was completed on time and on budget—a significant achievement, to be sure. But it does not answer the most important question: Did the project actually solve the problem it was designed to solve? Did the new BCMA system actually reduce medication errors? Did the new clinical decision support alert actually improve prescribing appropriateness? Did the new inventory management module actually reduce drug waste? The PIE is the formal process for answering these questions. It is the bridge between the project’s outputs (the new system, the new workflow) and its ultimate outcomes (improved safety, efficiency, and financial health).
This final phase of project management is, tragically, the one most often skipped. The pressure to move on to the next urgent initiative is immense. However, neglecting the PIE is a profound strategic error. It robs you of the ability to quantify your value to the organization. It prevents you from learning from your successes and failures, dooming you and others to repeat the same mistakes on future projects. And it fails to provide the evidence of return on investment (ROI) that leadership needs to justify funding your next great idea. Mastering the art of the post-implementation evaluation is how you transform yourself from a project manager who simply delivers projects into a strategic leader who delivers measurable, undeniable value.
Retail Pharmacist Analogy: The Six-Month Review of Your New Clinical Service
Let’s revisit the MTM and point-of-care testing service you launched. It’s been six months since the “go-live.” The service is running, patients are being seen, and things seem to be going well. Is that enough? For your manager, and for the long-term viability of the service, “seems to be going well” is not a business case. It’s time for a formal Post-Implementation Evaluation.
You don’t just guess at its success; you measure it against the goals you set in the planning phase:
- Measuring KPIs (Did We Do What We Said We’d Do?): Your initial goal was to enroll 50 diabetic patients in the first six months. You pull the reports. You’ve enrolled 62. Success! Another goal was to generate $5,000 in revenue. Your billing reports show $4,200. This is a variance you need to analyze. Why was revenue lower than projected? Were your billing codes wrong? Was the reimbursement rate lower than expected? This is critical data.
- Assessing Clinical Impact (Did We Help Patients?): You had five patients with A1c values over 9% at their initial visit. You check their six-month follow-up tests. Four of the five are now below 8%. This is powerful, tangible evidence of improved clinical outcomes that you can present to prescribers to get more referrals.
- Calculating ROI (Was It Worth the Cost?): You add up all your costs: the A1c machine ($1,500), supplies ($500), and the 100 hours of pharmacist time you and your team spent in training and setup (100 hours * $60/hr = $6,000). Total investment: $8,000. Your revenue was $4,200. Right now, your financial ROI is negative. This is not a failure; it’s a crucial finding. It prompts strategic questions: “How can we increase patient volume? Can we negotiate better reimbursement?”
- Capturing Lessons Learned (How Can We Do Better Next Time?): You sit down with your team. What were the biggest headaches? The technicians all agree that the scheduling software is clunky. You, as the pharmacist, found the documentation requirements for one insurance plan to be a nightmare. These are invaluable lessons. If you decide to launch a new cholesterol testing service, you now know you need to choose better software and build in more time for navigating payer requirements.
This structured review, backed by data, is what turns a “good idea” into a sustainable, scalable, and provably valuable part of your pharmacy practice. It’s the difference between being a busy pharmacist and a business-minded clinical leader.
15.5.2 Masterclass: Defining and Measuring Key Performance Indicators (KPIs)
A Key Performance Indicator (KPI) is a measurable value that demonstrates how effectively a project is achieving its key objectives. KPIs are the vital signs of your project’s health and success. Without them, you are flying blind, relying on gut feelings and anecdotal evidence to assess your project’s impact. Well-defined KPIs, established during the planning phase and measured during the post-implementation evaluation, provide the objective, data-driven evidence needed to declare a project a true success.
The challenge is to select the right KPIs. A common mistake is to measure everything possible, creating a “data dump” that overwhelms stakeholders and obscures the most important findings. The key is to focus on a handful of indicators that are directly and meaningfully linked to the project’s original goals as defined in the Project Charter and Scope Statement. If your project’s goal was to improve patient safety, then your KPIs must measure patient safety. If the goal was to improve efficiency, your KPIs must measure efficiency.
The Hierarchy of Metrics: Finding the Right Signal
Outcome vs. Process Metrics & Leading vs. Lagging Indicators
To choose effective KPIs, it’s helpful to understand two key distinctions:
- Outcome vs. Process Metrics:
- An Outcome Metric measures the ultimate result you are trying to achieve. It answers the question, “Did we solve the problem?” Example: Percent reduction in adverse drug events (ADEs).
- A Process Metric measures a task or activity that you believe will lead to the desired outcome. It answers the question, “Are people using the new system as intended?” Example: BCMA scanning compliance rate.
- Lagging vs. Leading Indicators:
- A Lagging Indicator is an “output” measurement. It’s easy to measure but hard to influence directly because it tells you what has already happened. ADE rate is a classic lagging indicator.
- A Leading Indicator is an “input” measurement. It’s often harder to measure but easier to influence, and it predicts future success. BCMA scanning compliance is a leading indicator; if compliance is high, you can predict that ADE rates will eventually fall.
The Takeaway: A strong PIE will measure a balanced mix of these metrics. You need process metrics and leading indicators to understand if the system is being used correctly in the short term, and you need outcome metrics and lagging indicators to prove that this usage led to the desired long-term results.
Masterclass Table: Sample KPIs for Common Pharmacy Informatics Projects
| Project Type | Primary Goal | Sample KPIs (with Metric Type) | Data Source |
|---|---|---|---|
| Barcode Medication Administration (BCMA) Implementation | Improve Patient Safety |
|
EHR Administration Logs, Safety Reporting System, User Surveys |
| New IV Workflow Management System | Improve Safety & Efficiency |
|
IV Workflow Software Reports, Safety Reporting System, Manual Timings |
| Clinical Decision Support (CDS) for Renal Dosing | Improve Prescribing Appropriateness |
|
EHR Data Analytics, Chart Review, Safety Reporting System |
| Automated Dispensing Cabinet (ADC) Optimization | Improve Efficiency & Inventory Control |
|
ADC Software Reports, Financial Reports, Manual Timings |
15.5.3 Masterclass: Calculating Return on Investment (ROI)
Return on Investment (ROI) is a performance measure used to evaluate the efficiency or profitability of an investment. In the corporate world, it’s a straightforward financial calculation. In healthcare, it’s more complex, but no less critical. Hospital leadership needs to see that the significant capital and operational resources invested in technology projects are generating tangible value. Being able to perform a credible ROI analysis is a skill that will dramatically increase your standing and influence within the organization. It allows you to speak the language of the C-suite and to build a powerful, data-driven case for your future initiatives.
The basic formula is simple. The complexity lies in accurately identifying and quantifying all the costs and all the benefits.
$$ROI (%) = \frac{(\text{Financial Gain from Investment} – \text{Cost of Investment})}{\text{Cost of Investment}} \times 100$$
Step 1: Quantifying the Total Cost of Investment
This is the most straightforward part of the calculation, but it requires meticulous tracking. The total cost is more than just the price tag of the software. It includes all resources consumed by the project.
| Cost Category | Description | BCMA Project Example |
|---|---|---|
| Hardware/Software Costs | The direct capital expenses for technology. | – 50 Handheld Scanners @ $1,200/each = $60,000 – BCMA Software License (1 year) = $50,000 |
| Vendor/Consultant Fees | External costs for implementation, training, or project management support. | – Vendor Implementation Services = $25,000 |
| Internal Labor Costs | The most commonly forgotten cost. This is the cost of the time that your internal team (including you) spends on the project. It’s calculated by multiplying hours worked by a “blended” hourly rate. | – Project Lead (You): 400 hrs @ $65/hr = $26,000 – EHR Analyst: 300 hrs @ $55/hr = $16,500 – Nurse Champion: 150 hrs @ $50/hr = $7,500 |
| End-User Training Costs | The cost of pulling end-users away from their normal duties for training. | – 200 Nurses * 4 hours of training * $50/hr = $40,000 |
| Total Investment Cost | The sum of all of the above costs. | $60k + $50k + $25k + $26k + $16.5k + $7.5k + $40k = $225,000 |
Step 2: Quantifying the Financial Gain (The Benefits)
This is the most challenging and most creative part of the analysis. You must translate the clinical and operational improvements from your KPIs into a monetary value. Benefits fall into two categories: hard savings and soft savings (also called cost avoidance).
Hard Financial Savings
These are direct, measurable reductions in actual cash expenditures that appear on a budget line item. They are the easiest to defend.
- Reduced drug waste from expired meds.
- Elimination of temporary staff hired to cover inefficiencies.
- Lower vendor fees from a retired legacy system.
Soft Savings / Cost Avoidance
These are measurable financial gains that result from avoiding a potential future cost or improving productivity. They require making credible assumptions.
- Preventing Adverse Drug Events (ADEs): The most powerful soft saving. Assigning a believable average cost to an ADE and then calculating the savings from reducing them.
- Improved Staff Productivity: Quantifying the value of time saved by nurses and pharmacists.
- Reduced Length of Stay (LOS): Attributing a reduction in patient days to the project’s impact.
The Credibility of Your Assumptions is Everything
When calculating soft savings, your entire analysis rests on the credibility of your assumptions. You cannot simply invent a number. You must base your assumptions on published literature, industry benchmarks, or internal historical data. For example, to calculate the cost of an ADE, you should cite a peer-reviewed study (e.g., “According to the Journal of Patient Safety, the average preventable ADE adds approximately $8,750 to the cost of a hospital stay…”). Documenting your sources is non-negotiable.
Putting It All Together: A Sample ROI Calculation for the BCMA Project
Time Period: The first year post-implementation.
Total Investment Cost: $225,000 (from above)
Benefit Calculation:
-
Cost Avoidance from Reduced ADEs:
- Pre-BCMA, the hospital had 50 preventable, wrong-medication errors per year.
- Post-BCMA, your KPI measurement shows this has been reduced by 80% to 10 errors per year.
- You prevented 40 ADEs.
- Using a literature-based cost of $8,750 per ADE.
- Annual Savings: 40 ADEs * $8,750/ADE = $350,000
-
Productivity Gains from Nursing Efficiency:
- You conducted a time-motion study showing the new workflow saves each nurse 10 minutes per shift.
- You have 200 nurses working, on average, 200 shifts per year.
- Total time saved: 200 nurses * 200 shifts * 10 min/shift = 400,000 minutes = 6,667 hours per year.
- Using a blended nursing rate of $50/hour.
- Annual Savings: 6,667 hours * $50/hr = $333,350
Total Annual Financial Gain: $350,000 (ADE Savings) + $333,350 (Productivity) = $683,350
ROI Calculation:
$$ROI (%) = \frac{($683,350 – $225,000)}{$225,000} \times 100 = \frac{$458,350}{$225,000} \times 100 \approx textbf{203.7%}$$
The Conclusion: You can now confidently report to leadership that the BCMA project delivered a 204% return on investment within the first year, in addition to its significant, unquantifiable benefits to patient safety and staff morale.
15.5.4 Masterclass: The Lessons Learned Process: The Engine of Continuous Improvement
While measuring KPIs and calculating ROI proves the value of your project, the Lessons Learned process is what makes the organization smarter for having done it. It is the final, crucial step in project closure, and its purpose is to formally capture, document, and disseminate the knowledge gained during the project for the benefit of all future initiatives. It is a systematic process for reflecting on the project journey to understand what went right, what went wrong, and, most importantly, why. A well-executed lessons learned session transforms individual experience—both positive and negative—into a durable, accessible organizational asset.
The most significant barrier to an effective lessons learned process is a culture of blame. If team members feel that the session is a witch hunt designed to assign fault for missed deadlines or budget overruns, they will be defensive, guarded, and dishonest. As the project lead and facilitator, your most important job is to establish a psychologically safe environment. You must relentlessly frame the discussion as a blameless, forward-looking exercise. The goal is not to punish past mistakes but to prevent future ones. It’s about improving the process, not judging the people.
Setting the Ground Rules for a Blameless Post-Mortem
Before you begin any lessons learned session, you must explicitly state the ground rules. Write them on a whiteboard for all to see.
- Rule #1: The Prime Directive. “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” (Norm Kerth, “Project Retrospectives: A Handbook for Team Reviews”)
- Rule #2: Focus on Process, Not People. We do not use names. We say, “The vendor communication process was challenging,” not “John was bad at communicating with the vendor.”
- Rule #3: Look Forward, Not Back. For every issue identified, the key question is not “Whose fault was this?” but “How can we, as an organization, prevent this from happening on the next project?”
- Rule #4: Everyone Participates. Every perspective is valuable, from the project sponsor to the frontline end-user. There is no hierarchy in a lessons learned session.
The “Start, Stop, Continue” Framework: A Simple, Powerful Agenda
While there are many ways to structure a lessons learned meeting, one of the most effective and straightforward is the “Start, Stop, Continue” framework. It focuses the conversation on concrete, actionable recommendations. You can structure the meeting by going through each project phase (Initiation, Planning, etc.) and asking the team three simple questions for each:
Masterclass Table: Facilitating a “Start, Stop, Continue” Session
| Question | Facilitator’s Goal | BCMA Project Examples |
|---|---|---|
| What should we START doing? | Identify positive behaviors or processes that were missing from this project but should be incorporated into all future projects. This is about discovering new best practices. |
|
| What should we STOP doing? | Identify processes, behaviors, or tools that were ineffective, caused problems, or provided negative value. This is about eliminating waste and sources of failure. |
|
| What should we CONTINUE doing? | Identify the things that went well and were critical to the project’s success. This is about recognizing and reinforcing best practices that already exist. |
|
The Final Deliverable: The Lessons Learned Report
The conversation is valuable, but its long-term impact depends on documentation. The output of the session should be a concise Lessons Learned Report. This report should not be a minute-by-minute transcript of the meeting. It should be a curated, actionable summary that includes:
- Project Overview: A brief reminder of the project’s goals and outcomes.
- Summary of Successes: What were the major achievements of the project?
- Summary of Challenges: What were the major obstacles encountered?
- Lessons Learned Analysis: The synthesized list of “Start, Stop, Continue” recommendations, categorized by project phase or knowledge area (e.g., Scope Management, Communication, Risk Management).
- Actionable Recommendations: For the most critical lessons, this section should include a specific recommendation and an “owner” responsible for seeing that the process improvement is implemented at an organizational level (e.g., “Recommendation: The PMO will update the standard project charter template to include a mandatory section on Wi-Fi infrastructure assessment. Owner: Director of PMO.”).
This document should be formally signed off by the project sponsor and then archived in a central, accessible repository—a “lessons learned library”—where it can be easily found and reviewed by project managers who are initiating new projects. This final act is what truly closes the loop, ensuring that the hard-won knowledge from your project pays dividends for the entire organization for years to come.
