Section 11.4: Maintenance, Downtime Procedures, and Safety Checks
A deep dive into the operational realities of managing automation. This section covers preventative maintenance, developing robust downtime procedures to ensure patient safety, and performing critical quality assurance and safety checks.
Maintenance, Downtime Procedures, and Safety Checks
From Technology User to Systems Guardian: Building a Resilient and High-Reliability Operation.
11.4.1 The Strategic Imperative of Resilience
In the previous sections, we have focused on the immense promise of automation: its ability to enhance safety, improve efficiency, and elevate the roles of your staff. We have architected new workflows and planned for the human side of change. Now, we must confront a stark and unavoidable reality: your technology will fail. Not “if,” but “when.” A server will crash, a mechanical arm will jam, a network connection will drop, the power will go out. In that moment, all the remarkable capabilities of your multi-million dollar automation will vanish.
How your pharmacy department functions in that moment of failure is the true test of your leadership and the ultimate measure of your operational design. The novice leader views maintenance and downtime as inconvenient, costly chores. The expert leader understands that building resilience—the capacity to maintain safe patient care in the face of system failure—is a core strategic function. A pharmacy that is merely technology-dependent is fragile; when the technology breaks, the pharmacy breaks. A pharmacy that is technology-enabled is resilient; it leverages automation for routine operations but has the plans, tools, and training to seamlessly revert to an intelligent manual process when required. Patient care never stops.
This section is dedicated to the unglamorous but absolutely essential work of building that resilience. We will explore the three pillars of a high-reliability technology program: proactive preventative maintenance to minimize failure, robust downtime procedures to manage failure when it occurs, and continuous safety checks to guard against the new types of errors that automation can introduce. Mastering these disciplines is what transforms you from a manager of a pharmacy that uses technology into the guardian of a truly resilient, high-reliability medication-use system.
Retail Pharmacist Analogy: The Airline Pilot’s Emergency Checklist
Imagine you are an airline pilot. 99.9% of your job is managing a highly automated system—the autopilot. The technology flies the plane with incredible precision and efficiency. This is your pharmacy automation in its normal, productive state.
However, a huge portion of your training, and what makes you a professional pilot, is not about managing the autopilot. It’s about what you do when the autopilot fails. Suddenly, an alarm blares—engine failure. Do you panic? Do you start randomly flipping switches? No. You and your co-pilot immediately and calmly pull out a physical, laminated checklist—the Quick Reference Handbook (QRH). This is your downtime procedure binder.
You have practiced this exact scenario dozens of times in a simulator. You know the procedure by heart, but you follow the checklist anyway, because under pressure, even experts can miss a step. You execute a series of pre-planned actions: you communicate with air traffic control (your command center communication), you re-route to the nearest airport, and you manually fly the plane to a safe landing. The passengers may have been scared, but they were never in unacceptable danger, because you had a robust, practiced plan for failure.
Furthermore, you didn’t just get in the plane and fly. Before every single flight, you perform a meticulous pre-flight walk-around inspection (your daily safety checks). And every 100 flight hours, the plane is taken into a hangar for deep inspection by mechanics (your preventative maintenance). Your entire profession is built on a culture of proactive maintenance, continuous safety checks, and rigorously practiced plans for managing failure. This is the exact culture you must build around your pharmacy’s technology.
11.4.2 Proactive Maintenance and Vendor Partnerships: The Foundation of Uptime
The most effective way to manage downtime is to prevent it from happening in the first place. A proactive, disciplined approach to preventative maintenance (PM) is the single most important factor in ensuring the long-term reliability and uptime of your automation. This is not an “if we have time” task; it is a core operational function that must be scheduled, resourced, and protected. It involves a partnership between your internal team, who handles the daily and weekly upkeep, and the vendor’s field service engineers, who perform the more complex, periodic maintenance.
Preventative Maintenance (PM): The Non-Negotiable Routine
PM is a scheduled series of tasks—cleaning, calibration, lubrication, diagnostics, and replacement of wear-and-tear parts—designed to keep the equipment operating within its specified parameters and to identify potential problems before they cause an unplanned outage. Your vendor will provide a detailed PM schedule. Your job as a manager is to integrate these tasks into your team’s standard work and hold them accountable for their completion.
Masterclass Table: Sample PM Schedule for a Medication Carousel
| Frequency | Responsible Party | Key Tasks | Documentation & Purpose |
|---|---|---|---|
| Daily | Designated Pharmacy Technician (Super-User) |
|
A simple daily checklist, initialed by the technician. Purpose: Ensures the basic user interface is clean and functional, preventing minor issues from escalating. |
| Weekly | Lead Technician or Pharmacy Informatics Specialist |
|
Signed and dated logbook. Purpose: A deeper health check to identify subtle signs of wear or software glitches before they cause a full shutdown. |
| Quarterly/Semi-Annually | Vendor Field Service Engineer (FSE) |
|
A formal service report from the vendor, which you must review and file. Purpose: The deep, expert-level maintenance required to ensure the long-term mechanical and software integrity of the system. This is what your maintenance contract pays for. |
Managing the Vendor Service Relationship
Your relationship with your vendor does not end after the sale. You are now partners in maintaining a critical piece of hospital infrastructure. The foundation of this partnership is the Service Level Agreement (SLA), which is a core component of your annual maintenance contract. The SLA is the vendor’s legally binding promise regarding the level of service and support they will provide. You must understand, negotiate, and hold your vendor accountable to the terms of your SLA.
Decoding Your Service Level Agreement (SLA)
When reviewing your maintenance contract, these are the key metrics to focus on:
- Guaranteed Uptime: The percentage of time the equipment is guaranteed to be operational. A typical guarantee for critical automation is 99.5% to 99.8%. You must track this, and your contract should include financial penalties (e.g., a credit on your next maintenance bill) if the vendor fails to meet this guarantee.
- Telephone Response Time: The maximum time it will take for a qualified support technician to call you back after you’ve placed a service call. For a critical system, this should be 30 minutes or less, 24/7/365.
- On-Site Response Time: The maximum time it will take for a Field Service Engineer to be physically at your hospital’s door after it’s been determined that a remote fix is not possible. This is highly dependent on your geography, but for a critical system, you should negotiate for 4-8 hours.
- Parts Availability: The contract should guarantee that the vendor maintains a local or regional depot of common replacement parts to avoid lengthy shipping delays.
11.4.3 Masterclass in Downtime Planning: Engineering for Failure
Downtime planning is a discipline of proactive, pessimistic imagination. You must operate under the assumption that every component of your technology ecosystem will fail at the worst possible moment, and you must have a clear, pre-defined, and practiced response ready. A well-written downtime procedure is one of the most important documents in your department. It ensures that in a moment of high-stress crisis, your team can shift from confusion to calm, coordinated action, ensuring that patient care continues safely and without interruption.
The Downtime Procedure Binder: Your “Red Book” of Resilience
Your downtime procedures cannot live on a shared network drive. If the hospital network is the cause of the downtime, your procedures would be inaccessible. Every pharmacy must maintain a physical Downtime Procedure Binder—often a bright red binder so it’s easily identifiable in an emergency. Multiple copies should exist in key locations. This binder is your operational insurance policy.
Masterclass Table: Anatomy of the Downtime Binder
| Section | Contents | Critical Purpose |
|---|---|---|
| 1. Emergency Call Tree | A single, laminated page with the names and 24/7 contact numbers for every person to be notified, in order. Includes: Pharmacy Manager, Director of Pharmacy, Internal IT On-Call, Vendor 24/7 Support Line, Key Super-Users. | Eliminates the chaos of “who do we call first?” in the first five minutes of a crisis. |
| 2. Downtime Activation Protocol | A one-page flowchart that defines what constitutes a “downtime event” and who is authorized to officially declare it and activate the Command Center. | Prevents premature or delayed activation of the full downtime response. |
| 3. System-Specific Procedures | Detailed, step-by-step, checklist-style procedures for every single piece of automation. (e.g., “ADC Unplanned Downtime,” “Carousel Unplanned Downtime”). These are the heart of the binder. | Provides the specific, actionable playbook for the frontline staff to follow for any given failure scenario. |
| 4. Manual Forms & Labels | A large supply of pre-printed paper forms: manual medication order forms, patient charging forms, controlled substance administration records, downtime medication labels. | Provides the physical tools needed to operate a fully manual pharmacy. You can’t be hunting for a printer or a template during a crisis. |
| 5. System Recovery Checklists | Step-by-step checklists for what must be done when the system comes back online. Includes steps like reconciling manual charges, resolving controlled substance count discrepancies, and clearing transaction queues. | Ensures a safe and orderly return to normal operations and prevents data loss or billing errors during the recovery phase. |
Deep Dive: The Catastrophic Carousel Downtime Scenario
It’s 2:00 PM on a busy Tuesday. The medication carousel, which holds 80% of your central pharmacy inventory, goes down with a critical mechanical failure. The vendor estimates an 8-hour repair time. This means the system will be down through the evening ADC restock and the first-dose rush for evening admissions. This is a full-blown operational crisis.
The First 15 Minutes: Coordinated Crisis Response
Step 1: Activate the Call Tree
The lead technician immediately notifies the pharmacy manager. The manager pulls out the Downtime Binder and starts the call tree, notifying the Director, IT, and the vendor’s support line to get an official ticket number and ETA.
Step 2: Communicate Broadly
The manager or director sends a hospital-wide broadcast page/email: “ATTENTION: The central pharmacy medication carousel is experiencing an unplanned downtime. Please anticipate delays in medication delivery. Pharmacy will be triaging all orders for clinical urgency. Please call the pharmacy for any STAT needs.”
Step 3: Establish a Command Center
The manager designates a specific area of the pharmacy as the command center. A pharmacist is assigned to staff the phones, log all incoming requests, and triage them by clinical urgency (e.g., a STAT antibiotic for a septic patient is #1 priority; a routine refill of a maintenance med is lower).
Step 4: Revert to Full Manual Workflow
All hands on deck. The team pulls out the downtime forms. Technicians are assigned as “runners” to manually search the static “downtime stash” and other areas for the triaged, high-priority medications. Every dose leaving the pharmacy is documented on a paper form.
11.4.4 The Guardian’s Role: Ongoing Safety and Quality Assurance Checks
The implementation of automation is not the end of the patient safety journey; it is the beginning of a new chapter. Automation brilliantly mitigates the risk of certain types of human error (e.g., picking the wrong drug from a shelf), but it can introduce new, more systemic risks if not managed with constant vigilance. The most insidious of these risks is automation complacency—the tendency for human operators to over-trust the technology, becoming less vigilant and critical in their own checks. As a leader, your role is to design and implement a robust Quality Assurance (QA) program that actively guards against this complacency and catches the new failure modes unique to an automated environment.
The Silent Killer: Automation Complacency
Automation complacency occurs when a system is highly reliable, leading users to stop performing their own verifications. A classic example is a technician who, after seeing the carousel present the correct drug 10,000 times in a row, stops performing a final glance at the label before placing it in the bin, assuming the machine is infallible. It is in that 10,001st time—when a stocking error or a system glitch occurs—that an error can reach the patient.
Your QA program must be designed to break this cycle. It should be built on the principle of “trust, but verify.” We trust the machine to be accurate, but we implement independent, periodic checks to verify that it is. This culture of respectful skepticism is a hallmark of a high-reliability organization.
Masterclass Table: The Pharmacy Automation QA Program
| QA Check | What It Prevents | Methodology & Frequency | Managerial Responsibility |
|---|---|---|---|
| ADC Barcode Refill Verification | The single most common ADC error: a technician placing the wrong medication into an ADC pocket during the manual refill process. | This is a workflow design requirement. The ADC system must be configured to require a barcode scan of the medication package before the system will open the pocket for refilling. | Ensure this feature is enabled and that staff are not using workarounds (e.g., scanning from a “cheat sheet” of barcodes). Audit this process regularly. |
| ADC Blind Counts for Controlled Substances | Diversion and count discrepancies. A standard “say-back” count (“The screen says 5, I see 5”) is prone to confirmation bias. | Configure the ADC to require a “blind count” for controlled substances. The user must count the quantity in the pocket and enter it into the system before the system reveals the expected quantity. | Work with your vendor to enable this functionality. It is a best practice strongly recommended by the DEA and ISMP. |
| Quarterly ADC Pocket Audits | Stocking errors that may have bypassed the barcode check, or medications that have become damaged or expired within the pocket. | On a rotating basis, a technician or pharmacist physically opens a selection of drawers on each ADC and verifies the contents against the system’s inventory record. This is a physical spot-check. | Develop a schedule to ensure every ADC is audited at least quarterly. Keep detailed logs of these audits for regulatory review. |
| Robot/Packager Canister Fill Verification | A catastrophic error where a canister is filled with the incorrect NDC, which would then be dispensed repeatedly. | Implement a mandatory, pharmacist-led, two-person verification process for every canister fill. The pharmacist must visually inspect the bulk manufacturer bottle and the canister and co-sign a logbook or electronic record before the technician is permitted to pour the medication. Barcode scanning can augment, but not replace, this visual check. | This must be an ironclad policy with zero exceptions. It is one of the most critical safety steps in the entire pharmacy. Audit your logs to ensure compliance. |
| Daily Override Report Review | Inappropriate overrides from ADCs that bypass pharmacist order review, potentially leading to patient harm from wrong doses, allergies, or contraindications. | A clinical pharmacist must review 100% of the previous day’s override transactions. For each override, the pharmacist verifies that a valid physician order was eventually entered and that the medication was clinically appropriate. | Ensure this task is assigned as a standard daily duty. Track and trend override data to identify specific units or individuals with high override rates, which may indicate a need for education or a formulary adjustment in that ADC. |