CPIA Module 14, Section 5: Ethical and Regulatory Implications of AI
MODULE 14: EMERGING TECHNOLOGIES & ARTIFICIAL INTELLIGENCE

Section 5: Ethical and Regulatory Implications of AI

A critical examination of the challenges that come with powerful technology. We will discuss issues of algorithmic bias, data privacy, the “black box” problem of unexplainable AI, and the evolving regulatory landscape for AI-driven clinical decision support tools.

SECTION 14.5

Ethical and Regulatory Implications of AI

Navigating the Human Consequences of Algorithmic Medicine.

14.5.1 The “Why”: The Hippocratic Oath for Algorithms

Throughout this module, we have explored the immense potential of AI and machine learning to revolutionize pharmacy practice. We’ve seen how these tools can predict risk, unlock insights from clinical notes, and optimize complex robotic workflows. The power is undeniable. But as with any powerful intervention—be it a new blockbuster drug or a sophisticated surgical technique—with great power comes an even greater responsibility. The core principles that have guided medical practice for centuries do not become obsolete in the age of AI; they become more critical than ever.

We must begin to think in terms of a Hippocratic Oath for algorithms. How do we ensure that these complex, data-driven systems adhere to the same ethical tenets we demand of ourselves as clinicians?

  • Beneficence (Do Good): Does the AI tool genuinely improve patient outcomes, or does it just add noise and create more work for clinicians?
  • Non-maleficence (Do No Harm): Could the AI tool’s recommendations, if flawed or biased, lead to patient harm? How do we safeguard against this?
  • Autonomy (Patient’s Right to Choose): How are patients informed that AI is being used in their care? How do we ensure the final decision remains with the patient and their clinician, not the algorithm?
  • Justice (Fairness): Does the AI tool work equally well for all patient populations, regardless of their race, gender, or socioeconomic status? Or does it risk amplifying existing health disparities?

These are not abstract philosophical questions. They are practical, operational challenges that you, as a pharmacy informatics analyst, will be on the front lines of addressing. An AI model is not a neutral, objective calculator. It is a product of the data it’s trained on, the choices made by its developers, and the context in which it’s deployed. It can inherit and even amplify the biases present in our healthcare system. It can make mistakes. It can be difficult to understand.

Therefore, your role must expand to that of a clinical ethicist for algorithmic systems. You are the crucial guardian at the intersection of technology and patient safety. You are the professional who is uniquely positioned to understand both the clinical pharmacology and the technical underpinnings of these tools, enabling you to ask the tough questions, demand transparency, and design the safeguards necessary to deploy them responsibly. This final section is arguably the most important in the module. It provides the critical framework you will need to ensure that as we integrate these powerful technologies into care, we do so with a steadfast commitment to our primary ethical obligation: the well-being of our patients.

Retail Pharmacist Analogy: The New “Miracle” Generic Launch

Imagine a major pharmaceutical company releases a new generic version of a blockbuster drug—let’s say, a new generic for apixaban. The sales rep tells you it’s a “miracle”: it’s 50% cheaper and a clinical trial showed it’s just as effective. This new generic is your new AI model—it promises better outcomes at a lower cost.

Does a responsible pharmacist immediately switch every single patient to the new generic without question? Absolutely not. You exercise your professional, ethical judgment. You apply a critical filter.

Your Ethical Evaluation Checklist:

  • Investigating Bias (Justice): You ask, “Who was in the clinical trial?” Was it only tested on young, healthy men? How do I know it will work the same in my 85-year-old female patient with renal impairment? You are questioning if the “training data” for the drug is representative of your actual patient population. This is the same as auditing an AI model for algorithmic bias.
  • Evaluating Explainability (Autonomy/Trust): The new pill is a different shape and color. You know that for your elderly, cognitively impaired patient, this change could cause confusion and lead to non-adherence or double-dosing. You need to be able to explain the change to the patient and their caregiver. If you don’t understand the potential downstream effects of the switch, you can’t obtain true informed consent. This is the “black box” problem. You need to understand the “why” behind the change.
  • Monitoring for Harm (Non-maleficence): You learn the new generic uses a different dye as an inactive ingredient. You check your patients’ allergy profiles. Does anyone have a documented allergy to this specific dye? You are proactively looking for potential, unforeseen adverse events before they happen. This is the same as post-deployment monitoring of an AI model to ensure it isn’t causing unintended harm.
  • Regulatory Compliance: Is this new generic fully FDA approved with an AB rating? Or is it from a gray-market overseas supplier with questionable documentation? You are checking its regulatory status to ensure it meets established safety standards.

Deploying a new AI model is exactly like managing this new generic launch. An informatics pharmacist cannot simply take the data scientist’s word that the model is “95% accurate.” You must act as the last ethical and clinical checkpoint. You have to ask who it was trained on, how it makes its decisions, how it might harm specific subpopulations, and whether it meets the regulatory standards for clinical use. You are, in effect, performing a “drug information” review on the algorithm itself before dispensing it into the clinical workflow.

14.5.2 The Specter of Bias: When Data Becomes Destiny

Of all the ethical challenges in clinical AI, algorithmic bias is the most insidious and potentially harmful. We often think of computers as being objective and neutral, but an AI model is only as objective as the data it learns from. Since our healthcare system has a long history of systemic biases and disparities in care, a model trained on historical data will inevitably learn, codify, and—most dangerously—amplify those same biases at a massive scale.

Algorithmic bias is defined as systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one arbitrary group of users over others. In healthcare, this can translate to a model consistently underestimating the health risks of minority populations, recommending less effective treatments for women, or misdiagnosing conditions in certain ethnic groups. As an informatics pharmacist, being able to identify the sources of bias is a critical skill.

Masterclass Table: Sources of Bias in the AI Lifecycle
Type of Bias When It Occurs Description Concrete Pharmacy Example
Historical Bias Data Generation The data itself reflects existing societal prejudices and historical inequities, even if it is measured perfectly. An AI model predicting opioid use disorder risk is trained on historical data. If, historically, clinicians were more likely to screen and diagnose low-income patients for substance use than affluent patients, the model will learn that socioeconomic status is a key predictor, unfairly flagging poorer patients even if their clinical behavior is identical to wealthier ones.
Representation Bias Data Collection / Sampling The data used to train the model is not representative of the target population. Certain groups are under-sampled or missing entirely. A new dosing algorithm for a novel anticoagulant is developed using clinical trial data in which 80% of the participants were white males. When this algorithm is deployed in a diverse hospital system, it may perform poorly and make unsafe recommendations for female or non-white patients whose physiology was not well-represented in the training data.
Measurement Bias Feature Engineering The feature chosen as a proxy for a real-world outcome is flawed or is measured differently across groups. A hospital wants to build a model to predict medication “adherence.” They choose to measure this by using data from a smart pill bottle that records every time it is opened. This method is biased against patients who use weekly pill organizers. These patients are perfectly adherent but will be measured as “non-adherent” by the system, leading the model to learn incorrect patterns.
Evaluation Bias Model Evaluation The model’s performance is only evaluated on an overall basis, hiding poor performance on specific subgroups. A new AKI prediction model reports 95% overall accuracy. However, when the informatics pharmacist insists on breaking down the results by race, they discover the model is 98% accurate for white patients but only 75% accurate for Black patients. The overall average hid a critical, inequitable failure.
Deployment Bias Implementation The model is technically sound, but the way it is integrated into the workflow creates biased outcomes. A readmission risk model provides scores to a discharge team. However, due to staffing constraints, the team only has time to act on the top 10 patients on the list. If the model’s scores are even slightly correlated with race or insurance status, the “top 10” will consistently over-represent certain groups, meaning other high-risk patients are systematically ignored simply due to their position on the list.
The Case of the Biased Kidney Function Equation

A famous real-world example of embedded bias is the traditional equation for estimating glomerular filtration rate (eGFR), which included a “race coefficient” that adjusted the eGFR upwards for patients identified as Black. This was based on the outdated and flawed assumption of biological differences in muscle mass.

Now, imagine an AI model trained on data from a hospital that used this biased equation. The model would learn from tens of thousands of examples that Black patients “have better kidney function” than their lab values would otherwise suggest. This could lead the model to systematically recommend higher, potentially toxic doses of renally-cleared drugs to Black patients, directly causing harm. This is a powerful example of how historical biases, even those with once-purported clinical justifications, can be learned and amplified by AI systems with dangerous consequences. As an informatics pharmacist, you must be vigilant in identifying and questioning these legacy biases in the data your models are trained on.

14.5.3 The “Black Box” Problem: The Crisis of Explainability and Trust

As we discussed in Section 14.1, there is often a trade-off between a model’s performance and its interpretability. The most powerful models, particularly deep learning neural networks, are often referred to as “black boxes.” They can take in thousands of features and produce a highly accurate prediction, but their internal decision-making process is so complex that it is virtually impossible for a human to understand exactly *why* a specific prediction was made. This presents a profound challenge to the adoption of AI in medicine.

Imagine a pharmacist receives an AI-driven alert: “Warning: Patient John Smith’s predicted risk of bleeding on apixaban is 92%. Recommend alternative therapy.” The pharmacist’s immediate, professional response is to ask, “Why?” If the system’s only answer is, “Because the 5,000 parameters of my neural network say so,” that is clinically unacceptable. Without an explanation, the clinician cannot:

  • Trust the recommendation: Is the model picking up on a real clinical signal or is it an error?
  • Verify the reasoning: Is the model’s logic clinically sound? Perhaps it’s basing its prediction on a lab value that is old or erroneous.
  • Learn from the model: If the model is seeing a pattern the human has missed, the explanation itself is a powerful educational tool.
  • Responsibly override the model: If the model’s reasoning is flawed, the clinician needs to be able to confidently override the recommendation.

This need for transparency has given rise to a critical subfield of AI research known as Explainable AI (XAI). The goal of XAI is not to make the complex models simpler, but to build a second layer of interpretation on top of them that can provide human-understandable explanations for their predictions.

Masterclass: An Introduction to XAI Techniques

As an informatics pharmacist, you will be a primary consumer of these explanations. You don’t need to code the algorithms, but you must understand what they are telling you.

LIME (Local Interpretable Model-agnostic Explanations)

LIME works by creating a simple, interpretable “local” model to explain a single prediction from a complex black box model. Imagine the black box model’s decision boundary is a very complex, curvy line. LIME essentially says, “I can’t explain the whole curvy line, but for this one specific patient (this one point in the data), I can approximate the line with a simple straight line right here.”

The Output: For the apixaban bleed risk alert, a LIME explanation might look like a simple bar chart showing the top 5 features that contributed to that specific prediction:

  • `Serum Creatinine > 2.5 mg/dL`: Pushed risk up by 30%
  • `Concomitant NSAID use`: Pushed risk up by 25%
  • `Age > 85`: Pushed risk up by 15%
  • `History of GI Bleed`: Pushed risk up by 12%
  • `Patient is on a PPI`: Pushed risk down by 10%
This is immediately understandable and clinically actionable for the reviewing pharmacist.

SHAP (SHapley Additive exPlanations)

SHAP is a more sophisticated method based on game theory. It explains a prediction by treating the features as “players” in a game, and it calculates the contribution of each “player” to the final outcome. SHAP values can provide not only local explanations (for a single patient) but also global explanations (how features impact the model’s predictions on average).

The Output: A SHAP “force plot” provides a powerful visualization for a single patient. It shows a baseline risk for the population, and then visually depicts each feature as an arrow that either pushes the risk higher (red arrows) or lower (blue arrows), arriving at the final prediction. This gives the clinician an intuitive feel for the competing factors that the model weighed to make its decision. This transparency is essential for building trust and allowing for informed clinical judgment.

14.5.5 The Evolving Regulatory Landscape

As AI tools become more integrated into clinical decision-making, they are drawing increasing scrutiny from regulatory bodies like the U.S. Food and Drug Administration (FDA). As an informatics pharmacist, you must have a working knowledge of this regulatory framework, as it will govern how AI models are developed, validated, and maintained in your institution.

The central regulatory concept is Software as a Medical Device (SaMD). The FDA defines SaMD as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. Many clinical AI tools, especially those that provide a diagnosis or a specific treatment recommendation, fall under this definition.

The FDA’s Risk-Based Classification for SaMD

The FDA uses a risk-based approach to regulate SaMD, classifying tools based on the seriousness of the medical situation they are used for and the significance of the information they provide to the healthcare decision.

SaMD Class Level of Risk Description Pharmacy AI Example Regulatory Scrutiny
Class I Low Software that informs clinical management, but does not diagnose or treat. Often used for simple calculations or organizing data. An app that calculates a patient’s creatinine clearance based on the Cockcroft-Gault equation. Lowest. Often exempt from premarket review but must follow basic quality standards.
Class II Medium Software that drives clinical management. It provides information that is used to make treatment decisions, but a clinician is expected to independently review and verify it. Our OIRD predictive model that provides a risk score to a pharmacist for review. The model isn’t making the decision, it’s providing information to help the pharmacist make a better one. Moderate. Requires premarket notification (a 510(k) submission) to demonstrate it is “substantially equivalent” to an existing legal device. Requires robust validation and quality system documentation.
Class III High Software that treats or diagnoses. These are high-risk devices where an error could result in serious patient harm or death. A “closed-loop” algorithmic system that automatically adjusts a patient’s insulin or heparin drip rate based on real-time data without a clinician in the loop for each adjustment. Highest. Requires a full Premarket Approval (PMA) application, which is akin to the process for a new high-risk drug and requires extensive clinical trial data to prove safety and efficacy.
The Challenge of a “Locked” vs. “Continuously Learning” Algorithm

A major regulatory challenge is that machine learning models are not static. Their performance can change or “drift” over time as patient populations and clinical practices evolve. Furthermore, we often want to retrain models on new data to improve them.

Traditional FDA approval is for a “locked” device. If you change it, you have to get it re-approved. This doesn’t work well for AI. To address this, the FDA has proposed a new framework based on a “Predetermined Change Control Plan.”

Under this model, when a developer submits an AI tool for approval, they also submit a detailed plan that specifies exactly how they will manage future model changes. This plan would include:

  • The specific types of modifications they plan to make (e.g., retraining on new data).
  • The methodology they will use to implement and validate those changes.
  • A commitment to ongoing performance monitoring and transparency.
If this plan is approved, the developer can then update and improve their model within the pre-agreed boundaries without needing to go through a full re-approval process for every single change. As an informatics pharmacist managing these tools, you will be responsible for executing and documenting this change control process within your health system.