The Challenges and Benefits of Generative AI in Kidney Care

While generative Artificial Intelligence (AI) has the potential to improve kidney care, it also poses substantial challenges. Applications under discussion touch on every aspect of treatment, including the creation of new prognostic tools and methodologies, personalized medical education for professionals and patients, and protocols that alleviate the burden of administrative tasks. FME is developing an AI framework for clinical workflows that considers both the benefits and risks that AI poses for the future of patient care.

The development of generative Artificial Intelligence (AI) has created excitement and prompted vigorous debate across various industries, including healthcare.¹ Dubbed “Gen AI,” this novel technology transcends conventional rule-based systems, data analysis, and predictions. Departing from the familiar confines of traditional AI, generative AI ventures into uncharted territory where machines wield the power of creativity sans human intervention.²

Understanding Generative AI’s Unconventional Pathways

Generative AI stands apart because of its ability to create human-like content—images, text, melodies, or even entire narratives—using complex computer algorithms. These differ from conventional machine learning algorithms that can generate simple outputs. Instead, generative models create new content based on an assortment of data on which they have been trained by weaving together semantic patterns and knowledge structures.^2,3

Potential Benefits of Generative AI in Kidney Care

Generative AI systems, particularly large language models (LLMs), hold numerous potential applications and may revolutionize several aspects of healthcare.^4,5,6 

Clinical Insights and Powerful Prognostic Tools: A recent systematic review highlighted that most published studies focus on the use of LLMs as medical chatbots and to generate patient information and clinical documentation as well as for patient education and to simplify imaging reports.⁷ Generative AI and multimodal LLMs may have direct clinical applications, such as generating diagnostic ^8,9,10,11 and prognostic ^12,13 predictions, given their ability to encode medical knowledge and/or interpret medical signs and symptoms similar to semantic elements.^12,13 For instance, Kanda and colleagues utilized an early natural language processing (NLP) architecture, word2vec, to analyze chronic kidney disease (CKD) literature, accurately predicting death and end-stage kidney disease (ESKD) onset. With the advent of more advanced LLMs, coupled with fine-tuning in the medical domain, highly accurate outcome predictions can be generated directly from medical notes, referral letters, and patients’ narratives without the need to document medical encounters in structured electronic health record systems, thus reducing documentation burden and limitations due to incomplete ontologies.^12,15

Personalized Care: New LLM architectures like pretrained transformers offer broader possibilities for analyzing multimodal data and detecting nuanced associations. These advancements enable languageunderstanding technologies to learn patterns across various data types, such as comorbidity codes, lab tests, images, clinical narratives, and patient-reported outcomes. For example, Savcisens et al. demonstrated the effectiveness of this approach in predictive modeling using life-events data, showing that these models could accurately predict diverse outcomes such as early mortality and personality nuances by learning patterns from detailed event sequences.¹⁶

Departing from the familiar confines of traditional AI, generative AI ventures into uncharted territory where machines wield the power of creativity sans human intervention.²

Efficiency and Cost Savings: Generative AI can alleviate the administrative burden on healthcare staff, including time-consuming non-medical tasks.^{17,18,19,20,21,22,23} Streamlining these tasks can save time, minimize disruptions, and potentially enhance patient-clinician interactions. Studies show that LLMs can summarize medical notes and dialogues with high accuracy^.24,25 For instance, FME Global Medical Office and Santa Barbara Smart Health developed software leveraging GPT-4 to transcribe patient-physician interactions, achieving reliable abstraction of ³³ medical elements, including pre-existing medical conditions, drug prescriptions, biochemical parameters, active problems, and treatment plans. The system produced a reliable and accurate summary of medical concepts in a small proof-of-concept study. 

FME is exploring how generative AI might streamline the process of collecting patient referral information, with the potential to expedite referrals and admissions and enhance data entry accuracy. We are also investigating the development of a ChatGPT-like tool to assist staff in offering targeted guidance for handling non-clinical tasks, with the goal of reducing staff burden and supporting new clinical leaders. This includes examining how the tool could navigate intricate requirements related to Worker’s Compensation and the Conditions for Coverage for ESKD Facilities. Additionally, FME aims to reduce patient attrition and improve their experience.^26,27,28 By considering the implementation of an AI-guided referral pathway and AI-powered case management, we hope to assist FME’s Continuity of Care team in identifying patients at high risk of attrition, conducting root cause analyses, and providing data-driven insights to case managers (Figure 1).

FME is exploring how generative AI might streamline the process of collecting patient referral information, with the potential to expedite referrals and admissions and enhance data entry accuracy.

Tailored Medical Education: Personalizing medical education for healthcare professionals and patients is another promising area of application for generative AI.^26,27,28 We utilized retrieval-augmented generation (RAG), a novel AI-driven approach, to efficiently process and extract meaningful information from published literature on uremic toxins. The process involved preparing a curated literature database, creating a vector database from curated literature, retrieving relevant information based on queries, and generating responses using LLMs incorporating retrieved information. Although RAG has significantly improved content generation, the potential for “hallucinations” persists, and the enhanced LLM outputs still require human verification. For more information on the hallucination topic, refer to “Potential Risks” below.

Comprehensive Use of Data and Knowledge: Dietary management is crucial for patients with kidney failure undergoing dialysis, but personalized advice is challenging due to varying food preferences and other factors. By leveraging LLMs, there is potential to integrate patient demographics, clinical data, and food preferences to create tailored recipe recommendations^.29,30 Renal Research Institute (RRI) tested the emergent ability of LLM to generate sound nutritional advice for people with CKD (Figure 2). 

While this approach has limitations in precise nutritional analysis for people with CKD, it’s important to note that this evaluation of LLM sheds light on the current knowledge base. For instance, in RRI’s study, ChatGPT underestimated calories, protein, fat, phosphorus, potassium, and sodium content on ChatGPT-generated recipes when compared with U.S. Department of Agriculture (USDA)-approved software. These discrepancies are much smaller with online pre-defined recipes (Figure 3). While the underlying knowledge basis of GPT-4 falls short in supporting nutritional analysis for people with kidney disease, incorporating LLMs in more complex architectures may enhance the accuracy of nutritional estimation.^31,32,33

Personalizing medical education for healthcare professionals and patients is another promising area of application for generative AI.^26,27,28

Potential Risks of Generative AI in Kidney Care

Generative AI offers unprecedented potential to revolutionize patient care, diagnosis, and treatment methodologies. However, substantial risks remain. 

Biased Outputs from Training Data: Generative models learn from the data on which they are trained. If their training samples and datasets include biases, then those models can generate outputs that are ethically questionable.6 In the realm of kidney care, such biases could propagate treatment disparities or inequalities. 

Privacy and Security Concerns: Generative AI’s ability to generate synthetic data, which resembles real data, is tremendously useful in research and model training, but this capability comes with privacy implications. If the original datasets used to train the generative AI are not adequately secured, there is a risk that the synthetic data could inadvertently reveal sensitive personal information. Furthermore, machine learning systems in sensitive domains such as healthcare are particularly vulnerable to adversarial AI attacks where malicious actors can manipulate or exploit the models by introducing carefully crafted inputs to the system^.34,35 

Hallucinations in AI Responses: In the context of generative AI, “hallucinations” refer to the generation of responses that are not logically or semantically coherent or are not relevant to the input prompt. These hallucinations can occur when generative AI formulates responses based on patterns or associations it has learned from its training data without fully understanding the meaning or context of the input prompt. This could pose serious risks to patient safety and well-being if implemented without proper verification or oversight.³⁶ 

Transparency and Explainability Challenges: Unlike traditional rule-based AI systems where decision-making logic is explicit and interpretable, generative AI models often operate as “black boxes,” making it difficult for clinicians and patients to comprehend how generative AI arrived at a particular decision.³⁶ Addressing this risk requires meaningful human-AI collaboration, which involves integrating AI systems seamlessly into clinical workflows to enhance efficiency, accuracy, and patient outcomes while preserving the critical role of human expertise, empathy, and judgment in delivering high-quality care.³⁷

Generative AI offers unprecedented potential to revolutionize patient care, diagnosis, and treatment methodologies. However, substantial risks remain.

Reflecting on Possibility

In our relentless pursuit of innovation, FME recognizes the immense potential of generative AI in revolutionizing clinical workflows. However, this potential must be harnessed responsibly. At FME, we are developing a trustworthy AI framework—one that prioritizes safety, security, and ethics. Our commitment extends beyond compliance to encompass the thoughtful integration of organizational values and change management principles. In this new era of healthcare, we remain steadfast in our mission to elevate patient care while upholding the highest standards of integrity and excellence.

At FME, we are developing a trustworthy AI framework—one that prioritizes safety, security, and ethics.