Despite AI advancements, human oversight remains essential

State-of-the-art artificial intelligence systems known as large language models (LLMs) are poor medical coders, according to researchers. Their study emphasizes the necessity for refinement and validation of these technologies before considering clinical implementation. The study extracted a list of more than 27,000 unique diagnosis and procedure codes from 12 months of routine care in the Mount Sinai Health System, while excluding identifiable patient data. Using the description for each code, the researchers prompted models from OpenAI, Google, and Meta to output the most accurate medical codes. The generated codes were compared with the original codes and errors were analyzed for any patterns.