Patient Medical Language

Overview

‍
The first step for a patient medical language model is to identify a safe structure for deployment and usage. A language model like this should NEVER attempt to replace a qualified medical professional (oftentimes illegal). Rather, this model should significantly speed up or otherwise assist existing human processes. For example, this model could help a medical safety team more quickly identify a questionable case, or it could help an overloaded administrative team more automatically prioritize a majority of time-sensitive inbound messages.

Product Iteration

‍
Next, if possible, a product iteration plan should be built that explains what would suffice for v0 and what would be expected from subsequent incremental development.

Training and Testing Data

‍
The next step would be to obtain a training/testing data set, consisting of messages and text fragments along with their appropriate classification. In the case where this doesn’t exist, a number of options are available, including partially synthesizing a data set.

Initial Experiments

‍
The next step would be to run initial experiments on the training data set using a variety of methods, some based on LLMs (like ChatGPT) and others on more “traditional” natural language processing techniques. Early results can be analyzed to determine the most suitable method(s) for fine tuning and detailed evaluation. The method ultimately favored for production might be evaluated not just on accuracy but also on auditability.

Deployment Interface

‍
Deployment interface design can help us more thoroughly evaluate and improve the model. If we ask medical providers easy questions like “did the AI correctly evaluate medical conditions in this message? yes/no” we can more quickly evaluate our model performance and improve it over time.