LLMs are typically developed through a process of training on vast amounts of data, the corpus. This costs a lot of time and money. ChatGPT-3, for example, cost $10M. This cost going down but it’s remains expensive. You can avoid this cost for specific use cases by “fine-tuning” a model with specific data or you can augment their prompts with reference data as in Retrieval Augmented Generation or RAG. The next stage in LLM development are models that update/evolve through time. This is what’s discussed in Sakana AI’s paper Transformer²: Self-Adaptive LLMs.