If you find some of the different terminology used for Large Language Models (LLMs) and AI confusing, you are not alone!

This is the first in a series of articles about AI, LLMs and Retrieval Augmented Generation (RAG) where we aim to explain clearly and succinctly, some of the key terminology you might be hearing about. We hope you find these posts helpful!

What are foundation models?

A foundation model is an AI model, trained on huge amounts of data (documents, audio, images, text….). It is trained to ‘generate’ the next word as it ‘learns’ the language. It should then be specialised and fine-tuned for a wide variety of applications and tasks, which then means it is no longer a foundation model!

What are LLMs?

A LLM is an umbrella term used for all foundation and specialised models.

For example:

In the case of Llama, the foundation model is not usable directly but serves as the foundation for all the subsequent specialised models. Llama instruct is a question and answering model and code Llama is a coding assistant.

All three models are LLMs.

What are the benefits and challenges of a foundation model?

In terms of benefits: 

Flexibility and adaptability

Foundation models are flexible and adaptable as they can be be fine-tuned for a wide range of tasks, saving time and resources compared to building new models from scratch for each specific task.

Cost efficient

While foundation models are costly, once you have them, you can adapt them as many times as you want on new tasks.

Accessibility

Open source foundation models are accessible as smaller companies with less access to computational resources can leverage these models to create innovative AI applications. (Note that there are many closed models which are not accessible!)

(Note – Open source foundation models – almost anyone can use, access the source code and customise the foundation model which in theory, improves accessibility, transparency etc.  Meta’s Llama 2 is an open source foundation model.  Chatgpt is not open source. 

As for the challenges: 

Bias

Foundation models are trained on large and diverse data sets which may contain biases present in the data, and which will be mirrored in the model’s outputs.

Security and privacy

The huge amounts of data needed to train a foundation model naturally raises security and privacy concerns.  The data should be secure and handled responsibility.

Lack of transparency

Foundation models can be a ‘black box’ .  The issue with data has already been highlighted.  In addition, it is important to understand how the foundation model generates its outputs to identify any potential errors or bias.  This is a hot topic with ongoing empirical studies.