Flows have many use cases, and not all of them require the use of large language models. That’s why this is an optional component. Connecting an LLM component allows you to pick LLM models and control the Generator output. The first and most popular one is the LLM OpenAI which connects you to ChatGPT.
What is the LLM Open AI component?
The LLM Open AI component connects ChatGPT to your flow. It’s an additional component to the Generator. While the Generator is where the magic happens, the LLM-type components allow you to control how it happens. The Generator uses ChatGPT-4 by default. If you wish to change the model or limit the generation capabilities, you can connect this component.
The LLM Open AI component can be found in the LLMs category of the flows editor. It contains these settings:
Max Tokens
Tokens represent the individual units of text the model processes and generates. Token usage varies with models, and a single token can be anything from words or subwords to a single character. Models are usually priced in millions of tokens.
The max tokens setting limits the total number of tokens that can be processed in a single interaction or request, ensuring the responses are generated within reasonable bounds. The default limit is 4,000 tokens, which is the optimal size for summarizing documents and several sources to generate an answer.
Model Name
ChatGPT has differently capable models, each with different pricing. For example, using the less advanced and older GPT-3.5 will cost less than using the newest 4o, but the quality and speed of the output will suffer.
Temperature
Temperature controls the variability of answers, ranging from 0 to 1.
A temperature of 0.1 will make the responses very to the point but potentially repetitive and deficient.
A high temperature of 1 allows for maximum creativity in answers but creates the risk of irrelevant or even hallucinatory responses.
For example, the recommended temperature for a customer service bot is between 0.2 and 0.5. This level should keep the answers relevant and to the script while allowing for a natural level of variation in responses.
How to connect the LLM OpenAI component to your flow
You’ll notice that the LLM component only has a handle on the right side. Nothing connects to it, but it connects to other components. These usually serve to alter the output further or merge it with other components.
LLMs can connect to these categories:
Generator
The generator connects the user Chat Input with the settings from the LLM Component and runs them through the LLM to create a response.
Splitters
These are methods to enhance the accuracy of output. They can use the input to create similar follow-up questions, decompose complex user queries, or expand on simple ones to provide more information.
