To leverage large language models (LLMs) or foundation models in your applications, you can use the Generative AI Hub on SAP AI Core. Like most other LLM applications, Generative AI Hub operates on a pay-per-use basis.
Generative AI Hub offers all major models on the market. There are open-source models that SAP has deployed such as models from Mistral AI. And there are models that SAP is a proxy for, such as the GPT models, Google models, models provided by Amazon Bedrock and more. You can easily switch between them, compare results, and select the model that works best for your use case.
SAP maintains strict data privacy contracts with LLM providers to ensure that your data remains secure and your prompts remain private.
To start using one of the available models in Generative AI Hub, you need to first deploy it. You need to deploy an orchestration service to access all available models. You can also deploy a single model directly via the Model Library. You can access your deployed models using the Python SDK, the SAP Cloud SDK for AI (JavaScript SDK), any programming language or API platform, or the user interface in SAP AI Launchpad.
For this CodeJam the necessary deployments have already been made.
👉 Open a new chat window by navigating to the Generative AI Hub tab and select Chat.
👉 Click Configure and have a look at the available fields.
Under Selected Model you will find all the deployed models. If there is no deployment this will be empty and you will not be able to chat. Because the orchestration service is enabled, you should see all available models here.
👉 Pick whatever model you would like to try! You can head over to the Model Library and compare models using the Leaderboard or Chart. You can check for example the cheapest model with the highest Helm score.
The Frequency Penalty parameter allows you to penalize words that appear too frequently in the text, helping the model sound less robotic.
Similarly, the higher the Presence Penalty, the more likely the model is to introduce new topics, as it penalizes words that have already appeared in the text.
The Max Tokens parameter allows you to set the size of the model's input and output. Tokens are not individual words but are typically 4-5 characters long.
The Temperature parameter allows you to control how creative the model should be, determining how flexible it is in selecting the next token in the sequence.
In the Chat Context tab, right under Context History, you can set the number of messages to be sent to the model, determining how much of the chat history should be provided as context for each new request. This is basically the size of the models memory.
You can also add a System Message to describe the role or give more information about what is expected from the model. Additionally, you can provide example inputs and outputs.
👉 Try out different prompt engineering techniques following these examples:
- Zero shot:
The capital of Poland is: - Few shots:
France - Paris Canada - Ottawa Ukraine - Kyiv Australia - - Chain of thought (Copy the WHOLE BLOCK into the chat window):
1. What is the most important city of a country? 2. In which country was the hot air balloon originally developed? 3. What is the >fill in the word from step 1< of the country >fill in the word from step 2<.
👉 Try to add something funny to the System Message like "always respond like a pirate" and try the prompts again. You can also instruct it to speak more technically, like a developer, or more polished, like in marketing.
👉 Have the model count the occurance of letters within a specific word. For example how often the letter r occurs in strawberry. Can you come up with a prompt that counts it correctly? If your model did it correctly good! - Try using an older version and see if the result changes.
The Prompt Editor is useful if you want to save a prompt and its response to revisit later or compare prompts. Often, you can identify tasks that an LLM can help you with on a regular basis. In that case, you can also save different versions of the prompt that work well, saving you from having to write the prompt again each time.
The parameters you were able to set in the Chat can also be set here. Additionally, you can view the number of tokens your prompt used below the response.
👉 Navigate to the Prompt Editor within Generative AI Hub.
👉 Select a model and set Max Tokens to the maximum.
👉 Paste the example below and click on Run.
👉 Give your prompt a Name, a Collection name, and Save the prompt.
👉 If you now head over to the Prompt Management, you will find your previously saved prompt there.
To run the prompt again click Open in Prompt Editor.
You can also select other saved prompts by clicking on Select.
-
Chain of thought prompt - customer support (Copy the WHOLE BLOCK into the prompt editor):
You are working at a big tech company and you are part of the support team. You are tasked with sorting the incoming support requests into: German, English, Polish or French. Review the incoming query. Identify the language of the query as either German, English, Polish, or French. - Example: 'bad usability. very confusing user interface.' - English Count the number of queries for each language: German, English, Polish, and French. Summarize the key pain points mentioned in the queries in bullet points in English. Queries: - What are the shipping costs to Australia? - Kann ich einen Artikel ohne Kassenbon umtauschen? - Offrez-vous des réductions pour les achats en gros? - Can I change the delivery address after placing the order? - Comment puis-je annuler mon abonnement? - Wo kann ich den Status meiner Reparatur einsehen? - What payment methods do you accept? - Czemu to tak długo ma iść do Wrocławia, gdy cena nie jest wcale niska? - Quel est le délai de livraison estimé pour le Mexique? - Gibt es eine Garantie auf elektronische Geräte? - I’m having trouble logging into my account, what should I do?
👉 If you still have time. Ask the LLM to come up with different support queries to have more data.
Now that you have experienced the AI Launchpad, if you see the Feedback icon in the upper right corner, please click it to complete a survey. It should take no more than 3 minutes of your time, but this feedback is extremely important for us.
By this point, you will know how to use the Generative AI Hub user interface in SAP AI Launchpad to query LLMs and store important prompts. You will also understand how to refine the output of a large language model using prompt engineering techniques.
- Generative AI Hub on SAP AI Core - Help Portal (Documentation)
- Prompt Engineering Guide is a good resource if you want to know more about prompt engineering in general.
- Prompt LLMs in the generative AI hub in SAP AI Core & Launchpad is a good tutorial on how to prompt LLMs with Generative AI Hub.