LLM Mesh#
The LLM Mesh is the common backbone for Enterprise Generative AI Applications. For more details on the LLM Mesh features of Dataiku, please visit Generative AI and LLM Mesh.
The LLM Mesh API allows you to:
Send completion and embedding queries to all LLMs supported by the LLM Mesh
Stream responses from LLMs that support it
Query LLMs using multimodal inputs (image and text)
Query the LLM Mesh from LangChain code
Interact with knowledge banks, and perform semantic search
Perform completion queries on LLMs#
Your first simple completion query#
This sample
import dataiku
# Fill with your LLM id. For example, if you have an OpenAI connection called "myopenai", LLM_ID can be "openai:myopenai:gpt-4o"
# To get the list of LLM ids, you can use project.list_llms() (see below)
LLM_ID = ""
# Create a handle for the LLM of your choice
client = dataiku.api_client()
project = client.get_default_project()
llm = project.get_llm(LLM_ID)
# Create and run a completion query
completion = llm.new_completion()
completion.with_message("Write a haiku on GPT models")
resp = completion.execute()
# Display the LLM output
if resp.success:
print(resp.text)
# GPT, a marvel,
# Deep learning's symphony plays,
# Thoughts dance, words unveil.
Multi-turn and system prompts#
You can have multiple messages in the completion
object, with roles
completion = llm.new_completion()
# First, put a system prompt
completion.with_message("You are a poetic assistant who always answers in haikus", role="system")
# Then, give an example, or send the conversation history
completion.with_message("What is a transformer", role="user")
completion.with_message("Transformers, marvels\nOf the deep learning research\nAttention, you need", role="assistant")
# Then, the last query of the user
completion.with_message("What's your name", role="user")
resp = completion.execute()
Multimodal input#
Multimodal input is supported on a subset of the LLMs in the LLM Mesh:
OpenAI
Bedrock Anthropic Claude
Azure OpenAI
Gemini Pro
completion = llm.new_completion()
with open("myimage.jpg", "rb") as f:
image = f.read()
mp_message = completion.new_multipart_message()
mp_message.with_text("The image represents an artwork. Describe it as it would be described by art critics")
mp_message.with_inline_image(image)
# Add it to the completion request
mp_message.add()
resp = completion.execute()
Completion settings#
You can set settings on the completion query
completion = llm.new_completion()
completion.with_message("Write a haiku on GPT models")
completion.settings["temperature"] = 0.7
completion.settings["topK"] = 10
completion.settings["topP"] = 0.3
completion.settings["maxOutputTokens"] = 2048
completion.settings["stopSequences"] = [".", "\n"]
resp = completion.execute()
Response streaming#
from dataikuapi.dss.llm import DSSLLMStreamedCompletionChunk, DSSLLMStreamedCompletionFooter
completion = llm.new_completion()
completion.with_message("Please explain special relativity")
for chunk in completion.execute_streamed():
if isinstance(chunk, DSSLLMStreamedCompletionChunk):
print("Received text: %s" % chunk.data["text"])
elif isinstance(chunk, DSSLLMStreamedCompletionFooter):
print("Completion is complete: %s" % chunk.data)
Read LLM Mesh metadata#
List and get LLMs#
import dataiku
client = dataiku.api_client()
project = client.get_default_project()
llm_list = project.list_llms()
By default, list_llms()
returns a list of DSSLLMListItem
. To get more details :
for llm in llm_list:
print(f"- {llm.description} (id: {llm.id})")
Text embedding#
import dataiku
EMBEDDING_MODEL_ID = "" # Fill with your embedding model id, for example: openai:myopenai:text-embedding-3-small
# Create a handle for the embedding model of your choice
client = dataiku.api_client()
project = client.get_default_project()
emb_model = project.get_llm(EMBEDDING_MODEL_ID)
# Create and run an embedding query
txt = "The quick brown fox jumps over the lazy dog."
emb_query = emb_model.new_embeddings()
emb_query.add_text(txt)
emb_resp = emb_query.execute()
# Display the embedding output
print(emb_resp.get_embeddings())
# [[0.000237455,
# -0.103262354,
# ...
# ]]
Knowledge Banks (KB)#
List and get KBs#
To list the KB present in a project:
import dataiku
client = dataiku.api_client()
project = client.get_default_project()
kb_list = project.list_knowledge_banks()
By default, list_knowledge_banks()
returns a list of DSSKnowledgeBankListItem
.
To get more details:
for kb in kb_list:
print(f"{kb.name} (id: {kb.id})")
To get a “core handle” on the KB (i.e. to retrieve a KnowledgeBank
object) :
KB_ID = "" # Fill with your KB id
kb_public_api = project.get_knowledge_bank(KB_ID)
kb_core = kb_public_api.as_core_knowledge_bank()
LangChain integration#
Dataiku LLM model objects can be turned into langchain-compatible objects, making it easy to:
stream responses
run asynchronous queries
batch queries
chain several models and adapters
integrate with the wider langchain ecosystem
Transforming LLM handles to LangChain model#
# In this sample, llm is the result of calling project.get_llm() (see above)
# Turn a regular LLM handle into a langchain-compatible one
langchain_llm = llm.as_langchain_llm()
# Run a single completion query
langchain_llm.invoke("Write a haiku on GPT models")
# Run a batch of completion queries
langchain_llm.batch(["Write a haiku on GPT models", "Write a haiku on GPT models in German"])
# Run a completion query and stream the response
for chunk in langchain_llm.stream("Write a haiku on GPT models"):
print(chunk, end="", flush=True)
See the langchain documentation for more details.
You can also turn it into a langchain “chat model”, a specific type of LLM geared towards conversation:
# In this sample, llm is the result of calling project.get_llm() (see above)
# Turn a regular LLM handle into a langchain-compatible one
langchain_llm = llm.as_langchain_chat_model()
# Run a simple query
langchain_llm.invoke("Write a haiku on GPT models")
# Run a chat query
from langchain_core.messages import HumanMessage, SystemMessage
messages = [
SystemMessage(content="You're a helpful assistant"),
HumanMessage(content="What is the purpose of model regularization?"),
]
langchain_llm.invoke(messages)
# Streaming and chaining
from langchain.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_template("Tell me a joke about {topic}")
chain = prompt | langchain_llm
for chunk in chain.stream({"topic": "parrot"}):
print(chunk.content, end="", flush=True)
See the langchain documentation for more details.
Creating Langchain models directly#
If running from inside DSS, you can also directly create the Langchain model:
from dataiku.langchain.dku_llm import DKULLM, DKUChatLLM
langchain_llm = DKUChatLLM(llm_id="your llm id") # For example: openai:myopenai:gpt-4o
Response streaming#
LangChain adapters (DKULLM and DKUChatLLM) also support streaming of answer:
from dataiku.langchain.dku_llm import DKULLM, DKUChatLLM
from langchain_core.messages import HumanMessage, SystemMessage
langchain_llm = DKUChatLLM(llm_id="your llm id") # For example: openai:myopenai:gpt-4o
messages = [
SystemMessage(content="You're a helpful assistant"),
HumanMessage(content="What is the purpose of model regularization?"),
]
for gen in langchain_llm.stream(messages):
print(gen)
Using knowlege banks as LangChain retrievers#
Core handles allow users to leverage the Langchain library and, through it:
query the KB for semantic similarity search
combine the KB with an LLM to form a chain and perform complex workflows such as retrieval-augmented generation (RAG).
In practice, core handles expose KBs as a Langchain-native vector store through two different methods:
as_langchain_retriever()
returns a genericVectorStoreRetriever
objectas_langchain_vectorstore()
returns an object whose class corresponds to the KB type. For example, for a FAISS-based KB, you will get alangchain.vectorstores.faiss.FAISS
object.
import dataiku
client = dataiku.api_client()
project = client.get_default_project()
kb_core = project.get_knowledge_bank(KB_ID).as_core_knowledge_bank()
# Return a langchain.vectorstores.base.VectorStoreRetriever
lc_generic_vs= kb_core.as_langchain_retriever()
# Return an object which type depends on the KB type
lc_vs = kb_core.as_langchain_vectorstore()
# [...] Move forward with similarity search or RAG
Reference documentation#
|
A handle to interact with a DSS-managed LLM. |
|
An item in a list of llms |
A handle to interact with a completion query. |
|
A handle to interact with a multi-completion query. |
|
|
|
A handle to interact with a completion response. |
|
A handle to interact with an embedding query. |
|
A handle to interact with an embedding query result. |
|
An item in a list of knowledege banks |
|
A handle to interact with a DSS-managed knowledge bank. |