"It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.\n",
"\n",
"It is often useful to have a model return output that matches a specific schema. One common use-case is extracting data from text to insert into a database or use with some other downstream system. This guide covers a few strategies for getting structured outputs from a model.\n",
"\n",
"## The `.with_structured_output()` method\n",
"\n",
"There are several strategies that models can use under the hood. For some of the most popular model providers, including [OpenAI](/docs/integrations/platforms/openai/), [Anthropic](/docs/integrations/platforms/anthropic/), and [Mistral](/docs/integrations/providers/mistralai/), LangChain implements a common interface that abstracts away these strategies called `.with_structured_output`.\n",
":::info Supported models\n",
"\n",
"You can find a [list of models that support this method here](/docs/integrations/chat/).\n",
"\n",
"By invoking this method (and passing in [JSON schema](https://json-schema.org/) or a [Pydantic](https://docs.pydantic.dev/latest/) model) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.\n",
":::\n",
"\n",
"You can find the [current list of models that support this method here](/docs/integrations/chat/).\n",
"This is the easiest and most reliable way to get structured outputs. `with_structured_output()` is implemented for models that provide native APIs for structuring outputs, like tool/function calling or JSON mode, and makes use of these capabilities under the hood.\n",
"\n",
"Let's look at some examples of this in action! We'll use Pydantic to create a simple response schema.\n",
"This method takes a schema as input which specifies the names, types, and descriptions of the desired output attributes. The method returns a model-like Runnable, except that instead of outputting strings or Messages it outputs objects corresponding to the given schema. The schema can be specified as a [JSON Schema](https://json-schema.org/) or a Pydantic class. If JSON Schema is used then a dictionary will be returned by the Runnable, and if a Pydantic class is used then Pydantic objects will be returned.\n",
"\n",
"As an example, let's get a model to generate a joke and separate the setup from the punchline:\n",
"\n",
"```{=mdx}\n",
"import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
"If we want the model to return a Pydantic object, we just need to pass in desired the Pydantic class:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 38,
"id": "070bf702",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Joke(setup='Why was the cat sitting on the computer?', punchline='Because it wanted to keep an eye on the mouse!', rating=None)"
"Joke(setup='Why was the cat sitting on the computer?', punchline='To keep an eye on the mouse!', rating=None)"
]
},
"execution_count": 13,
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
@ -88,6 +96,8 @@
"\n",
"\n",
"class Joke(BaseModel):\n",
" \"\"\"Joke to tell user.\"\"\"\n",
"\n",
" setup: str = Field(description=\"The setup of the joke\")\n",
" punchline: str = Field(description=\"The punchline to the joke\")\n",
" rating: Optional[int] = Field(description=\"How funny the joke is, from 1 to 10\")\n",
@ -98,25 +108,27 @@
"structured_llm.invoke(\"Tell me a joke about cats\")"
]
},
{
"cell_type": "markdown",
"id": "00890a47-3cdf-4805-b8f1-6d110f0633d3",
"metadata": {},
"source": [
":::tip\n",
"Beyond just the structure of the Pydantic class, the name of the Pydantic class, the docstring, and the names and provided descriptions of parameters are very important. Most of the time `with_structured_output` is using a model's function/tool calling API, and you can effectively think of all of this information as being added to the model prompt.\n",
":::"
]
},
{
"cell_type": "markdown",
"id": "deddb6d3",
"metadata": {},
"source": [
"The result is a Pydantic model. Note that name of the model and the names and provided descriptions of parameters are very important, as they help guide the model's output.\n",
"\n",
"We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Pydantic. This dict should contain three properties:\n",
"\n",
"- `name`: The name of the schema to output.\n",
"- `description`: A high level description of the schema to output.\n",
"- `parameters`: The nested details of the schema you want to extract, formatted as a [JSON schema](https://json-schema.org/) dict.\n",
"\n",
"In this case, the response is also a dict:"
"We can also pass in a [JSON Schema](https://json-schema.org/) dict if you prefer not to use Pydantic. In this case, the response is also a dict:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 8,
"id": "6700994a",
"metadata": {},
"outputs": [
@ -124,30 +136,37 @@
"data": {
"text/plain": [
"{'setup': 'Why was the cat sitting on the computer?',\n",
" 'punchline': 'To keep an eye on the mouse!'}"
" 'punchline': 'Because it wanted to keep an eye on the mouse!',\n",
" 'rating': 8}"
]
},
"execution_count": 3,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"structured_llm = llm.with_structured_output(\n",
" {\n",
" \"name\": \"joke\",\n",
" \"description\": \"Joke to tell user.\",\n",
" \"parameters\": {\n",
" \"title\": \"Joke\",\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"setup\": {\"type\": \"string\", \"description\": \"The setup for the joke\"},\n",
" setup: str = Field(description=\"The setup of the joke\")\n",
" punchline: str = Field(description=\"The punchline to the joke\")\n",
"\n",
"\n",
"class ConversationalResponse(BaseModel):\n",
" \"\"\"Respond in a conversational manner. Be kind and helpful.\"\"\"\n",
"\n",
" response: str = Field(description=\"A conversational response to the user's query\")\n",
"\n",
"\n",
@ -212,7 +226,7 @@
{
"data": {
"text/plain": [
"Response(output=ConversationalResponse(response=\"I'm just a collection of code, so I don't have feelings, but thanks for asking! How can I assist you today?\"))"
"Response(output=ConversationalResponse(response=\"I'm just a digital assistant, so I don't have feelings, but I'm here and ready to help you. How can I assist you today?\"))"
]
},
"execution_count": 5,
@ -229,9 +243,225 @@
"id": "e28c14d3",
"metadata": {},
"source": [
"If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.\n",
"Alternatively, you can use tool calling directly to allow the model to choose between options, if your [chosen model supports it](/docs/integrations/chat/). This involves a bit more parsing and setup but in some instances leads to better performance because you don't have to use nested schemas. See [this how-to guide](/docs/how_to/tool_calling/) for more details."
]
},
{
"cell_type": "markdown",
"id": "9a40f703-7fd2-4fe0-ab2a-fa2d711ba009",
"metadata": {},
"source": [
"### Streaming\n",
"\n",
"We can stream outputs from our structured model when the output type is a dict (i.e., when the schema is specified as a JSON Schema dict). \n",
"\n",
":::info\n",
"\n",
"Note that what's yielded is already aggregated chunks, not deltas.\n",
"\n",
":::"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "aff89877-28a3-472f-a1aa-eff893fe7736",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{}\n",
"{'setup': ''}\n",
"{'setup': 'Why'}\n",
"{'setup': 'Why was'}\n",
"{'setup': 'Why was the'}\n",
"{'setup': 'Why was the cat'}\n",
"{'setup': 'Why was the cat sitting'}\n",
"{'setup': 'Why was the cat sitting on'}\n",
"{'setup': 'Why was the cat sitting on the'}\n",
"{'setup': 'Why was the cat sitting on the computer'}\n",
"{'setup': 'Why was the cat sitting on the computer?'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': ''}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on the'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on the mouse'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on the mouse!'}\n",
"{'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on the mouse!', 'rating': 8}\n"
"system = \"\"\"You are a hilarious comedian. Your specialty is knock-knock jokes. \\\n",
"Return a joke which has the setup (the response to \"Who's there?\") and the final punchline (the response to \"<setup> who?\").\n",
"\n",
"Here are some examples of jokes:\n",
"\n",
"example_user: Tell me a joke about planes\n",
"example_assistant: {{\"setup\": \"Why don't planes ever get tired?\", \"punchline\": \"Because they have rest wings!\", \"rating\": 2}}\n",
"\n",
"You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See [this how-to guide](/docs/how_to/tool_calling/) for more details."
"example_user: Tell me another joke about planes\n",
"example_assistant: {{\"setup\": \"Cargo\", \"punchline\": \"Cargo 'vroom vroom', but planes go 'zoom zoom'!\", \"rating\": 10}}\n",
"\n",
"example_user: Now about caterpillars\n",
"example_assistant: {{\"setup\": \"Caterpillar\", \"punchline\": \"Caterpillar really slow, but watch me turn into a butterfly and steal the show!\", \"rating\": 5}}\"\"\"\n",
"few_shot_structured_llm.invoke(\"what's something funny about woodpeckers\")"
]
},
{
"cell_type": "markdown",
"id": "3c12b389-153d-44d1-af34-37e5b926d3db",
"metadata": {},
"source": [
"When the underlying method for structuring outputs is tool calling, we can pass in our examples as explicit tool calls. You can check if the model you're using makes use of tool calling in its API reference."
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "d7381cb0-b2c3-4302-a319-ed72d0b9e43f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'setup': 'Crocodile',\n",
" 'punchline': \"Crocodile 'see you later', but in a while, it becomes an alligator!\",\n",
"For more on few shot prompting when using tool calling, see [here](/docs/how_to/function_calling/#Few-shot-prompting)."
]
},
{
@ -239,9 +469,17 @@
"id": "39d7a555",
"metadata": {},
"source": [
"### Specifying the output method (Advanced)\n",
"### (Advanced) Specifying the method for structuring outputs\n",
"\n",
"For models that support more than one means of outputting data, you can specify the preferred one like this:"
"For models that support more than one means of structuring outputs (i.e., they support both tool calling and JSON mode), you can specify which method to use with the `method=` argument.\n",
"\n",
":::info JSON mode\n",
"\n",
"If using JSON mode you'll have to still specify the desired schema in the model prompt. The schema you pass to `with_structured_output` will only be used for parsing the model outputs, it will not be passed to the model the way it is with tool calling.\n",
"\n",
"To see if the model you're using supports JSON mode, check its entry in the [API reference](https://api.python.langchain.com/en/latest/langchain_api_reference.html).\n",
"\n",
":::"
]
},
{
@ -253,7 +491,7 @@
{
"data": {
"text/plain": [
"Joke(setup='Why was the cat sitting on the computer?', punchline='Because it wanted to keep an eye on the mouse!')"
"Joke(setup='Why was the cat sitting on the computer?', punchline='Because it wanted to keep an eye on the mouse!', rating=None)"
]
},
"execution_count": 6,
@ -274,13 +512,9 @@
"id": "5e92a98a",
"metadata": {},
"source": [
"In the above example, we use OpenAI's alternate JSON mode capability along with a more specific prompt.\n",
"## Prompting and parsing model directly\n",
"\n",
"For specifics about the model you choose, peruse its entry in the [API reference pages](https://api.python.langchain.com/en/latest/langchain_api_reference.html).\n",
"\n",
"## Prompting techniques\n",
"\n",
"You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that don't support `.with_structured_output()` or other built-in approaches.\n",
"Not all models support `.with_structured_output()`, since not all models have tool calling or JSON mode support. For such models you'll need to directly prompt the model to use a specific format, and use an output parser to extract the structured response from the raw model output.\n",
"\n",
"### Using `PydanticOutputParser`\n",
"\n",
@ -289,7 +523,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 31,
"id": "6e514455",
"metadata": {},
"outputs": [],
@ -341,7 +575,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 37,
"id": "3d73d33d",
"metadata": {},
"outputs": [
@ -366,7 +600,7 @@
"source": [
"query = \"Anna is 23 years old and she is 6 feet tall\"\n",
Bagatur
3 weeks ago
committed by
GitHub