Cookbook：使用 Litefuse 追踪 OpenAI Structured Outputs

在本 cookbook 中，你将学习如何使用 Litefuse 监控 OpenAI Structured Outputs。

什么是 structured outputs？

从非结构化输入生成结构化数据是当下 AI 的核心使用场景之一。Structured outputs 能让链式 LLM 调用、UI 组件生成以及基于模型的评估更加可靠。Structured Outputs 是 OpenAI API 的一项新能力，它在 JSON 模式与函数调用基础上做了增强，能强制模型输出严格遵守某个 schema。

如何在 Litefuse 中追踪 structured output？

如果你使用 OpenAI Python SDK，可以使用 Litefuse 提供的替换式封装，只需修改一行 import 就能获得完整日志。由此你可以在 Litefuse 中监控 OpenAI 生成的 structured output。

- import openai
+ from langfuse.openai import openai
 
Alternative imports:
+ from langfuse.openai import OpenAI, AsyncOpenAI, AzureOpenAI, AsyncAzureOpenAI

步骤 1：初始化 Litefuse

使用 Litefuse UI 项目设置中的 API 密钥初始化 Langfuse 客户端，并将密钥添加到环境变量中。

%pip install langfuse openai --upgrade

import os
 
# Get keys for your project from the project settings page
# https://litefuse.cloud
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_BASE_URL"] = "https://litefuse.cloud"
 
# Your openai key
os.environ["OPENAI_API_KEY"] = ""

步骤 2：数学辅导示例

在本示例中，我们将构建一个数学辅导工具，输出由结构化对象组成的数组，对应解题的每一步。

这种结构非常适合需要逐步展示每一个步骤的应用，让用户可以按自己的节奏阅读解答过程。

（示例改编自 OpenAI cookbook）

注意： 虽然 OpenAI 也通过其 beta API（client.beta.chat.completions.parse）提供结构化输出解析，但该方式目前不支持设置 Litefuse 特定属性，例如 name、metadata、userId 等。请按下文示例，使用标准的 client.chat.completions.create 并通过 response_format 参数实现。

# Use the Langfuse drop-in replacement to get full logging by changing only the import.
# With that, you can monitor the structured output generated by OpenAI in Langfuse.
from langfuse.openai import OpenAI
import json
 
openai_model = "gpt-4o-2024-08-06"
client = OpenAI()

在 response_format 参数中你现在可以通过 json_schema 提供一份 JSON Schema。当 response_format 配合 strict: true 使用时，模型输出会严格遵循提供的 schema。

函数调用的方式大体相同，但新增参数 strict: true 后，可以确保提供的函数 schema 被严格遵循。

math_tutor_prompt = '''
    You are a helpful math tutor. You will be provided with a math problem,
    and your goal will be to output a step by step solution, along with a final answer.
    For each step, just provide the output as an equation use the explanation field to detail the reasoning.
'''
 
def get_math_solution(question):
    response = client.chat.completions.create(
    model = openai_model,
    messages=[
        {
            "role": "system",
            "content": math_tutor_prompt
        },
        {
            "role": "user",
            "content": question
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "math_reasoning",
            "schema": {
                "type": "object",
                "properties": {
                    "steps": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "explanation": {"type": "string"},
                                "output": {"type": "string"}
                            },
                            "required": ["explanation", "output"],
                            "additionalProperties": False
                        }
                    },
                    "final_answer": {"type": "string"}
                },
                "required": ["steps", "final_answer"],
                "additionalProperties": False
            },
            "strict": True
        }
    }
    )
 
    return response.choices[0].message

# Testing with an example question
question = "how can I solve 8x + 7 = -23"
 
result = get_math_solution(question)
 
print(result.content)

{"steps":[{"explanation":"We need to isolate the term with the variable, 8x. So, we start by subtracting 7 from both sides to remove the constant term on the left side.","output":"8x + 7 - 7 = -23 - 7"},{"explanation":"The +7 and -7 on the left side cancel each other out, leaving us with 8x. The right side simplifies to -30.","output":"8x = -30"},{"explanation":"To solve for x, divide both sides of the equation by 8, which is the coefficient of x.","output":"x = -30 / 8"},{"explanation":"Simplify the fraction -30/8 by finding the greatest common divisor, which is 2.","output":"x = -15 / 4"}],"final_answer":"x = -15/4"}

# Print results step by step
 
result = json.loads(result.content)
steps = result['steps']
final_answer = result['final_answer']
for i in range(len(steps)):
    print(f"Step {i+1}: {steps[i]['explanation']}\n")
    print(steps[i]['output'])
    print("\n")
 
print("Final answer:\n\n")
print(final_answer)

Step 1: We need to isolate the term with the variable, 8x. So, we start by subtracting 7 from both sides to remove the constant term on the left side.

8x + 7 - 7 = -23 - 7


Step 2: The +7 and -7 on the left side cancel each other out, leaving us with 8x. The right side simplifies to -30.

8x = -30


Step 3: To solve for x, divide both sides of the equation by 8, which is the coefficient of x.

x = -30 / 8


Step 4: Simplify the fraction -30/8 by finding the greatest common divisor, which is 2.

x = -15 / 4


Final answer:


x = -15/4

步骤 3：在 Litefuse 中查看 trace

现在你可以在 Litefuse 中查看 trace 以及 JSON schema。

Litefuse 中的示例 trace

在 Litefuse UI 中查看示例 trace

替代方案：使用 SDK 的 `parse` 辅助方法

新版 SDK 新增了一个 parse 辅助方法，允许你直接使用自己的 Pydantic 模型，而无需手写 JSON schema。

from pydantic import BaseModel
 
class MathReasoning(BaseModel):
    class Step(BaseModel):
        explanation: str
        output: str
 
    steps: list[Step]
    final_answer: str
 
def get_math_solution(question: str):
    response = client.beta.chat.completions.parse(
        model=openai_model,
        messages=[
            {"role": "system", "content": math_tutor_prompt},
            {"role": "user", "content": question},
        ],
        response_format=MathReasoning,
    )
 
    return response.choices[0].message

result = get_math_solution(question).parsed
 
print(result.steps)
print("Final answer:")
print(result.final_answer)

[Step(explanation='To isolate the term with the variable on one side of the equation, start by subtracting 7 from both sides.', output='8x = -23 - 7'), Step(explanation='Combine like terms on the right side to simplify the equation.', output='8x = -30'), Step(explanation='Divide both sides by 8 to solve for x.', output='x = -30 / 8'), Step(explanation='Simplify the fraction by dividing both the numerator and the denominator by their greatest common divisor, which is 2.', output='x = -15 / 4')]
Final answer:
x = -15/4