LLM Connections

LLM connections are used to call models in the Litefuse Playground or for LLM-as-a-Judge evaluations.

Setup

GET /api/public/llm-connections
PUT /api/public/llm-connections

Supported providers

The Litefuse platform is currently supporting the following LLM providers:

• OpenAI
• Azure OpenAI
• Anthropic
• Google AI Studio
• Google Vertex AI
• Amazon Bedrock

You may connect to third party LLM providers if their API schema implements the schema of one of our supported provider adapters. For example, you may connect to Mistral by using the OpenAI adapter in Litefuse to connect to Mistral’s OpenAI compliant API.

OpenAI-compatible and Anthropic-compatible providers

Litefuse includes provider-specific compatibility handling for selected providers that expose OpenAI-compatible or Anthropic-compatible APIs but require small protocol adjustments for reliable structured output in the Playground and evaluator jobs.

This is especially relevant for LLM-as-a-Judge evaluations, which rely on structured responses and tool/function calling.

• Qwen / DashScope: Litefuse detects Qwen connections by provider name or DashScope host, normalizes Anthropic-compatible /v1/messages base URLs, disables thinking by default unless you override it, and uses function calling for structured output to avoid JSON mode incompatibilities.
• MiniMax: Litefuse detects MiniMax connections by provider name or MiniMax host, normalizes Anthropic-compatible /v1/messages base URLs, keeps reasoning_split: true by default for OpenAI-compatible requests, uses function calling for structured output, removes <think>...</think> content from responses before parsing, and applies provider-specific thinking defaults for models such as MiniMax-M2.7 and MiniMax-M3.

To use these providers, create an LLM connection with the OpenAI or Anthropic adapter, set the provider API key, set the provider’s compatible Base URL, and add the model name under Custom model names if it is not listed by default.

You can still use provider options to pass explicit provider-specific overrides when the selected provider supports them.

If your provider exposes a compatible API but is not listed above, you can still connect it through the matching adapter. The provider must support tool/function calling in the selected API format for LLM-as-a-Judge evaluations to return structured scores reliably.

Advanced configurations

Additional provider options

LLM calls from a created LLM connection can be configured with a specific set of parameters, such as temperature, top_p, and max_tokens.

However, many LLM providers allow for additional parameters, including reasoning_effort, service_tier, and others when invoking a model. These parameters often differ between providers.

You can provide additional configurations as a JSON object for all LLM invocations. In the model parameters settings, you will find a “provider options” field at the bottom. This field allows you to enter specific key-value pairs accepted by your LLM provider’s API endpoint.

Please see your providers API reference for what additional fields are supported:

• Anthropic Messages API Reference
• OpenAI Chat Completions API Reference
• Google Gemini API Reference
• Google Vertex AI Gemini API Reference

This feature is currently available for the adapters for:

• Anthropic
• OpenAI
• Google AI Studio
• Google Vertex AI
• AWS (Amazon Bedrock)

Example for forcing reasoning effort minimal on an OpenAI gpt-5 invocation:

Example provider options for Google models (Google AI Studio and Google Vertex AI) to control thinking:

{
  "thinkingLevel": "MEDIUM",
  "thinkingBudget": 2048
}

thinkingLevel controls the reasoning depth, and thinkingBudget sets the token budget for thinking. These options are only valid for Google models that support thinking.

Connecting via a gateway

If you route LLM calls through an OpenAI-compatible gateway such as LiteLLM, OpenRouter, or Portkey, you can use it as your LLM connection for the Playground and LLM-as-a-Judge evaluations.

To set this up:

1. Navigate to Project Settings > LLM Connections and click Add new LLM API key.
2. Select OpenAI as the provider.
3. Enter the API key for your gateway.
4. Under Advanced Settings, set the Base URL to your gateway’s endpoint, e.g. https://your-litellm-instance.com/v1.
5. Add any custom model names that your gateway exposes.

For OpenAI-compatible gateways, here is an example tool calling request that must be handled by the gateway to support LLM-as-a-Judge in Litefuse:

curl -X POST 'https://<host set in project settings>/chat/completions' \
-H 'accept: application/json' \
-H 'content-type: application/json' \
-H 'authorization: Bearer <api key entered in project settings>' \
-H 'x-test-header-1: <custom header set in project settings>' \
-H 'x-test-header-2: <custom header set in project settings>' \
-d '{
  "model": "<model set in project settings>",
  "temperature": 0,
  "top_p": 1,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "max_tokens": 256,
  "n": 1,
  "stream": false,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "extract",
        "parameters": {
          "type": "object",
          "properties": {
            "score": {
              "type": "string"
            },
            "reasoning": {
              "type": "string"
            }
          },
          "required": [
            "score",
            "reasoning"
          ],
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    }
  ],
  "tool_choice": {
    "type": "function",
    "function": {
      "name": "extract"
    }
  },
  "messages": [
    {
      "role": "user",
      "content": "Evaluate the correctness of the generation on a continuous scale from 0 to 1. A generation can be considered correct (Score: 1) if it includes all the key facts from the ground truth and if every fact presented in the generation is factually supported by the ground truth or common sense.\n\nExample:\nQuery: Can eating carrots improve your vision?\nGeneration: Yes, eating carrots significantly improves your vision, especially at night. This is why people who eat lots of carrots never need glasses. Anyone who tells you otherwise is probably trying to sell you expensive eyewear or does not want you to benefit from this simple, natural remedy. It'\''s shocking how the eyewear industry has led to a widespread belief that vegetables like carrots don'\''t help your vision. People are so gullible to fall for these money-making schemes.\nGround truth: Well, yes and no. Carrots won'\''t improve your visual acuity if you have less than perfect vision. A diet of carrots won'\''t give a blind person 20/20 vision. But, the vitamins found in the vegetable can help promote overall eye health. Carrots contain beta-carotene, a substance that the body converts to vitamin A, an important nutrient for eye health.  An extreme lack of vitamin A can cause blindness. Vitamin A can prevent the formation of cataracts and macular degeneration, the world'\''s leading cause of blindness. However, if your vision problems aren'\''t related to vitamin A, your vision won'\''t change no matter how many carrots you eat.\nScore: 0.1\nReasoning: While the generation mentions that carrots can improve vision, it fails to outline the reason for this phenomenon and the circumstances under which this is the case. The rest of the response contains misinformation and exaggerations regarding the benefits of eating carrots for vision improvement. It deviates significantly from the more accurate and nuanced explanation provided in the ground truth.\n\n\n\nInput:\nQuery: {{query}}\nGeneration: {{generation}}\nGround truth: {{ground_truth}}\n\n\nThink step by step."
    }
  ]
}'