Score Configs
Score Configs define the structure of scores in a project. They make evaluation results consistent by defining the score name, data type, optional value range, and categories that reviewers or automated evaluators should use.
You can use Score Configs when you:
- Manually annotate traces, observations, sessions, or experiment results in the Litefuse UI
- Create annotation queues with fixed scoring dimensions
- Send scores through the API or SDK
- Analyze score trends in Score Analytics
Where to find Score Configs
Open a project and go to Project Settings → Score Configs.
From there, you can review existing configs, create new configs, and archive configs you no longer want to use for new annotations.
Score Configs are project-level settings. Creating or editing a config in one project does not affect other projects.
Built-in Score Configs
New projects include a built-in Score Config so you can start annotating without setting up scoring infrastructure manually.
| Name | Type | Range or values | Typical use |
|---|---|---|---|
Correctness | Numeric | 0 to 1 | Measure whether the output is correct for the task. |
New projects also include a Correctness Queue annotation queue that uses this config. Existing projects are not changed automatically.
Create a custom Score Config
Use custom Score Configs when your team needs a project-specific metric, rubric, or label set.
Open Score Configs
Go to Project Settings → Score Configs.
Add a config
Click Add new score config.
Choose the data type
Select one of the supported data types:
NUMERIC: A number, optionally constrained by a minimum and maximum value.BOOLEAN: A true/false score.CATEGORICAL: A fixed set of labels, each mapped to a numeric value.
Define the scoring criteria
Add a clear name and description. The description should tell reviewers or evaluators what the score means and how to apply it.
Save the config
After saving, the config can be used in manual annotations, annotation queues, and score ingestion workflows.
Choosing names and ranges
Use stable, descriptive names because score names are used in filters, analytics, APIs, and dashboards.
Good examples:
answer_qualitygroundednessretrieval_relevancepolicy_violation
For numeric scores, prefer a consistent 0 to 1 range unless your metric has a domain-specific scale. This makes it easier to compare metrics across dashboards and experiments.
Using Score Configs with scores
When a score references a Score Config, the score must match the config:
- The score name must equal the config name.
- The value must match the config data type.
- Numeric values must fit within the configured range, if a range is set.
- Categorical and boolean values must map to the configured categories.
This validation helps prevent inconsistent data such as accuracy = 10 when your accuracy scale is 0 to 1.
Archive old configs
Archive a Score Config when you no longer want users to apply it to new annotations. Existing historical scores remain visible and can still be analyzed.
Use archiving instead of renaming old configs when the meaning of a metric changes. If the rubric changes materially, create a new config so old and new scores remain comparable within their own definitions.