nithilan.dev Résumé
← All selected work

Project 01 / AI evaluation & developer tooling

jevcheck.

Behavioral contracts for TypeSafe Jev model upgrades. I built a Python library and CLI to record baseline answers, compare candidate models, and fail CI when the contract breaks.

pip install jevcheckView on PyPI

Listed in Hashgraph Online’s Awesome AI Plugins catalog.

Python 3.11+PydanticTypeSafe SDKGitHub Actions
Explore the repository

Turn an upgrade into a test

An application can depend on more than an answer’s shape. A support workflow may need a billing ticket to remain classified as billing, with enough confidence to take the next step. A candidate model can satisfy the response schema while changing that decision.

jevcheck tests those expectations for TypeSafe Jev’s System One API. It loads a contract, evaluates candidate answers, and reports unchanged cases, confidence regressions, and answer flips. The same checks run against live responses or local replay files.

The question is compatibility with an explicit contract. The result tells you which tested behaviors changed and why. It does not rank a model’s general intelligence or prove that an upgrade is safe for every input.

Specify the behavior that matters

A JSON or JSONL contract ties a baseline model to named cases. Each case contains input state, typed questions, and expectations. Contract-wide defaults establish floors and tolerances; individual fields can override them.

Three answer types, three comparison rules
AnswerWhat the evaluator checks
ChoiceAn expected category, a confidence floor, and an allowed drop from baseline confidence.
NoulA yes/no threshold over a 0–1 value, optional absolute drift tolerance, and minimum values.
ScoreA change in the rounded score level, with a tolerance that can suppress that flip, plus confidence checks.

Pydantic validates the boundary: duplicate case IDs, unknown expected fields, empty expectations, and rules for the wrong answer type are rejected. Candidate answers carry typed choice, noul, or score data. Probability maps must be finite, nonnegative, and sum to one within the declared tolerance.

Some rules are deliberately more specific than a generic numeric difference. A score change within the same rounded level is not an answer flip, even when its absolute distance exceeds the score tolerance. Inclusive drift boundaries use a small epsilon so binary floating-point subtraction does not turn an exact decimal boundary into a failure.

One evaluator, two sources of truth

eval compares a candidate with the expectations written in a fixture. The record and compare workflow adds another reference: what the baseline model actually answered.

Record baselineAnswers by case ID
Derive expectationsKeep floors & tolerances
Evaluate candidateReuse the comparison rules

I kept one evaluation engine. contract_from_baseline() fills expected choices, values, and baseline confidence from the recorded answers while preserving the contract’s resolved floors and tolerances. It then passes the derived contract into the existing evaluate() function.

This separates intended behavior from observed behavior without creating two competing implementations of the comparison rules. The original contract is not rewritten. A saved baseline can be reused, or compare can fetch both models live.

Baseline data must be valid before it becomes the reference. Missing baseline cases, missing expected fields, and answer types that disagree with their questions are rejected as input errors. A missing answer on the candidate side is instead reported as a behavioral flip.

Read the difference

The support-triage fixtures make the distinction concrete. In the breaking replay, ticket 002 changes its intent from general to billing. Ticket 003 keeps the billing label, but its intent confidence falls from 0.92 to 0.71 and its urgency confidence falls from 0.91 to 0.71.

FIG. 02 / Catch the behavioral changejevcheck
RecordCompareReport
Support triage · intent answers
CaseBaselineCandidate
001billing0.93 confidencebilling0.93 confidenceUnchanged
002general0.88 confidencebilling0.81 confidenceAnswer flip
003billing0.92 confidencebilling0.71 confidenceConfidence drop

Breaking · exit 1. One unchanged case, one answer flip, one confidence regression.

The answer can stay the same and still fail. Ticket 003 keeps “billing,” but drops below its 0.85 confidence floor. Intent fields shown; its urgency confidence also regresses. Repository replay fixtures, not live model results.

Both fields on ticket 003 fail, but the report counts cases. A case inherits its most severe field outcome: answer flip, then confidence regression, then unchanged. The fixture therefore produces one unchanged case, one flip, and one regression across three cases.

jevcheck compare fixtures/support-triage.json \
  --from fixtures/replay-baseline.json \
  --to jev-1.14 \
  --answers fixtures/replay-breaking.json

This command uses local replay files and needs no API key. The fixture labels jev-1.13 and jev-1.14 are example identifiers, not verified live model versions. The comparison above was checked against the repository’s evaluator.

Verify which model answered

A useful comparison has to identify its inputs. For a concrete model pin, every response must report the exact requested model name. A missing identity or a different version fails the check; shorthand names are not silently expanded.

Floating names containing latest or preview are rejected by default. Explicit opt-in permits an alias, but the response must resolve to a nonempty concrete name, which the report records. A response containing another floating alias still fails.

The typed client adapts SDK responses, while the evaluation and comparison paths enforce response identity. These boundaries keep malformed data or the wrong model from being mistaken for a successful behavioral check.

Make the result useful in CI

The CLI turns a readable report into a release check through distinct exit codes. A composite GitHub Action installs a selected jevcheck package version and runs eval, record, or compare.

CLI outcomes
ExitMeaning
0Compatible comparison, or a successful baseline recording.
1A behavioral contract failed.
2Usage, input, or model-identity error.
3Operational failure, such as an API error or invalid API response.

Replay files make the comparison path repeatable in CI without calling the model service. The repository includes tests for contract validation, pinning, answer adaptation, comparison behavior, CLI failures, and floating-point boundaries. Live evaluation remains available through the SDK-backed client.

The checked-in Action uses environment variables and a Bash argument array to pass inputs to the CLI. It rejects conflicting baseline sources and candidate names instead of guessing which one the caller intended.

Scope and open-source contribution

jevcheck is an MIT-licensed Python library and command-line tool, with a composite GitHub Action. Version 0.2.0 is published on PyPI and classified as alpha-stage software. Its current scope is TypeSafe Jev’s typed System One answers.

Hashgraph Online’s Awesome AI Plugins catalog includes jevcheck in Development & Workflow. Maintainer kantorcodes merged the listing contribution, PR #395, on September 20, 2026.

The replay example demonstrates the implementation’s decisions; it is not a measured comparison of live models or evidence of production adoption. A passing contract covers the selected cases and rules. Broader evaluation still depends on the quality and coverage of those cases.

Follow the implementation

Reviewed September 20, 2026. Implementation links are pinned to the inspected source snapshot.

Next project

Meridian