LlmCall.Codec.Conformance (llm_call v0.1.0)

Copy Markdown View Source

Fixture-based assertions for codec authors.

These are test support, not a runtime dependency: a third-party codec can call them from its own suite to check the parts of the behaviour that are easy to get subtly wrong. Every function returns :ok or raises an ExUnit.AssertionError naming the failed expectation.

test "my codec conforms" do
  Conformance.assert_info(MyCodec)
  Conformance.assert_path(MyCodec, request, "/v1/my-endpoint")
end

The assertions take the caller's fixtures, so an out-of-tree codec supplies its own payloads: nothing here assumes a specific provider.

Summary

Functions

Couples capabilities.streaming to the streaming callbacks.

Asserts that capabilities/2 and validate_request/2 agree, in both directions.

Asserts that a provider error envelope decodes to a canonical %Error{}.

Asserts that a successful payload decodes to a canonical response.

Asserts well-formed info/0 metadata (see LlmCall.Conformance).

Asserts that a malformed payload is a typed error, never a raise and never a fabricated response.

Asserts the encoded request's method, path, body shape, and credential-free headers.

Asserts that a codec rejects a request it cannot represent in full.

Asserts which decoded content parts a codec accepts back in an assistant message.

Drives a codec's stream over the caller's chunks and asserts event shape and order.

Asserts that StreamEvent.raw follows the capture_raw policy.

Asserts that a streaming request actually asks the provider to stream.

Asserts that a stream state is bound to the codec that created it.

Asserts that a streamed tool-call sequence assembles into complete calls.

Functions

assert_capabilities(codec, request, config)

@spec assert_capabilities(module(), LlmCall.Request.t(), LlmCall.Config.t()) :: :ok

Couples capabilities.streaming to the streaming callbacks.

streaming: true requires all three callbacks; streaming: false requires none of them. Capabilities are per codec, not per request, so a protocol that streams only for some models cannot express that here — declare true and reject the other requests in validate_request/2.

assert_capability_consistency(codec, request, config)

@spec assert_capability_consistency(module(), LlmCall.Request.t(), LlmCall.Config.t()) ::
  :ok

Asserts that capabilities/2 and validate_request/2 agree, in both directions.

A declared capability must accept a minimal representative: tools, tool_choice, structured_output, modalities, image_input, file_input, reasoning.supported, every declared efforts level, budget_tokens, and summary. An undeclared one must be a validation error. The base request must otherwise be valid for the codec.

reasoning.adaptive, reasoning.default_effort, parallel_tool_calls, streaming_tool_calls, prompt_cache, and stateful are declarations only: the canonical IR has no request field that expresses them, so there is nothing to check in either direction.

assert_capture_raw(codec, provider_payload)

@spec assert_capture_raw(module(), LlmCall.wire_value()) :: :ok

assert_decode_error(codec, provider_payload, opts \\ [])

@spec assert_decode_error(module(), LlmCall.wire_value(), keyword()) :: :ok

Asserts that a provider error envelope decodes to a canonical %Error{}.

Pass code: and retryable?: to pin the normalization: without them the assertion only sees that the result is a provider error, and a codec that ignores the payload entirely would pass.

assert_decode_response(codec, provider_payload)

@spec assert_decode_response(module(), LlmCall.wire_value()) :: :ok

Asserts that a successful payload decodes to a canonical response.

Checks the IR boundary: canonical parts, a stop reason from the closed union, canonical usage, and raw as a map or nil. Provider-specific shapes leaking through content fail here.

assert_info(codec)

@spec assert_info(module()) :: :ok

Asserts well-formed info/0 metadata (see LlmCall.Conformance).

assert_invalid_response(codec, provider_payload)

@spec assert_invalid_response(module(), LlmCall.wire_value()) :: :ok

Asserts that a malformed payload is a typed error, never a raise and never a fabricated response.

Call it with a payload the codec cannot decode — %{} for most protocols.

assert_path(codec, request, expected_path, method \\ :post)

@spec assert_path(module(), LlmCall.Request.t(), String.t(), :post | :get) :: :ok

Asserts the encoded request's method, path, body shape, and credential-free headers.

method defaults to :post. A codec that uses encode: false for a per-request method passes the one it produces.

assert_rejects(codec, request, config)

@spec assert_rejects(module(), LlmCall.Request.t(), LlmCall.Config.t()) :: :ok

Asserts that a codec rejects a request it cannot represent in full.

The core promise is that an unrepresentable canonical field is a validation error, never a silent drop. This is the assertion a package uses to prove it, and the one that pins a protocol's unsupported fields as a test rather than a convention.

assert_replays(codec, request, expectations)

@spec assert_replays(module(), LlmCall.Request.t(), keyword()) :: :ok

Asserts which decoded content parts a codec accepts back in an assistant message.

A response the caller can decode but not replay makes a tool loop impossible, so the accept/reject boundary is part of the contract rather than an implementation detail. Pass the rule per part and the assertion builds the assistant turn itself:

Conformance.assert_replays(codec, request,
  text: :ok,
  tool_call: :ok,
  thinking: :rejected,
  thinking_without_signature: :rejected,
  raw: :rejected
)

:ok means encode_request/2 succeeds with that part in an assistant message; :rejected means it is a :validation error that names the part — never a silent drop. A part the caller omits is not checked: a protocol that cannot express it at all has no rule to pin.

The parts are :text, :thinking, :thinking_without_signature, :tool_call, and :raw. What each codec answers is the table in docs/design.md §8.2.

assert_reserved_options(codec, request, config, reserved_options)

@spec assert_reserved_options(module(), LlmCall.Request.t(), LlmCall.Config.t(), [
  map()
]) :: :ok

assert_stream(codec, request, chunks)

@spec assert_stream(module(), LlmCall.Request.t(), [LlmCall.wire_value()]) :: :ok

Drives a codec's stream over the caller's chunks and asserts event shape and order.

Order rules: :message_start at most once and before any delta; :message_stop/:error at most once and terminal. A {:done, state} result stops the stream, and finish_stream/2 afterwards must not emit content.

assert_stream_capture_raw(codec, request, chunks)

@spec assert_stream_capture_raw(module(), LlmCall.Request.t(), [LlmCall.wire_value()]) ::
  :ok

Asserts that StreamEvent.raw follows the capture_raw policy.

:none and :response keep no provider event; :all keeps at least one, and every retained payload is a map. A codec that copies provider chunks into every event regardless of the policy fails here.

assert_stream_request(codec, request, opts \\ [])

@spec assert_stream_request(module(), LlmCall.Request.t(), keyword()) :: :ok

Asserts that a streaming request actually asks the provider to stream.

This is the encoding-side counterpart to assert_stream/3. A codec can implement every streaming callback and still never request SSE, in which case the transport receives a buffered response and the decoders are dead code.

Three things must hold:

  • stream: true and stream: false must encode to different requests, so the flag reaches the wire at all;
  • the streaming request must carry a streaming signal: body["stream"] == true, an SSE query value (alt=sse), a stream query flag, or a path that differs from the non-streaming one and names the streaming endpoint;
  • the non-streaming request must not carry that signal.

A signal is a disjunction, so this cannot tell you that a protocol needs all of its signals (Gemini needs both alt=sse and the streaming path). Pass require: [query: %{"alt" => "sse"}] or require: [body: %{...}] to pin the ones your protocol cannot lose.

Pass marker: (1-arity, receives the streaming request) when the protocol signals streaming some other way, and config: when the codec needs a non-default LlmCall.Config to encode. The base request must be streamable: forcing stream: true on a request the codec rejects (Gemini with an image modality, say) is reported as such, and a codec that declares streaming: false is named as the cause.

assert_stream_roundtrip(codec, request)

@spec assert_stream_roundtrip(module(), LlmCall.Request.t()) :: :ok

assert_stream_state_ownership(codec, other_codec, request)

@spec assert_stream_state_ownership(module(), module(), LlmCall.Request.t()) :: :ok

Asserts that a stream state is bound to the codec that created it.

Passing one codec's state to another must be a validation error, so a caller cannot mix two protocols' stream state by accident.

assert_tool_call_assembly(codec, request, chunks, expected_calls, opts \\ [])

@spec assert_tool_call_assembly(
  module(),
  LlmCall.Request.t(),
  [LlmCall.wire_value()],
  pos_integer(),
  keyword()
) :: :ok

Asserts that a streamed tool-call sequence assembles into complete calls.

Runs the events through LlmCall.StreamAssembler, so a codec fails here if it fragments arguments in a way the assembler cannot key by index, or invents arguments the provider never sent.

expected_calls pins the count; pass json_decoder: to supply a decoder other than :json (the assembler never decodes JSON itself).