Status: draft for implementation
Date: 2026-11-12
Project: llm_call
Module namespace: LlmCall
1. Summary
LlmCall is a call-oriented SDK for LLM APIs: a canonical wire IR, protocol
codecs, and a one-call client on Finch whose JSON layer is replaceable.
The library ships the canonical IR, the codec behaviours, the conformance suites, four reference adapters — the dialect families the IR is written against — and a set of vendored media adapters. Provider coverage is still not a goal: the IR's value comes from the breadth of knowledge behind it, not from the breadth of shipped code. A vendored adapter tracks its provider's API; the IR it maps into is the stable surface.
LlmCall.complete/3 turns a provider-neutral request into a round trip:
validate, encode, send, decode. It deliberately does not own model discovery,
authentication lookup, approvals, retries, or an agent loop.
The encode/decode functions remain public, so a caller that already owns a transport can use the codecs without the client.
The core idea:
canonical request
|
v
LlmCall.Client (provider + endpoint + defaults)
|
v
LlmCall.Codec.encode_request/2
|
v
encoded request ---- LlmCall.HTTP (default Finch) ----> provider
|
v
provider response term (LlmCall.JSON, default Elixir JSON)
|
v
LlmCall.Codec.decode_response/2
|
v
canonical responseThis is intentionally a smaller layer than a complete LLM client. It can be used by a higher-level client, an agent framework, or a custom application.
2. Position
2.1 What problem this solves
Providers do not agree on:
- request envelopes (
messagesversusinput,contents, etc.) - system prompt placement
- tool definitions and tool choice
- tool call and tool result representation
- content blocks and multi-modal inputs
- response envelopes (
choices, output items, content blocks) - streaming event shapes
- stop reasons, usage accounting, and errors
- state and continuation semantics
LlmCall provides one stable Elixir data model for these wire-level concepts.
2.2 What this is not
LlmCall is not:
- a provider gateway: it does not route by model name or keep a model catalog
- a model registry or pricing database
- a tokenizer or context-window planner
- a retry/rate-limit/circuit-breaker layer
- an authentication manager
- a tool executor
- an approval, timeout, or audit system
- an agent loop, memory, prompt-template, or chain framework
- a replacement for
ReqLLMor LangChain - a provider catalog, or a promise to support every provider. Beyond the reference and vendored adapters in this repository, an adapter is an application's own module or a third-party package (see §14)
Its HTTP execution is in the core on Finch; JSON encoding is behind
LlmCall.JSON (Elixir JSON by default). SSE framing (LlmCall.SSE) is a core
module, and WebSocket connections go through the LlmCall.WS behaviour with
LlmCall.WS.Mint as the implementation. HTTP and SSE are not replaceable seams;
WebSocket has a behaviour, but no config-level transport resolution.
2.3 Relationship to tool_call
LlmCall does not depend on the tool_call project.
The tool concepts in LlmCall are wire data only:
LlmCall.ToolSpec: what the model may callLlmCall.ToolCall: what the model asked to callLlmCall.ToolResult: the result of a call, ready to send back to the model
The application maps between them; neither project depends on the other:
A ToolCall.describe/2 entry becomes a %LlmCall.ToolSpec{} with its
name, description and parameters.
ToolCall.describe/2 returns a provider-neutral name/description/parameters
per tool, and a decoded %LlmCall.ToolCall{} carries the name and arguments
ToolCall.run/4 needs.
2.4 Relationship to LangChain and ReqLLM
| Project | Focus |
|---|---|
| LangChain Elixir | Chains, conversation flow, tool execution, callbacks |
| ReqLLM | Complete LLM client with provider transports, registry, streaming, tools |
LlmCall | Dialect codecs, canonical wire IR, one-call client |
LlmCall can be used underneath those systems. Its client is deliberately narrow
— one call, no model metadata, no tool execution, no retry policy.
3. Goals
- Provide a small, explicit canonical wire IR.
- Support multiple native protocol families with independent codecs.
- Preserve provider-specific data through raw/escape-hatch fields.
- Ship a one-call client on Finch whose JSON layer is replaceable.
- Keep auth keys caller-supplied, and keep model registries, tool execution, approval/audit policies and retries out of the core.
- Let applications plug in any tool-call framework.
- Make dialect adapters testable with fixtures, without network access.
- Make capability differences explicit instead of silently dropping fields.
- Keep OpenAI Chat Completions and OpenAI Responses as distinct protocols.
4. Non-goals
The core must not contain:
- API key storage, key lookup, OAuth or AWS signing
- model registry or capability database
- retry/backoff/rate-limit policy
- tool execution or callbacks
- approval, audit, or timeout policies
- conversation persistence or server-side state management
- evaluation, rerank, or embedding orchestration
- prompt templates, chains, memory, or agent loops
HTTP execution is in the core on Finch; JSON encoding is behind LlmCall.JSON
and can be replaced. SSE framing lives in LlmCall.SSE, and WebSocket
connections use the LlmCall.WS behaviour; neither is part of the completion transport. The core may define the data
needed by the layers above, but not implement them.
5. Design principles
- A wire protocol, not a provider. A dialect codec describes a wire format. A provider is an endpoint plus authentication plus model availability.
- Canonical does not mean lossless. Canonicalize common concepts; keep
unknown/provider-specific data in
raworprovider_options. - No silent drops. If a canonical field cannot be represented by a protocol, fail with a protocol/capability error or require an explicit provider option.
- Terms, not JSON strings. Codecs return JSON-compatible Elixir terms.
The client encodes them with
Config.json. - No hidden state. Streaming and continuation state are explicit.
- Tool execution is outside. Tool calls are data; running them is the application's job.
- Small core, one replaceable edge. HTTP is Finch; JSON sits behind a behaviour with a default. A tool framework is used directly, not wrapped.
- No
term()in canonical public fields. Canonical fields use explicit unions/structs.term()is reserved for genuinely opaque adapter state (for example, a protocol stream state) and transport-decoded input at the behaviour boundary.
6. Approaches considered
A. Transport-free codec library
Canonical IR and codecs only; the application owns HTTP, SSE and JSON.
- smallest dependency surface; usable from any stack
- every caller re-implements SSE framing,
[DONE]handling and error mapping
This was the first shape of the library. Superseded because duplicating that layer in every application cost more than shipping it.
B. SDK with a replaceable JSON edge (chosen)
Canonical IR and codecs plus a one-call client on Finch; JSON lives behind
LlmCall.JSON, with Elixir JSON as the default; providers are user modules.
- one call works after
mix deps.get - the JSON module can be replaced per application, provider or client
- costs two runtime dependencies (Finch, Mint.WebSocket)
C. Provider gateway
Built-in provider catalog, routing by model name.
- the most convenient first call
- the library inherits every base URL, auth shape and model rename, and it contradicts "no model catalog"
Rejected.
D. Framework-specific adapters only
Implement adapters for one agent framework.
Rejected: duplicates protocol work across frameworks and creates no reusable foundation.
7. Architecture
+----------------------+
| application / agent|
+----------+-----------+
|
canonical request / response
|
+----------v-----------+
| LlmCall |
| canonical wire IR |
+----------+-----------+
|
+----------v-----------+
| LlmCall.Dialect.* |
| encode/decode |
+----------+-----------+
|
encoded request / events
|
+----------v-----------+
| LlmCall.HTTP |
| Finch |
+----------+-----------+
|
HTTP / JSON
|
+----------v-----------+
| LLM provider |
+----------------------+Core layers
- Canonical IR — request, messages, content, tool data, response, streaming events, usage, errors.
- Dialect codec — encoding request terms and decoding response/event terms.
- Capabilities — explicit protocol/model capability description.
- Client — provider declaration, endpoint, auth shape, and the HTTP scope.
- Transport —
LlmCall.HTTPon Finch, and theLlmCall.WSbehaviour for WebSocket connections.LlmCall.JSONis the one replaceable seam.
8. Canonical IR
The names below use the existing project namespace, LlmCall.
8.0 Single source of truth and raw capture
Normalized fields are the authoritative contract. raw is an optional,
immutable provider snapshot, never a second source that callers or codecs may
use to update canonical fields after decoding.
Rules:
rawisnilby default.- Raw capture is controlled by
LlmCall.Config.capture_raw, and it covers every payload a codec keeps,Error.rawincluded::none— no raw payloads (default):response— retain the full response payload onResponse.raw:all— also retain provider events onStreamEvent.raw
- A provider error envelope is a response payload:
Error.rawfollows the same policy, so:noneleaves itnil. - A codec decodes normalized fields in one pass. It must not later overwrite
them from
raw. - Do not store the same full payload in
Usage.raw,provider_metadata, andResponse.raw. Those duplicate slots are intentionally absent from this design. ToolCall.raw_argumentsis an optional lossless wire projection. If present,argumentsis the decoded form ofraw_arguments(ornilwhen decoding fails); the two fields must not be updated independently.Content.Rawis only for an unknown content block. It is not a copy of the full provider response.LlmCalldoes not provide a JSON encoder; if raw payloads are persisted, the application decides whether to serialize them.- The canonical response/content/usage should be sufficient for normal use; raw is for diagnostics, provider extensions, and future capabilities.
8.1 Request
A request is a %LlmCall.Request{} holding model, system, messages, tools,
tool_choice, output, modalities, reasoning, params, provider_options and
stream. params is an atom-keyed map of the six sampling and length controls
(temperature, top_p, top_k, max_tokens, stop, seed); a nil value means
"provider default" and is not sent. The struct defaults are tool_choice: :auto,
output: :text, modalities: [:text], params: %{}, provider_options: %{} and
stream: false.
Fields:
model— provider/model identifier as understood by the selected protocol and endpoint.system—nil, a string, or a list of text parts. The codec decides where it belongs in the provider request.messages— user/assistant turns. Tool results are content parts, not a mandatory:toolrole.tools— optional neutral tool specifications.tool_choice—:auto | :none | :required | {:tool, name} | {:allowed, [name]}.output—:text | :json_object | {:json_schema, name, schema, strict?}. This describes the text format only.modalities— which output modalities the caller accepts, default[:text].[:text, :image]means a response may contain generated image parts as well as text. It is a capability request, not a demand:[:text, :image]still allows a text-only response. See Media generation.reasoning— optional%LlmCall.Reasoning{}; controls thinking/reasoning effort or budget where the protocol supports it. Provider-specific controls remain inreasoning.provider_options.params— common sampling/length controls.provider_options— explicit escape hatch for provider-specific options.stream— whether the caller wants streaming events.
provider_options is intentionally not a catch-all for values that the
canonical layer can represent. It is for features that are genuinely
provider-specific.
A codec reserves the wire keys it derives from the request envelope (model,
messages, system, contents, tools, stream), so a request-level override
is a validation error. Parameter-level native keys stay reachable:
provider_options.gemini.generationConfig.candidateCount or a nested
tool_choice.disable_parallel_tool_use are explicit caller intent, merged beside
the canonical values. Config.default_provider_options may not override a
parameter-level canonical field, so a default cannot silently beat an explicit
request. Reasoning-specific native controls belong on
Reasoning.provider_options, which merges into the protocol's reasoning object
(and is therefore subject to the same reserved-key rule, because OpenAI Chat
merges it into the body root).
The namespace keys are checked, not just their values: a codec names the
namespaces it reads (:openai, [:gemini, :google], ...) and rejects any other
key in a request, so provider_options: %{openai_chat: ...}, a string key, or
a present-but-nil namespace is a validation error rather than a silently ignored
option. Config.default_provider_options is application-wide: only the
namespaces this codec reads are checked there, so one Config can carry defaults
for several providers.
A nested native object stays reachable in both sources, with its canonical keys
reserved: generationConfig.candidateCount and
stream_options.include_obfuscation are accepted, generationConfig.temperature
and stream_options.include_usage are not, because the codec derives those from
params.
Fields a protocol cannot carry
A protocol with no wire field for a canonical value must reject it, not drop it — except for the fields below, which are local to the IR and never meant to reach a provider. They are listed deliberately so a reviewer does not have to rediscover them:
ToolCall.index— stream bookkeeping.LlmCall.StreamAssemblerkeys fragments by it; a request tool call has no index on the wire.Content.Image.media_typewhensourceis a URL (Anthropic): the protocol infers the type from the URL, and itsimageblock has nomedia_typefor that source. Base64 sources do carry it.- The
nameinoutput: {:json_schema, name, schema, strict?}(Gemini): Gemini'sresponseSchemahas no schema name. The name stays a caller-side label; the codec keeps the schema and drops the label. OpenAI emits it; Anthropic rejects structured output entirely. Request.modalitiesequal to[:text](the default) — the protocol default, so it is omitted rather than written.
Everything else is a rejection: an unsupported content variant
(Content.Video in a chat request), ToolSpec.strict on Anthropic or Gemini,
{:allowed, names} on OpenAI or Anthropic, and a params key the codec does not
implement.
test/llm_call/write_path_audit_test.exs enumerates every canonical-derived
wire key and asserts each is rejected from every source that can write it, so
this list is the only place a dropped field may be added.
The decode side has the mirror problem: a provider field can be ignored. Envelope
data, echoes (choices[].index, candidates[].index), discriminators
(tool_calls[].type, "type" => "message"), and an implied role
(message.role) are deliberately not modelled; they remain reachable through
Response.raw when capture_raw keeps it.
test/llm_call/decode_path_audit_test.exs deletes every provider leaf and
requires the canonical struct to change, or the leaf to be listed with a reason.
Adding a field to a provider fixture without mapping it fails that test.
The streaming path has the same rule one level down:
test/llm_call/stream_delta_audit_test.exs deletes each chunk leaf and
compares the whole {type, delta, index} event list, so every StreamDelta field
and every StreamEvent.index has a provider source. The same mutation applied to
a field the codec does not know about must produce a Raw event, never silence.
8.2 Message
A message is a %LlmCall.Message{}: role is :user or :assistant, and
content is a list of content parts.
System instructions stay in Request.system, because provider placement
differs:
- OpenAI Chat: system/developer message
- OpenAI Responses:
instructions - Anthropic: top-level
system - Gemini:
systemInstruction - Bedrock Converse: top-level
system
Tool results are content parts inside a user message — a %LlmCall.ToolResult{}
with call_id, name, content (a string or text parts) and is_error.
Adapters translate this into provider-specific forms:
- OpenAI Chat:
role: "tool"message - OpenAI Responses:
function_call_outputinput item - Anthropic:
tool_resultcontent block - Gemini:
functionResponsepart - Bedrock Converse:
toolResultblock - Cohere:
toolmessage
8.3 Content parts
Initial canonical content types:
%LlmCall.Content.Text{text: String.t()}%LlmCall.Content.Image{source: source(), media_type: String.t() | nil}`%LlmCall.Content.File{source: source(), media_type: String.t() | nil, filename: String.t() | nil}
%LlmCall.Content.Video{source: source(), media_type: String.t() | nil}%LlmCall.ToolCall{}%LlmCall.ToolResult{}%LlmCall.Content.Thinking{text: String.t(), signature: String.t() | nil}%LlmCall.Content.Raw{type: String.t() | nil, data: map()}
ToolCallandToolResultare the shared canonical structs defined below; there are no duplicateContent.ToolCallorContent.ToolResultwrappers.Rawis an escape hatch for provider-specific blocks that the codec does not understand. It must be preserved in canonical responses and rendered back when possible.Replay
A response that can be decoded but not sent back makes a tool loop impossible, so which content parts a codec accepts in an assistant message is part of the contract. The answers differ because the request shapes do:
"Rejected" is aPart OpenAI Chat OpenAI Responses Anthropic Messages Gemini GenerateContent Content.Textyes yes yes yes ToolCallyes yes yes yes Content.Thinkingwithsignaturerejected rejected yes yes Content.Thinkingwithoutsignaturerejected rejected rejected yes Content.Rawrejected rejected yes yes :validationerror that names the part, never a silent drop. OpenAI's request bodies are closed sets, so no provider-specific block can go back in, and its reasoning output is not replayable either — Responses continues the conversation withprevious_response_idinstead. Anthropic rejects an unsigned thinking block because its API does. Gemini accepts an unsigned one because its API only requires a signature where a tool call follows the thought.Content.Rawis rendered back as itsdataalone: a block decoded from a provider already contains the discriminator, sotypestays a caller-side label. A block built by hand must therefore carry the discriminator insidedata.LlmCall.Codec.Conformance.assert_replays/3pins this table per codec.8.3.1 Reasoning
Reasoning config is separate fromparamsbecause providers expose different axes:%LlmCall.Reasoning{}carrieseffort(:none | :minimal | :low | :medium | :high | :xhigh | :max | nil),budget_tokens(a positive integer ornil),summary(:auto | :concise | :detailed | nil) andprovider_options(a map).effortis a canonical hint, not a guarantee that every provider accepts every value.nilmeans the provider/model default.budget_tokensis for providers that expose an explicit thinking budget.summaryis for providers that expose a separate reasoning-summary control;nilmeans "do not ask for one".- A provider adapter maps
effort/budget_tokens/summaryto its native fields; if it cannot express a non-nil canonical field, it must return a validation/capability error rather than silently dropping it. provider_optionsis for native controls with no canonical equivalent.
- OpenAI:
reasoning.efforton Responses;reasoning_efforton Chat. - Anthropic: adaptive thinking / budget thinking and effort controls; exact shape depends on the API version.
- Gemini:
thinkingConfiglevel or budget, depending on model. - Bedrock/other: native provider fields through the adapter or
provider_options.
8.4 ToolSpec
%LlmCall.ToolSpec{}carrieskind(:functionor:builtin),name,description,parameters(a JSON Schema object ornil),strict(nilor a boolean) andprovider_options.parametersis a JSON Schema object (map()), not a JSON string or a provider-specific schema struct.nilis valid for a built-in tool that has no function schema.kind: :functionis the normal function-tool case.kind: :builtinis a provider-hosted tool (web search, grounding, server tools). Itsnameis the vendor's own identifier (web_search_20250305,google_search), and its native knobs go inprovider_options, merged into the codec's tool shape.parametersandstrictare rejected for this kind, because a hosted tool has no local schema.- The library never interprets a hosted tool's name; it answers only "will this
codec forward it?" through
Capabilities.builtin_tools. - No callback, function, approval policy, or execution metadata belongs here.
strictis optional; dialect adapters reject it when unsupported or map it to their equivalent.
8.5 ToolCall
%LlmCall.ToolCall{}carriesid,name,arguments,raw_arguments,arguments_error(nil | {:invalid_json, reason} | :no_json_decoder) andindex.Theargumentsmap is the decoded form where decoding is safe. Keep the raw JSON string because:- OpenAI Chat and Cohere carry arguments as a JSON string.
- Anthropic, Gemini, and Bedrock commonly carry a decoded object.
- Applications may need to preserve invalid or partial fragments.
- Some frameworks expect the raw string.
id: nilis valid because GeminifunctionCalldoes not have a call id.Whenraw_argumentsis present,argumentsis its decoded value. If decoding fails,argumentsisnilandarguments_errorsays why —{:invalid_json, reason}for a payload that is not decodable JSON or not an object, and:no_json_decoderwhenLlmCall.StreamAssemblerwas built without a decoder.reasonis a message, never the payload:raw_argumentsis the only place the payload travels. The codec must not silently turn malformed arguments into%{}.8.6 ToolResult
%LlmCall.ToolResult{}carriescall_id,name,content(a string or text parts) andis_error.Adapters validate required identifiers:- OpenAI Chat requires
call_id. - OpenAI Responses requires
call_id. - Anthropic uses
tool_use_id. - Gemini uses
nameand no id. - Bedrock uses
toolUseId. - Cohere uses
tool_call_id.
8.7 Response
%LlmCall.Response{}carriesid,model,content(content parts),stop_reason(a member of the union in §8.11),usageandraw.contentis a list of content parts, not provider-specificchoicesor output items.rawis amap() | nilprovider snapshot, populated only whenConfig.capture_rawis:responseor:all.Response.rawmust not be treated as a second response model. Usecontent/usage/stop_reason; use raw only when provider-specific information was not represented canonically.LlmCall.Response.text/1andtool_calls/1are conveniences overcontent; adding more does not change the IR.8.8 Usage
%LlmCall.Usage{}carries six counters —input_tokens,output_tokens,total_tokens,cache_read_tokens,cache_write_tokens,reasoning_tokens— each a non-negative integer ornil.Pricing and cost calculation are outside the core.Cost is not a wire field. It depends on provider, model, region, cache read/write pricing, batch/tier discounts, and reasoning-token pricing. The canonicalUsagerecords the token dimensions needed by an upper layer, butLlmCalldoes not store price tables or computed cost. If an application needs cost tracking, use an optionalllm_call_costpackage or its own model-pricing layer. Provider-specific billing fields, if returned, belong inrawor a provider-specific extension.8.9 Error
%LlmCall.Error{}carrieskind,message,code,status,reason,retryable?,retry_after_msandraw;kindis one of:validation,:invalid_response,:provider,:transport.codeis the provider's machine-readable code as a string (OpenAI'scode, Anthropic's error type, Google's status such as"RESOURCE_EXHAUSTED", MiniMax's numeric code stringified).statusis the HTTP status, and only the client sets it: a codec never sees the response status, so it always leaves itnil.reasonis the structured counterpart of a:transportmessage, and only the client sets it:{:unknown_pool, name}when the configured Finch pool is not running,:exceptionwhen the HTTP transport raised,:disconnectedwhen the Responses WebSocket is not connected,{:interrupted, why}when an in-flight lane was dropped,{:connection_exited, why}when the WebSocket connection process ended, and otherwise the transport's own reason (:econnrefused,:timeout,:closed,:nxdomain,{:tls_alert, ...}). A codec leaves itnil, and so does a failure that carries astatus. Match onreasoninstead of parsingmessage; a non-transport error never sets it.retry_after_msis transport metadata: the client parses theRetry-Afterresponse header (delta-seconds or IMF-fixdate) and sets it on the error; a codec leaves itnil. It is a hint, likeretryable?; backoff policy stays outside.rawis the provider payload the error was decoded from, and it followsConfig.capture_rawlike every otherraw: it isnilunless the policy is:responseor:all.kindsays who is speaking::validation— the request cannot be expressed (codec, or the client for an unencodable body or an undeclared codec name);:invalid_response— a payload could not be decoded, by the codec or by the client when a2xxbody is not JSON;:provider— a provider error envelope, decoded by the codec, withstatusfilled in by the client;:transport— never reached a provider: connection failure, timeout, or a non-2xxbody that is not a JSON envelope.
:transport; the client never produces:provider.execute/2andfetch/2are the exception: they perform a request the caller already encoded, with no codec to decode an envelope, so they report any non-2xxas:transportwithstatusset. Usecomplete/3,generate/3orpoll/3when you want the provider envelope decoded.8.10 Capabilities
Capabilities are explicit and per protocol/config. They are protocol-level:capabilities/2takes the request for signature symmetry but does not inspect the model; a model-specific limit surfaces as a provider error rather than a fabricated local guess.Field Values What it says toolsboolean function tools are accepted builtin_toolsfalse/:passthrough/ list of namesprovider-hosted tools parallel_tool_calls,tool_choiceboolean structured_output:none/:json_object/:json_schemamodalitieslist of :text,:image,:videoimage_input,file_inputboolean reasoning%LlmCall.Reasoning.Capabilities{}supported,efforts,default_effort,budget_tokens,adaptive,summarystreaming_tool_calls,prompt_cache,stateful,streamingboolean builtin_toolsdeclares provider-hosted tools:falserejects them,:passthroughforwards any name the caller supplies, and a list forwards only those names.validate_request/2on a codec can reject unsupported combinations before a transport sends anything. No silent field dropping.8.11 Canonical type specifications
Each canonical module's@type tis the authoritative shape, and ExDoc renders it next to the module. What a shape cannot state is recorded here: which unions are closed, and what each struct is for.Four unions are closed, and changing one is an IR change:@type capture_raw :: :none | :response | :all @type modality :: :text | :image | :video @type wire_value :: nil | boolean() | number() | String.t() | [wire_value()] | %{optional(String.t()) => wire_value()} @type stop_reason :: :end_turn | :tool_calls | :max_tokens | :stop_sequence | :content_filter | :error | :unknownModule What it carries Rules LlmCall.Requestmodel, system, messages, tools, tool_choice, output, modalities, reasoning, params, provider_options, stream §8.1 LlmCall.Messagerole and a list of content parts §8.2 LlmCall.Content.Text/.Image/.File/.Video/.Thinking/.Rawone content part each; a source()is a string,{:url, _},{:base64, _}or{:file_id, _}§8.3 LlmCall.Reasoning/LlmCall.Reasoning.Capabilitieseffort, budget_tokens, summary, provider_options; what a protocol supports §8.3.1 LlmCall.ToolSpec/.ToolCall/.ToolResulttool wire data; no schema derivation, no dispatch §8.4–§8.6 LlmCall.Responseid, model, content, stop_reason, usage, raw §8.7 LlmCall.Usagesix token counters, provider-reported and token-only §8.8 LlmCall.Errorkind, message, code, status, reason, retryable?, retry_after_ms, raw §8.9 LlmCall.Capabilitiesprotocol-level support flags §8.10 LlmCall.Configdefault_provider_options, capture_raw, json §10, §16 LlmCall.EncodedRequestmethod, path, query, headers, body §11 LlmCall.StreamStatethe owning codec plus opaque data §12 LlmCall.Codec.Infoname, protocol, wire version, ir_version §10 LlmCall.StreamEventand itsStreamDeltatypes are specified in Streaming.9. OpenAI Chat Completions vs Responses
These are separate dialect adapters, not one endpoint toggle.
Implement as:Aspect Chat Completions Responses Endpoint /v1/chat/completions/v1/responsesRequest messages[]inputitems or stringSystem system/developer message instructionsState stateless optional store+previous_response_idResponse choices[0].messageoutput[]items +output_textTool call assistant tool_callsfunction_calloutput itemTool result role: "tool"messagefunction_call_outputinput itemBuilt-in tools limited function tools broader built-in tools depending model/account Streaming choices[0].deltachunkstyped response.*eventsToken limit max_tokens/max_completion_tokensmax_output_tokensCompatibility widely copied by OpenAI-compatible APIs mostly OpenAI-native
Both are implemented. The canonical IR is shared, but the response, event and state mappings are independent. Responses is available over HTTP/SSE and, for persistent multiplexed requests, over the WebSocket connection in §24.1.OpenAI Responses state is deliberately provider-specific.LlmCall.Dialect.OpenAI.Chat # POST /v1/chat/completions LlmCall.Dialect.OpenAI.Responses # POST /v1/responsesprevious_response_idandstorelive inRequest.provider_options(for example under an:openaikey) and are interpreted by the Responses adapter;LlmCalldoes not model server-side conversation state. The canonicalResponse.idremains available for the caller to continue the conversation.10. Codec contract
%LlmCall.Codec.Info{}carriesname,protocol,versionandir_version.nameidentifies the codec instance andprotocolthe wire family; they are equal for every codec today (one codec per family) and differ only when one module serves a family with variants.versionis the wire version the codec targets — Anthropic's2026-10-21, Gemini'sv1beta, Seedance'sv3— andnilwhen the protocol has no versioned surface (OpenAI Chat and Images).ir_versionis the canonical IR contract the codec was written against.Callback Contract info/0%LlmCall.Codec.Info{}encode_request(request, config){:ok, %EncodedRequest{}}or an errordecode_response(term, config){:ok, %Response{}}or an errorvalidate_request(request, config):okor a:validationerrorcapabilities(request, config)%LlmCall.Capabilities{}decode_error(term, config)%LlmCall.Error{}, never an error tupleinit_stream(request, config){:ok, %StreamState{}}or an errordecode_stream_event(term, config, state){:ok, events, state},{:done, state}, or an errorfinish_stream(state, config){:ok, events, state}or an errorinit_stream/2,decode_stream_event/3andfinish_stream/2are optional, so a non-streaming codec implements no dummy callbacks.LlmCall.Codecis a behaviour.LlmCall.Dialect.*modules are concrete dialects of one wire protocol; each implements that behaviour. The namespace isDialectrather thanProtocolso it cannot be read as Elixir'sdefprotocol, and the same word is used for theusemacro (LlmCall.Dialect.Adapter,LlmCall.Media.Dialect.Adapter).info/0lets a codec declare its protocol name, wire version, and the canonical IR version it implements.init_stream/2,decode_stream_event/3, andfinish_stream/2are optional so a non-streaming codec does not have to implement dummy callbacks. When streaming is unavailable, the public API returns a typedLlmCall.Errorrather than raising.Streaming state is never exposed as a bareterm(): it is wrapped in opaquely typedLlmCall.StreamState{}, and the public API checksStreamState.codec(state) == selected_codecbefore delegating, so passing an Anthropic state to the OpenAI codec is a typed error. Codec implementations useStreamState.data/1andStreamState.new/2; thedataremains deliberately opaque.%LlmCall.Config{}contains codec defaults and capture policy, not the canonical model selection:default_provider_options,capture_rawandjson.default_provider_optionsis merged beforeRequest.provider_options; the request value wins per key, and a namespace present in both merges recursively, so a nested native object keeps the config keys the request does not repeat.Request.modelis the only model selection.Configdoes not contain credentials or a base URL.10.1 Codec discovery and extension
Codec selection is explicit:the module is passed toLlmCall.encode/3, or named bycodec:on a call.There is no provider or codec registry in the core. A higher-level client may map provider/model names to codec modules, butLlmCalldoes not. This keeps a new dialect adapter independent of model catalogs and authentication.Adding a new codec should require only:- A module that
usesLlmCall.Dialect.Adapter(orLlmCall.Media.Dialect.Adapter), declaring itsprovider_optionsnamespaces (@namespaces, ornamespaces:on the media macro) and supplying@path,@headers,info/0,capabilities/0, the protocol's ownvalidate_request/2, and a privatebody/2that returns the wire body. - Dialect-specific request/response/stream fixtures.
- Capability declarations for the models/endpoints the codec supports.
use LlmCall.Dialect.Adapter, codec: :openai_chatinjects the behaviour, the canonical aliases,@codec, andencode_request/2;encode: falseleavesencode_request/2to the codec when the path or method varies per request. The macro reads@pathand@headersonce the module finishes and raises at compile time if either is missing. The media macro additionally injects the sharedvalidate_request/2pipeline; itsnamespaces:option is required, so a new media codec has to state whichprovider_optionskeys it reads.Adapters should not hand-roll the shared rules:LlmCall.Dialect.Helpersis the adapter-authoring surface, and reimplementing it produces adapters the conformance suite cannot check consistently.It should not require a change to canonicalRequest,Message,Response, orToolstructs for already-representable concepts.10.2 Canonical type evolution
Canonical IR changes follow these rules:- Add new optional fields with defaults in a minor release.
- Add new content-part variants to the
Content.part()union only when existing codecs can ignore them safely or return a capability error for them. - Unknown provider content is preserved in
Content.Raw; a codec must not discard it. Rawis an escape hatch, not a permanent substitute for a canonical type. When a provider feature becomes common, promote it to a typed content/field in a documented release.- Removing or changing the meaning of an existing canonical field requires a major release.
- Adding an optional field with a default does not bump
Codec.Info.ir_version; the version changes only when existing codecs must adapt to the change. - Changing required codec callbacks, or making an optional streaming callback required, is a breaking change.
Codec.Info.ir_versionidentifies the canonical IR contract a codec was implemented against.
10.3 Codec conformance tests
The conformance suites are published fromlib/so adapter packages can call them:
They are test support, not runtime dependencies: theylib/llm_call/conformance.ex # shared assertions lib/llm_call/codec_conformance.ex # LlmCall.Codec.Conformance lib/llm_call/media/conformance.ex # LlmCall.Media.Conformanceimport ExUnit.Assertions(part of every Elixir installation) and are meant to be called from a test.What they check today:info/0metadata- streaming-callback presence matching
capabilities.streaming(anddecode_operation/2matchingLlmCall.Media.Capabilities.async) - encoded path/method/body, a map-shaped query, and credential-free headers
- response decoding at the IR boundary: canonical parts, a stop reason from the
closed union, canonical usage, and
rawas a map ornil - a malformed response as a typed
:invalid_responseerror, never a raise or a fabricated response - provider error envelopes as a canonical
%Error{}: string-or-nil code,status: nil, booleanretryable? - stream event shape and order, driven over the caller's chunks
(
assert_stream/3);:message_deltamay carryStreamDelta.Rawfor an unmapped provider field, and{:done, state}must not be followed by content - streamed tool-call assembly through
LlmCall.StreamAssembler(assert_tool_call_assembly/5) - the encoding side of streaming (
assert_stream_request/3):stream: trueandstream: falsemust encode differently, the streaming request must carry a signal —body["stream"], an SSE query, or a path that differs from the plain one — and the non-streaming request must not. A codec can implement every streaming callback and still never ask for SSE; nothing else in the suite catches that. The signal check is a disjunction, sorequire:pins the ones a protocol cannot lose (Gemini needs bothalt=sseand the streaming path). - the reserved-key rule (
assert_reserved_options/4): each of the codec's own wire keys is rejected as aprovider_optionsentry, which is the "no silent override" promise as a test. capture_rawonResponse.raw,StreamEvent.raw,LlmCall.Media.Response.raw, andOperation.raw- stream roundtrip and stream-state ownership (the request is passed in, so an out-of-tree codec is checkable)
Operationstate-machine invariants- capability declarations matching validation, in both directions, for media
and chat codecs (
assert_capability_consistency/3on both sides) - rejection of a request with a field the codec cannot express
(
assert_rejects/3) — the "no silent drops" promise as a test
10.4 Native provider options
provider_optionsis a namespaced escape hatch, not a free-for-all: each codec declares the native keys it knows and a small type contract for them. Known keys are validated (:boolean,:string,:map,:list,:number,:integer, or an enum); an invalid value is a:validationerror. Unknown keys still pass through, because a provider may add a field before the codec learns about it.This is documentation plus early diagnostics, not a catalog: the codec does not infer model support, pricing, or routing from an option. The first schemas cover OpenAI Responses (access_programs,store,previous_response_id,include,parallel_tool_calls,truncation,background,service_tier,safety_identifier,prompt_cache_key), OpenAI Chat (logprobs,top_logprobs,logit_bias,stream_options,parallel_tool_calls,service_tier,user,store,metadata,prediction,modalities,audio,web_search_options), Anthropic Messages, and Gemini GenerateContent.11. Encoded request boundary
An encoded request is an%LlmCall.EncodedRequest{}withmethod,path,query,headersandbody.path,query,headers, andbodyare protocol-level facts.bodyis a JSON-compatible term, not an encoded JSON string.- Headers may include required protocol headers.
- Headers must not include credentials; authentication is added by the client,
from the provider's
authshape and the caller'sapi_key. - The client prepends the base URL, merges the provider's
query:under the encoded query, and performs JSON encoding with the configured module. The provider-level parameters carry vendor facts that are not protocol facts — MiniMax'sGroupId, Azure OpenAI'sapi-version— and a codec's own key wins on a collision, becausealt=ssebelongs to the protocol.
MyApp.OpenAI.client(api_key: ...), then callLlmCall.complete(client, request).The boundary stays public for callers that own a transport:encode withLlmCall.encode/3, serialiseencoded.body, and issue the request withencoded.method,encoded.path,encoded.headersandencoded.query.12. Streaming
The client parses SSE (LlmCall.SSE) and yields normalized events; codecs see provider event terms only.Event type Delta Meaning :message_start,:content_start,:content_stop,:message_stopnone structural :text_delta%StreamDelta.Text{text}answer text :thinking_delta%StreamDelta.Thinking{text, signature}reasoning text, or its signature :tool_call_start%StreamDelta.ToolCall{id, name, index}opens a call :tool_call_delta%StreamDelta.ToolCall{arguments}appends an argument fragment :tool_call_stop%StreamDelta.ToolCall{}closes a call :message_delta%StreamDelta.MessageDelta{stop_reason, usage}, or%StreamDelta.Raw{data}stop reason or usage; Rawwhen a provider field is unmapped:errornone the provider reported a failure typeis a closed union of those names, and no caller should have to pattern-match provider JSON.raw(a map ornil) is populated only whenConfig.capture_raw == :all.deltais a closed union of those delta structs. It must not beterm(), and each delta type must be usable without pattern-matching raw provider JSON.rawis provider-specific but still typed as a JSON object (map()) ornil, and it is populated only whenConfig.capture_raw == :all, so normal streaming does not retain a second full event per chunk. If a provider sends a non-object sentinel, the transport or codec normalizes it to{:done, state}or wraps it in a map; it must not leak an arbitrary term into the public type.A reasoning block is not one event: the text arrives incrementally, and a provider that signs its thinking blocks — Anthropic on every block, Gemini on the thought parts before a function call — sends the signature in a separate event whosetextis"". A caller that folds onlytextbuilds a block the next request cannot carry: the fold has to appenddelta.textand keepdelta.signaturewhen it is set.12.1 Streaming tool-call assembly
The wire layer can emit tool-call fragments before the model has finished its output. A separate, pureLlmCall.StreamAssemblerturns those fragments into completeLlmCall.ToolCallvalues.LlmCall.StreamAssembleris opaque:new/1takes an optionaljson_decoder,feed/2returns{:ok, assembler, completed_calls}, andfinish/1flushes whatever is still pending.Rules:- Pending calls are keyed by
StreamEvent.index. :tool_call_startrecordsidandname.:tool_call_deltaappends theargumentsstring fragment.:tool_call_stop,:content_stop, or:message_stopfinalizes pending calls.finish/1finalizes any calls that did not receive an explicit stop event.- Tool calls with multiple entries are independent and may complete at different times.
- The assembler does not execute tools and does not send tool results.
- A completed call may be emitted before
:message_stopwhen the provider emits a per-block stop. The caller may start local side effects then, but it must send the tool result in the next model request after the current response completes.
new/1accepts an optionaljson_decoder, receiving the accumulated argument string and returning{:ok, map()} | {:error, String.t()}.json_decoderreceives the accumulated argument string and returns{:ok, map()} | {:error, String.t()}. If no decoder is provided, the assembler returns the completeToolCallwitharguments: nilandarguments_error: :no_json_decoder. If decoding fails, the field carries{:invalid_json, reason}— the decoder's own message when it returns a string, and a fixed phrase otherwise — whileraw_argumentspreserves the payload.Provider timing differs:
Early execution is an application decision.Dialect Tool-call completion signal Early execution before message end OpenAI Chat Completions finish_reason: "tool_calls"/ final chunknormally at message end OpenAI Responses response.output_item.donefor the function calloften before response end Anthropic Messages content_block_stopfor the tool blockyes, when the block stops Gemini GenerateContent complete functionCallpartyes Bedrock Converse tool-use block stop / message stop provider-dependent Cohere Chat final tool-call chunk / finish reason usually at response end LlmCallonly exposes complete calls; it does not run them or insert results into an active response.LlmCall.stream/3owns the transport side.on_event:is an observer, not a middleware: a 1-arity function called for every event before the reducer, includingfinish_stream/2output. Its return value is ignored, it cannot filter or replace events, and it stops being called after the reducer halts. An application that needs to modify the stream combines theEnumerablewithStream.map/2; raw HTTP bytes are not exposed as a hook. There is no tee: reducing a%LlmCall.Stream{}twice issues the request twice.- parse SSE framing (
LlmCall.SSE) - decode JSON payloads with
Config.json - detect provider stream terminators such as
[DONE]and callfinish_stream/2 - raise
LlmCall.HTTP.Errorwhen the body fails mid-stream
decode_stream_event/3.Codec responsibilities:- turn provider event terms into normalized events
- maintain protocol-specific stream state
- preserve unknown events in
Raw - return
{:done, state}when the protocol has a terminator
It may combine tool call deltas into completeLlmCall.StreamAssemblerToolCallvalues, but it must not execute them.13. Tool framework integration
The core only defines tool wire data. It does not call tools.tool_callLlmCalldoes not wraptool_call: the application mapsToolCall.describe/2into%LlmCall.ToolSpec{}values, and after decoding dispatches each call throughToolCall.run/4withcall.nameandcall.arguments. The library neither runs the call nor builds the follow-up request.Dependency direction:llm_call no tool_call dependency tool_call no llm_call dependencyOther frameworks
Any framework can map its own tool definition toLlmCall.ToolSpecand consumeLlmCall.ToolCall/LlmCall.ToolResult. No framework is privileged.MCP
MCP is a tool/resource transport, not an LLM completion protocol. An MCP tool catalog can be converted intoLlmCall.ToolSpecvalues outside the core.14. Dialect adapters
Phase 1
LlmCall.Dialect.OpenAI.ChatLlmCall.Dialect.OpenAI.ResponsesLlmCall.Dialect.Anthropic.MessagesLlmCall.Dialect.Gemini.GenerateContent
Vendored media adapters
These ship in this repository underLlmCall.Media.Dialect.*. They are not part of the stable API surface: a provider API change can force an adapter change in a minor release, which is why they are named separately from the reference adapters. SeeLlmCall.Mediafor the contract.LlmCall.Media.Dialect.OpenAI.ImagesLlmCall.Media.Dialect.Gemini.Predict(Imagen)LlmCall.Media.Dialect.Seedance.VideoLlmCall.Media.Dialect.MiniMax.Video
Beyond this repository
There is no roadmap of adapters the library owes, and no provider list to complete. An adapter that is not vendored here is an application module or a package, in or outside this repository, following the same behaviour and the same conformance suite.CONTRIBUTING.mdstates the bar.OpenAI-compatible APIs
Do not create a separate protocol for every OpenAI-compatible provider.Use the OpenAI Chat codec with a provider/endpoint configuration:- base URL
- auth headers
- model id
- capability flags
- provider-specific path/query
Provider/protocol routing
A provider is not a protocol:- Azure OpenAI uses OpenAI Chat/Responses shapes with deployment and API-version configuration.
- Vertex AI can route Gemini-shaped and Anthropic-shaped models.
- Bedrock can expose Converse, Anthropic Messages, and OpenAI-compatible surfaces depending on the model/endpoint.
15. Module layout
lib/llm_call/holds the core: canonical structs, the two codec behaviours, the client, the Finch transport, the SSE parser and the pure stream assembler.dialect/*holds the reference adapters — the protocol families the IR is written against — andmedia/dialect/*the vendored media adapters, which ship with the library but are not part of the stable surface.mix docslists every module; the file tree is not repeated here, and nothing in the core depends on a dialect module: an adapter is selected by name at call time.Optional packages live outside this repository:llm_call_costfor pricing andllm_call_langchainfor a LangChain adapter.LlmCall.HTTPis Finch andLlmCall.WS.Mintis the WebSocket implementation; there is no HTTP behaviour and no tool behaviour (tool_callis a companion the application adds itself).Test support and the compiled examples are not part of the design:test/support/holds the fixture payloads, the two test servers, the JSON doubles and the codec fakes, andexamples/holds provider modules compiled in:devand:testso they cannot rot.16. Public API
Two tiers. The one-call client is the default:complete/3,stream/3andgenerate/3take a client, the request and the call options (codec:,headers:, pluson_event:onstream/3);poll/3advances a media operation;execute/2andfetch/2run a request the caller already encoded.A caller that owns a transport keeps the encode/decode boundary:encode/3,decode_response/3,init_stream/3,decode_stream_event/4,finish_stream/3,validate_request/3,capabilities/3, andStreamAssembler.new/1,feed/2,finish/1.headers:is a list of string{name, value}pairs for this call only. The client merges them in this order: client/provider headers, then call headers, then the codec's encoded headers. Later entries win, so a call can override a provider default (a gateway routing header, a beta flag), while a protocol header emitted by the codec still wins. An invalid list is a:validationerror before any request. Any option outside the ones a call lists is a:validationerror too, so a misspelledcodec:cannot silently fall back to the default. WebSocket handshake headers are connection-level: passheaders:toConnection.start_link/1; they are merged with the client headers and reused on reconnect.encodereturns{:ok, %LlmCall.EncodedRequest{}}; the media operations return their documented success/error tuples, andcapabilitiesandmedia_capabilitiesreturn{:ok, struct} | {:error, %LlmCall.Error{}}.JSON resolution has a compile-time default and runtime overrides: the default is read withApplication.compile_env(:llm_call, :json_library, ...)and is ElixirJSON, so changing it recompiles the library, while a provider declaration, a clientjson_library:option or an explicit%LlmCall.Config{}overrides it.LlmCall.JSON.encode/2anddecode/2accept a module with taggedencode/1/decode/1(Torque, Jason, Poison) or bangencode!/1/decode!/1(ElixirJSON) and normalize to tagged tuples.capture_rawhas the same shape —Application.compile_env(:llm_call, :capture_raw, :none), overridden by%LlmCall.Config{}— and there is no provider or call-level capture option.Dispatch never raises for a module that implements the other behaviour: each entry checks a behaviour-unique callback (encode_request/2for completion,encode_submit/2for media) and returns a:validationerror. The exception isdecode_error/3, whose callback exists in both behaviours.17. Testing
The core suite must not need a live API. Every adapter is pinned with fixtures it supplies, and the suite proves four things beyond the happy path:- Request fixtures — a canonical request to an exact body, path, query and headers; tool definitions and tool choice; system placement; multi-modal content.
- Response and stream fixtures — text, tool calls, multiple blocks,
usage/stop-reason mapping and provider errors; for streams, fragmented arguments,
multiple calls by
index, early completion, the terminator, unknown-event preservation andcapture_rawat:none/:response/:all. - Capabilities and contracts — an unsupported field is rejected rather than
dropped, a declared capability accepts its representative, cost is never
fabricated, and handing a streaming call to a non-streaming codec or a
StreamStateto the wrong codec is a typed error. - Audits — three mutation suites keep silent drops out:
- encode: every canonical-derived wire key is rejected from every source that can write it (§8.1);
- decode: deleting a provider field either changes the canonical struct or is
listed as an ignored envelope/echo field, and never raises; an absent field is
nil/:unknown, never a fabricated value; - stream deltas: the same audit over chunks, so every
StreamDeltafield (text,thinking/signature, tool-callid/name/arguments/index,stop_reason/usage) and everyStreamEvent.indexhas a provider source, and an unmapped field becomes aRawevent.
test/llm_call/(write_path_audit_test.exs,decode_path_audit_test.exs,stream_delta_audit_test.exs), and the published conformance assertions run over every adapter this package ships.18. Failure and compatibility policy
- Provider-specific features are opt-in through explicit fields or
provider_options. - Unsupported canonical fields produce
LlmCall.Error{kind: :validation}or a capability error. - Unknown provider blocks are retained as
Content.Raw; full response raw is retained only whencapture_rawrequests it. - The canonical IR is versioned; breaking changes require a major release.
rawdata is never part of the stable cross-provider contract; it is a best-effort escape hatch.- New provider capabilities should be added through provider options before becoming canonical fields.
19. Anti-goals
The project moved from a transport-free codec library to a call-oriented SDK. Three previous anti-goals were reversed deliberately; the rest stand.No longer anti-goals:- A
Finchdependency. It is a required runtime dependency, because the one-call path must work aftermix deps.get. - A fixed HTTP client.
LlmCall.HTTPis Finch; it is not a seam. JSON remains replaceable throughLlmCall.JSON. - A tool runner. Still an anti-goal, and now there is no tool adapter either:
schema derivation and dispatch belong to
tool_callor an application module, and building the follow-up request stays with the caller.
- API-key lookup (the key is always passed in)
generate_text-style prompt helpers beyondcomplete/3- a model catalog or provider registry (providers are user modules, passed explicitly)
- an agent loop
- a prompt template or memory system
- automatic retries, backoff, or rate limiting (a timeout is configuration)
- provider-specific branches in user application code beyond the provider module
- a canonical model of individual hosted tools
20. Decisions
Settled:- Core transport: the library ships one HTTP client on Finch; there is no HTTP behaviour. Was "stay transport-free", then "replaceable transport"; now fixed. JSON remains the replaceable edge.
- JSON encoding: the client encodes and decodes with the configured module; codecs use the same module for tool-call arguments. Was "transport encodes".
- Tool data in core: neutral
ToolSpec,ToolCall,ToolResult; no schema derivation, no dispatch, no dependency ontool_call. An application mapsToolCall.describe/2intoToolSpecitself. - First protocol set: OpenAI Chat, Anthropic Messages, Gemini GenerateContent.
- OpenAI Responses: a core reference adapter over HTTP/SSE and the Responses WebSocket connection.
- Stream assembler: pure helper in core, protocol-neutral.
- SSE parser: in core, because the default client streams.
- Codec discovery: explicit codec module selection, named on a provider module; no core model catalog or registry.
21. Implementation order
The order the release followed, and what it taught, are recorded in roadmap.md.22. Design success criteria
This design is working if:- A caller can use
LlmCallwithout any agent framework. - A third party can write an adapter, decode against their own fixtures, and verify its shapes, capture policy, stream plumbing, capabilities, and operation states with the conformance suite — without touching core.
- An adapter outside this repository needs no core release; a vendored one is allowed to change with its provider without changing the IR.
- A caller reaches a provider with one function, without assembling a request or parsing SSE.
- The JSON layer can be replaced without touching a codec, per application, per provider, or per client; HTTP is Finch.
- A tool framework can be used directly; the library models tool wire data only and depends on no tool library.
- Adding a dialect adapter does not require changing canonical messages.
- Provider-specific features remain accessible through explicit escape hatches.
- The core has no auth lookup, model catalog, provider registry, or agent loop.
- Dialect fixtures are readable and testable without network access.
- A caller can generate images and run video generation jobs without writing provider-specific HTTP, and without the core polling or downloading anything.
23. Media generation (image and video)
The media IR is first-class and stable; media adapters are vendored underLlmCall.Media.Dialect.*and are not part of the stable surface. It uses two entry points over one content vocabulary:- a completion that may return generated media (
Request.modalities, decoded intoResponse.content), and - a dedicated media request (
LlmCall.Media.Request->LlmCall.Media.Response), which may be synchronous or a long-running operation.
messages,system, ortools; flattening a completion request into{prompt, n, size}would be a hidden convention.23.1 Canonical additions
%LlmCall.Content.Video{}mirrorsContent.Imagewith asourceand amedia_type.%LlmCall.Media.Request{}carriesmodality(:imageor:video),model,prompt,references(nil, or image and video parts),paramsandprovider_options. Everyparamskey is optional and nil-by-default:n,size,aspect_ratio,duration_s,seed,negative_prompt.%LlmCall.Media.Response{}carriesassets,revised_prompt,usage,operationandraw.%LlmCall.Operation{}carriesid,status(:queued,:in_progress,:completed,:failed),progress,result,next,next_kind(:pollor:asset),next_media_type,errorandraw.%LlmCall.Media.Capabilities{}carriesmodalities,async,max_n(a positive integer or:unsupported),sizes,aspect_ratios,durations_s,max_references(a non-negative integer ornil),seedandnegative_prompt.LlmCall.Media.Requesthas nostream, nosystem, and notoolsfield.promptis the provider's prompt field, not a message list.referencesare images or videos the generation is conditioned on. Only JSON-carriable content is allowed:%Content.Image{}and%Content.Video{}with{:url, _},{:base64, _}, or string sources. Anything else is a validation error. A codec that would need a multipart upload rejectsreferences.Content.Audiodoes not exist. Audio is not in scope; adding a type no codec can fill would be dead weight (§10.2).23.2 Completion-side media output
Request.modalities(default[:text]) is what the caller will accept:an empty list, duplicates, or a value outside
:text | :image | :videois a validation error;- Gemini GenerateContent maps it to
generationConfig.responseModalities(upcased), omitting the field for the default[:text]; - Gemini GenerateContent with
:video, and OpenAI Chat or Anthropic Messages with:imageor:video, are validation errors; stream: truewith:imageor:videoinmodalitiesis a validation error. Progressive media streaming would need a new delta type and touches the assembler and every codec; it is out of scope. A caller that wants images calls without streaming.Capabilities.modalitiesdeclares the supported set per codec (default[:text]), so a caller can check before encoding.- Decoding does not change: generated media arrive as
%Content.Image{}or%Content.Video{}parts inResponse.content.
23.3 Media validation and capabilities
validate_request/2is aLlmCall.Media.Codeccallback, implemented per codec. It must consultcapabilities/2so that a knob outside the declared surface is a:validationerror before anything is sent.List-valued capability fields enumerate accepted values:sizes: []means the codec accepts nosizevalue;max_n: :unsupportedmeans the codec accepts nonvalue (the provider takes one asset per request), while an integermax_naccepts1..max_n;max_references: nilmeans the codec accepts no reference; an integer is the number it takes (2for a first/last-frame pair,1for a first frame);- a
nilrequest knob means "provider default" and is always accepted.
paramskeys outside the canonical set in §23.1 are validation errors, ignoring keys whose value isnil(anilknob is "provider default", not a request). A codec rejects a non-nil knob it cannot express.provider_optionsis namespaced and merged like the completion side; the namespace is declared when the adapterusesLlmCall.Media.Dialect.Adapter(namespaces: [:openai]), so an unknown key is a validation error. A key that would replacemodel,prompt,instances, ormodality— or a canonical param such assize,n, orduration— is a validation error too, which is what keeps aConfigdefault from beating an explicitparamsentry. A protocol's own nested parameter object stays reachable:parametersis not reserved, becauseLlmCall.Config.deep_merge/2recurses into it, so a native Imagen knob such aspersonGenerationmerges beside the canonicalsampleCount(but the canonical keys inside it are reserved).23.4 Operation state machine
One rule governs every operation:nextis always the encoded request the client (or your transport) executes next, built by the codec. There is no "encode a poll request from an id" callback.decode_operation/2receives only the payload and the config. A codec is stateless, so every continuation value — a file id, a cursor, a page token — must travel insidenext(path, query, or headers). A protocol whose cursor cannot be expressed there is out of scope.next_kind: :poll-> decode the response body and calldecode_operation/3.next_kind: :asset-> the response body is media bytes. The core never touches them; the transport gives them to the caller withnext_media_type.status: :queued | :in_progress->result: nil,error: nil,nextset,next_kind: :poll.status: :completedwithnext_kind: :asset->result: %LlmCall.Media.Response{assets: []}(the asset is reachable only throughnext).status: :completedwithnext: nil->result: %LlmCall.Media.Response{assets: [...]}.status: :completedwithnext_kind: :pollis not produced: a codec that knows the asset needs another request says so in the same decode result.status: :failed->errorset,result: nil,next: nil.decode_submit/2may return%LlmCall.Media.Response{assets: [...]}withoperation: nilwhen the submit response already carries the finished asset.
idmay benilon a follow-up step: a codec is stateless, so a poll whose payload does not repeat the job id (MiniMax's file-retrieve step) cannot supply it.The library never polls, sleeps, retries, or downloads beyond the request the caller asks for. The client loop is:generate/3until the operation is:in_progress,poll/3until it is:completedwithnext_kind: :asset, thenfetch/2for the bytes.poll/3executesop.nextand decodes the body withdecode_operation/2; an:assetoperation is rejected with a pointer atfetch/2.fetch/2never follows a redirect: a3xxis a:transporterror with the status, and since%LlmCall.Error{}carries no headers theLocationis lost. A caller whose asset URL redirects (CDN, pre-signed link) builds an%LlmCall.HTTP.Request{}for that URL and callsLlmCall.HTTP.request/2, which returns the response with headers. A caller that owns a transport callsLlmCall.encode_media/3andLlmCall.decode_operation/3directly instead.Operation.rawfollowscapture_raw(:responseand:all).23.5
LlmCall.Media.Codecbehaviour and public APICallback Contract info/0%LlmCall.Codec.Info{}capabilities(request, config)%LlmCall.Media.Capabilities{}validate_request(request, config):okor a:validationerrorencode_submit(request, config){:ok, %EncodedRequest{}}or an errordecode_submit(term, config){:ok, %LlmCall.Media.Response{}}or an errordecode_operation(term, config){:ok, %LlmCall.Operation{}}or an error, optionaldecode_error(term, config)%LlmCall.Error{}, never an error tupledecode_operation/2is the only optional callback.Entry point Purpose encode_media/3validate and encode a submit decode_media/3decode a submit response decode_operation/3decode a poll or asset payload validate_media/3run the codec's validation only media_capabilities/3the codec's declared surface encode_media/3delegates to the codec, which validates first.decode_operation/3returns a typed:validationerror when the codec has nodecode_operation/2, exactly like the optional streaming callbacks.Media codecs reuseLlmCall.Codec.Info.ir_versionstays1: adding optional canonical fields with defaults does not change the contract an existing codec was written against.23.6 Dialect adapters
The media adapters are vendored with the library underLlmCall.Media.Dialect.*: they ship in thellm_callapplication, follow the same conformance suite as the reference chat adapters, and are not part of the stable API surface (seeLlmCall.Media). The media IR they map into is stable; they are not.Phase M1, synchronous image:LlmCall.Media.Dialect.OpenAI.Images—POST /v1/images/generations, body{model, prompt, n?, size?}.response_format: "b64_json"is sent only fordall-e-*models:gpt-image-1rejects the key and returns base64 anyway, andprovider_options.openaimay override the value.references,aspect_ratio,duration_s,negative_prompt, andseedare validation errors. Responsedata[]:b64_json->{:base64, data},url->{:url, url}.revised_promptis the first non-nil value; the rest stay inraw.LlmCall.Media.Dialect.Gemini.Predict(Imagen) —POST /v1beta/models/{model}:predict, body{instances: [{prompt}], parameters: {sampleCount?, aspectRatio?, negativePrompt?}}.size,duration_s,seed, andreferencesare validation errors. Nativeparametersknobs (personGeneration,safetySetting,outputOptions, ...) merge throughprovider_options. Responsepredictions[].bytesBase64EncodedplusmimeType-> base64 image; a prediction withraiFilteredReasonand no bytes is skipped while another prediction has bytes, and is a:providererror when none do.
LlmCall.Media.Dialect.Seedance.Video(BytePlus ModelArk) — submitPOST /api/v3/contents/generations/taskswith{model, content: [{type: "text", text: prompt}, {type: "image_url", image_url: {url}, role}?], ratio?, duration?, seed?};nextisGET /api/v3/contents/generations/tasks/{id}withnext_kind: :poll. Statuses map asqueued->:queued,running->:in_progress,succeeded->:completedwith%Content.Video{source: {:url, content.video_url}}, andfailed | cancelled | expired->:failed.size,negative_prompt, andnare validation errors;resolution,watermark,generate_audio,camera_fixed, andservice_tierare native knobs reached throughprovider_options.referencesaccepts one or two images: the first isrole: "first_frame"and the secondrole: "last_frame", because ModelArk requires the roles to tell the pair apart.durations_sdeclares the union across Seedance models (2-30 s); a model-specific limit is a provider error.LlmCall.Media.Dialect.MiniMax.Video(Hailuo,v1) — submitPOST /v1/video_generationwith{model, prompt, duration?, first_frame_image?}->{task_id};nextisGET /v1/query/video_generation?task_id={id}. OnSuccessthe query returns only afile_id, so the operation stays:in_progresswithnext=GET /v1/files/retrieve?file_id={id}, and that payload completes the job with%Content.Video{source: {:url, file.download_url}}.Fail->:failed. StatusesPreparing | Queueing | Processing->:in_progress.size,aspect_ratio,seed, andnegative_promptare validation errors;resolution,prompt_optimizer,fast_pretreatment,callback_url, andextraare reached throughprovider_options.referencesaccepts one image (first frame).- Both codecs treat a provider error envelope as a
:providererror. MiniMax reports errors as HTTP 200 with a non-zerobase_resp.status_code, so a non-zerobase_respis an error regardless of the transport status.
23.7 Media non-goals
- audio input or output;
- media streaming (partial or progressive assets), including
stream: truewith an image/video modality; - multipart uploads (image edits/variations from a local file);
- downloading media bytes, polling loops, retries, timeouts, or job persistence;
- cost and model catalogs for media models.
24. WebSocket transport
WebSocket is a transport, not a completion protocol.LlmCall.WSis a behaviour with five operations:Callback Contract connect(url, headers, opts){:ok, conn}or{:error, reason}feed(conn, socket_message){:ok, frames, conn}or an error|
send_frame(conn, {:text | :binary, binary})|{:ok, conn}or an error | |recv(conn, timeout)|{:ok, frame, conn}or an error | |close(conn, code, reason)|:ok|LlmCall.WS.Mintimplements it withMint.WebSocket: it performs the upgrade, owns the socket in the calling process, answers pings insiderecv/2andfeed/2, and returns a peer close as{:close, code, reason}.recv/2is the blocking form;feed/2lets a process that already owns the socket (a connection process) decode the messages Mint delivers in active mode. Connections are HTTP/1.1; Mint's HTTP/2 extended CONNECT is not used.Mint's HTTP proxy options are forwarded (ws_opts: [proxy: ..., proxy_headers: ...]), sows://through a forward proxy andwss://through aCONNECTtunnel are transport concerns of the proxy, not of the OpenAI endpoint. SOCKS proxies are not supported by Mint.24.1 Responses connection
LlmCall.Dialect.OpenAI.Responses.Connectionis aGenServerthat owns one WebSocket connection and multiplexesresponse.createcalls. It is a transport connection, not conversation state:previous_response_idandstorestay with the caller.Connection.start_link(client: client)opens the socket,create/3starts a lane and returns the stream to reduce.- Each create names a
stream_idlane (generated when omitted). Events are decoded per lane by the Responses codec and delivered asStreamEvents to the%Connection.Stream{}returned bycreate/3. - Events are routed by the
stream_idthe server echoes, which it does for named lanes. A server that does not echo it can only be attributed while one lane is in flight;lane_for/2degrades to that single lane and, once an event has proven the echo is missing, a second concurrentcreate/3is a:validationerror instead of a response that would later be unroutable. Back-to-back creates are allowed before the first event, because that is how the documented endpoint runs parallel conversations and the echo is unknown until then. An event that arrives with no route at all disconnects the connection with{:unroutable_event, :stream_id_not_echoed}rather than delivering it to the wrong lane. steer/3sendsresponse.steerwith the lane'sprevious_response_idand the caller'sinput; a lane that has not receivedresponse.createdis a:validationerror.cancel/2sendsresponse.cancelwithresponse_idand ends the lane. Neither event carriesstream_id: the provider routes them by response id.storeandprevious_response_idstay in the request'sprovider_options; the connection does not model server-side conversation state.response_id/2returns the provider id fromresponse.created, so a caller can use it asprevious_response_id. Finished lanes keep the last 32 ids.- The connection reconnects before the 60-minute limit (
:max_age_ms, default 55 minutes) and on an unexpected close, with exponential backoff (:retry_delay_ms5s,:max_retry_delay_ms60s) and a bounded budget (:max_reconnect_attempts, default 5;:infinityfor unbounded retries). A failed reconnect does not stop the connection process: it becomes:disconnected,create/3returns a:transporterror, andreconnect/1retries on demand. - An in-flight response cannot be resumed: its subscriber receives a
:transporterror. Continuing a conversation is the caller's decision. - The URL is built from the client's
base_urlplus the Responses path, withhttp/httpsmapped tows/wss; Azure's/openaibase path is preserved. The provider'squery:parameters apply here too, merged over any query already present inbase_url, so a vendor that keys on a URL parameter (GroupId,api-version) still gets it on the handshake.
25. Boundary with catalog, routing and evaluation layers
A full LLM client such as ReqLLM grows a catalog, provider registry, execution metadata, evaluation, rerank, embeddings, pricing and gateway contracts. Those are real needs, but they are a different layer.LlmCalldeliberately stops at the wire boundary; this section records the split so the core does not drift.25.1 What belongs here
- Canonical wire IR: request, messages, content, tool data, response, stream events, usage tokens, errors.
- Dialect codecs and their capability declarations (protocol-level, not model-level).
- One-call HTTP/SSE and Responses WebSocket transports, with reconnect for the WebSocket connection.
- Provider declaration modules (
use LlmCall.Provider): origin, auth shape, codec names, and transport defaults. provider_optionsas a namespaced, documented escape hatch for native fields.
25.2 What belongs above
- Model catalog, model IDs, limits, modalities per model, pricing and cost estimates.
- Routing by execution metadata, gateway contracts, fallback between providers, retries/backoff policy and rate limiting.
- Evaluation, rerank and embedding orchestration: each is a separate protocol
family with its own endpoint and response shape, not a field on
Request. - Subscription OAuth, Claude Code version management and other product-specific auth flows; the key/header is supplied by the caller.
- Telemetry, tracing, budgets and approval policies.
25.3 Rules that follow
- A new operation is a new IR + codec.
Mediais the precedent: separate request, response, capabilities and codec behaviour. Evaluation, rerank and embeddings would follow the same shape instead of growingRequest. - Provider-specific does not mean catalog. A native field such as an access
program or a thought-summary toggle stays in
provider_options; it does not become a model registry. - Capabilities are protocol-level.
Capabilitiesanswers "can this wire format express it", not "does this model support it". Model limits surface as provider errors; a caller that needs model selection consults its own catalog. - Transport metadata stays at the boundary. Response headers,
Retry-Afterand provider-reported cost are wire data.Error.retry_after_msexposes the retry hint; the core never computes a catalog estimate or applies a retry policy. Provider-reported cost stays inraw(usecapture_raw: :response) or in the upper layer; the canonicalUsageremains token-only. - Dynamic headers are call options.
complete/3,stream/3,generate/3andpoll/3acceptheaders:; the precedence is client headers, then call headers, then codec headers. A call header can override a provider default (gateway routing, a Claude Code version, a beta flag); a protocol header emitted by the codec still wins. Authentication remains caller-supplied; a call header is an explicit override, not a policy. - Routing uses public metadata. An external router can read
Codec.Info,CapabilitiesandProvider.__provider__/0; it does not need a registry in the core. - No silent precedence. When native options interact (for example a structured-output request that disables thought summaries), the codec must either validate the conflict or document the winning rule; it must not drop the loser silently.