Get the latency graph

GET/v1/analytics/latency

Get the latency graph.

Returns latency over time for the organization's requests: a summary of the whole window and one data point per time bucket, each with the total latency, the time spent waiting on the provider, the gateway's own overhead and, for streams, the time to the first chunk. Empty buckets are included with 0 requests. The window defaults to the 24 hours before end_time (default now) and can be at most 366 days, with at most 1500 buckets. Requests from every region that holds the organization's logs are included; a region that did not answer is listed in unavailable_regions.

Every list parameter takes comma-separated values (or the parameter repeated) and matches any of them.

With workspace_id, requires access to each workspace named: an organization owner or admin, a member of the workspace, an organization-scoped va_ key with the admin role, or a va_ key scoped to the workspace. Without it, covers every workspace (less exclude_workspace_ids) and requires an organization owner or admin, or an organization-scoped va_ key with the admin role; a workspace-scoped key reads its own workspace. Errors: 400 for a missing organization, a window that is empty, too long or has too many buckets, an unknown granularity, time zone or status, a malformed provider_model, or exclude_workspace_ids together with workspace_id.

Authorization

AuthorizationBearer va_
A platform API key (va_...), created under Settings > API keys. Scoped to an organization or one workspace, with an admin or member role. Used by the management API.

Query parameters

organizationstring
the organization's slug; defaults to the va_ key's organization
workspace_idstring<uuid>[]
only these workspaces' requests (UUIDs)
exclude_workspace_idsstring<uuid>[]
without workspace_id, leave out these workspaces' requests (UUIDs)
start_timestring
start of the window, inclusive (ISO 8601 date-time with offset); default 24 hours before end_time
end_timestring
end of the window, exclusive (ISO 8601 date-time with offset); default now
granularitystring
the bucket size: minute, hour, day, week, or auto (default: by minute up to 3 hours, by hour up to 7 days, by day up to 90 days, then by week)
tzstring
the IANA time zone buckets are aligned to, e.g. America/Toronto; default UTC
providerstring[]
only requests to these providers, e.g. openai or OPENAI
modelstring[]
only requests for these model ids
provider_modelstring[]
only these provider and model pairs, written provider__model, e.g. openai__gpt-4o
virtual_key_slugstring[]
only requests made with these virtual keys (vk-...)
config_slugstring[]
only requests routed through these routing configs
prompt_slugstring[]
only requests made from these prompts
guardrail_slugstring[]
only requests any of these guardrails applied to (gr-...)
guardrail_statusstring[]
only requests whose combined guardrail result is one of these: PASS, MONITOR, REDACT, BLOCK or BYPASS
user_idstring[]
only requests attributed to these end users (the request's user field)
request_idstring[]
only the requests with these request ids
cache_statusstring[]
only requests with one of these cache results: HIT, SEMANTIC_HIT, MISS, BYPASS, DISABLED or REFRESH
operation_typestring[]
only these kinds of operation, as on the log search (chat requests have none)
regionstring[]
only requests served in these regions, e.g. us-east-2
status_codeinteger<int32>[]
only requests that ended with one of these HTTP status codes
statusstring
success (status code below 400) or error (400 and above)
streamingboolean
true for streamed requests only, false for the others
tool_scan_statusstring
only requests whose tool-result scan had this result: PASS, MONITOR, REDACT or BLOCK; FLAGGED matches any of the last three
searchstring
text to find anywhere in the stored request or response body, case-insensitive; only matches logs whose bodies were stored
min_tokensinteger<int32>
only requests with at least this many total tokens
max_tokensinteger<int32>
only requests with at most this many total tokens
min_prompt_tokensinteger<int32>
only requests with at least this many prompt (input) tokens
max_prompt_tokensinteger<int32>
only requests with at most this many prompt (input) tokens
min_completion_tokensinteger<int32>
only requests with at least this many completion (output) tokens
max_completion_tokensinteger<int32>
only requests with at most this many completion (output) tokens
min_costnumber<double>
only requests that cost at least this much, in USD
max_costnumber<double>
only requests that cost at most this much, in USD
min_duration_msinteger<int64>
only requests that took at least this long, in milliseconds
max_duration_msinteger<int64>
only requests that took at most this long, in milliseconds

Responses

200application/json

objectstring
always analytics-graph
organizationstring
the organization's slug
workspace_idsstring<uuid>[]
The workspaces covered (UUIDs); null when every workspace of the organization is covered.
start_timestring<date-time>
The start of the window, inclusive (ISO 8601 date-time).
end_timestring<date-time>
The end of the window, exclusive (ISO 8601 date-time).
granularitystring
the bucket size used: minute, hour, day or week
tzstring
the IANA time zone the buckets are aligned to
summaryobject
Latency for one time bucket, or for the whole window. The top-level figures are the total latency, from the request's arrival at the gateway until the response was complete; the nested figures split it. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
Child attributes
- timestampstring<date-time>
  the start of the bucket (ISO 8601 date-time); null on the summary
- requestsinteger<int64>
  how many matching requests the bucket holds; when 0, the figures are null
- avginteger<int64>
  the average total latency, in milliseconds
- p50integer<int64>
  the median total latency, in milliseconds
- p90integer<int64>
  the 90th percentile total latency, in milliseconds
- p95integer<int64>
  the 95th percentile total latency, in milliseconds
- p99integer<int64>
  the 99th percentile total latency, in milliseconds
- maxinteger<int64>
  the slowest total latency, in milliseconds
- providerobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- gateway_overheadobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- first_tokenobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- provider_first_tokenobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
data_pointsobject[]
One entry per time bucket, oldest first, empty buckets included.
Child attributes
- timestampstring<date-time>
  the start of the bucket (ISO 8601 date-time); null on the summary
- requestsinteger<int64>
  how many matching requests the bucket holds; when 0, the figures are null
- avginteger<int64>
  the average total latency, in milliseconds
- p50integer<int64>
  the median total latency, in milliseconds
- p90integer<int64>
  the 90th percentile total latency, in milliseconds
- p95integer<int64>
  the 95th percentile total latency, in milliseconds
- p99integer<int64>
  the 99th percentile total latency, in milliseconds
- maxinteger<int64>
  the slowest total latency, in milliseconds
- providerobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- gateway_overheadobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- first_tokenobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
- provider_first_tokenobject
  Latency figures for a set of requests, in milliseconds. Percentiles are within about 3% of the exact value (exact below 18 ms); the average and maximum are exact.
  Child attributes
  - requestsinteger<int64>
    how many requests the figures cover; when 0, the other figures are null
  - avginteger<int64>
    the average, in milliseconds
  - p50integer<int64>
    the median, in milliseconds
  - p90integer<int64>
    the 90th percentile, in milliseconds
  - p95integer<int64>
    the 95th percentile, in milliseconds
  - p99integer<int64>
    the 99th percentile, in milliseconds
  - maxinteger<int64>
    the slowest, in milliseconds
regionsstring[]
the regions whose logs were read
unavailable_regionsstring[]
Regions that hold the organization's logs but did not answer, so the figures leave them out.

VectorAxis is built by GK Labs, a member of NVIDIA Inception.