Prometheus metrics reference

BYOC exposes Prometheus metrics for monitoring the agent and its workspaces. All metrics are gauges. Use these metrics to build dashboards and alerts in your monitoring system, such as Datadog through its Prometheus/OpenMetrics integration.

The example queries use PromQL syntax; the same metric names and label filters work when you query from other monitoring tools, such as Grafana or Datadog.

Replace <clusterId> with your cluster ID and <vcid> with the workspace (vcluster) ID. Numeric state values map to the states listed in each metric's description. Use these mappings for value-to-text display and alert conditions.

Metric source

The agent emits all metrics in this reference from the pyxis-api-gateway pods, which belong to the pyxis-api-gateway Deployment. The pods expose the metrics on port 9888.

Configure your monitoring tool to scrape port 9888 on the pyxis-api-gateway pods.

Agent metrics

Agent-level metrics need only the clusterId label.

pyxis_control_plane_info returns one series per control-plane component. The name label identifies the component (root, global, cluster, flink, flink-byoc, or a workspace ID), the type label identifies its category (root, product, flink-vc), and the state label carries the state as text. Filter by name to monitor each core control plane individually.

PromQL query:

1max(pyxis_cluster_health_state{clusterId="<clusterId>"})

PromQL query:

1max(pyxis_control_plane_info{clusterId="<clusterId>", name="root"})

PromQL query:

1max(pyxis_control_plane_info{clusterId="<clusterId>", name="global"})

PromQL query:

1max(pyxis_control_plane_info{clusterId="<clusterId>", name="cluster"})

PromQL query:

1max(pyxis_control_plane_info{clusterId="<clusterId>", name="flink"})

PromQL query:

1max(pyxis_control_plane_info{clusterId="<clusterId>", name="flink-byoc"})

Workspace metrics

Workspace-level metrics need both the clusterId and vclusterId labels. The max() aggregation collapses duplicate series, for example from multiple scrape sources, into a single value per workspace.

PromQL query:

1max(pyxis_vcluster_state{clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1max(pyxis_vcluster_allocated_resource{resourceType="cpu", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1max(pyxis_vcluster_allocatable_resource{resourceType="cpu", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1max(pyxis_vcluster_allocated_resource{resourceType="memory", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1max(pyxis_vcluster_allocatable_resource{resourceType="memory", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1(max(pyxis_vcluster_allocated_resource{resourceType="cpu", clusterId="<clusterId>", vclusterId="<vcid>"}) * 100)
2/ max(pyxis_vcluster_allocatable_resource{resourceType="cpu", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1(max(pyxis_vcluster_allocated_resource{resourceType="memory", clusterId="<clusterId>", vclusterId="<vcid>"}) * 100)
2/ max(pyxis_vcluster_allocatable_resource{resourceType="memory", clusterId="<clusterId>", vclusterId="<vcid>"})

PromQL query:

1max(pyxis_vcluster_component_state{clusterId="<clusterId>", vclusterId="<vcid>", componentKey="Namespace"})

PromQL query:

1max(pyxis_vcluster_component_state{clusterId="<clusterId>", vclusterId="<vcid>", componentKey="RAMSecurityToken"})

PromQL query:

1max(pyxis_vcluster_component_state{clusterId="<clusterId>", vclusterId="<vcid>", componentKey="Monitor:vvp-sts-credential-secret"})

PromQL query:

1max(pyxis_vcluster_component_state{clusterId="<clusterId>", vclusterId="<vcid>", componentKey="ResourceQuota"})

Notes

Visualize state metrics (pyxis_vcluster_state, pyxis_vcluster_component_state, pyxis_control_plane_info, pyxis_cluster_health_state) as a latest-value stat panel with value-to-text mapping, and as a time series to observe transitions.

CPU values are in cores. Memory values are in bytes.

Suggested baseline alerts: cluster health = 0 (Failed), any control-plane component = 0 (NotReady), workspace state = 0 (NotReady), any workspace component = 0 (NotReady), CPU or memory allocation above 80% of capacity.