Configuring Prometheus Monitoring and Grafana

Applies to: fluss-bundle chart, fluss-grafana chart, Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

This manual configures monitoring for an Apache Fluss™ cluster deployed with the fluss-bundle chart: it installs the Prometheus reporter plugin on the Fluss servers, enables the reporter, exposes its metrics to Prometheus, and installs the fluss-grafana chart's dashboards. A Prometheus instance and Grafana are assumed to be running in the cluster already.

Prerequisites

• A Fluss cluster deployed with the fluss-bundle chart. See Deploying Fluss on Kubernetes.
• A Prometheus instance in the cluster that discovers targets either by Service annotations or through the Prometheus Operator and its ServiceMonitor resources. Pick whichever matches your existing setup.
• Grafana running in the cluster, with the Prometheus instance that scrapes Fluss registered as a datasource.

Installing the Prometheus Reporter Plugin

The Prometheus reporter is a server plugin: the metrics-prometheus setup, installed by the setup installer in an init container. Install it on both the coordinator and the tablet servers, since the reporter runs in every Fluss server. The setup installs into the prometheus plugin directory, so the server container mounts the plugin volume at /opt/fluss/plugins/prometheus.

The values for this init container are the single-setup example in Installing Fluss Setups, which installs metrics-prometheus on both server roles. To install other setups in the same init container, see Multiple setups in one init container.

Enabling the Metrics Reporter in Apache Fluss

Enable the reporter under the chart's fluss.metrics block, never under fluss.configurationOverrides: the chart refuses metrics.reporters and metrics.reporter.prometheus.port there and fails the install with a validation error.

1fluss:
2  metrics:
3    reporters: prometheus
4    prometheus:
5      port: 9249

With the reporter enabled, the chart:

• Renders the reporter settings into /opt/fluss/conf/server.yaml on every server pod, as metrics.reporters: prometheus and metrics.reporter.prometheus.port set to the configured port, 9249 by default.
• Creates one metrics Service per server role, named from the Helm release: <RELEASE>-coordinator-server-metrics-hs and <RELEASE>-tablet-server-metrics-hs, each exposing the reporter port under the port name metrics.

With fluss.metrics.reporters empty, the chart creates no metrics Service.

On a running cluster, the reporter settings apply once the server pods restart: an upgrade that changes only these values restarts none, so restart them as Metrics endpoint refuses connections shows.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Exposing the Metrics Endpoint to Prometheus

Prometheus discovers the metrics Services in one of two ways: through scrape annotations on the Services, or through a ServiceMonitor the Prometheus Operator reads. Pick the one your Prometheus uses. The reporter serves its metrics at /.

Annotation-based scraping

Add the scrape annotations to the metrics Services through fluss.metrics.prometheus.service.annotations:

1fluss:
2  metrics:
3    reporters: prometheus
4    prometheus:
5      port: 9249
6      service:
7        annotations:
8          prometheus.io/scrape: "true"
9          prometheus.io/path: "/"
10          prometheus.io/port: "9249"

Your Prometheus configuration needs a scrape job that keeps the Kubernetes Services annotated prometheus.io/scrape: "true" and reads the scrape path and port from prometheus.io/path and prometheus.io/port. The job must also relabel the Service's namespace into a namespace label: the Grafana dashboards filter every query by it.

The prometheus.io/port annotation must equal fluss.metrics.prometheus.port, the port the reporter listens on.

ServiceMonitor-based scraping

Label the metrics Services and keep the port name, then create a ServiceMonitor that selects that label. The fluss-bundle chart creates no ServiceMonitor; you create it yourself, and it needs the Prometheus Operator's monitoring.coreos.com/v1 API in the cluster.

1fluss:
2  metrics:
3    reporters: prometheus
4    prometheus:
5      port: 9249
6      service:
7        portName: metrics
8        labels:
9          monitoring: enabled

Save the ServiceMonitor as fluss-servicemonitor.yaml. Its port matches fluss.metrics.prometheus.service.portName, and its path is /, where the reporter serves:

1apiVersion: monitoring.coreos.com/v1
2kind: ServiceMonitor
3metadata:
4  name: fluss-metrics
5spec:
6  selector:
7    matchLabels:
8      monitoring: enabled
9  endpoints:
10    - port: metrics
11      path: /

Create it in the Fluss namespace: it has no namespace selector, so it selects Services in its own namespace only.

1kubectl apply --namespace <NAMESPACE> -f fluss-servicemonitor.yaml

The Prometheus Operator picks up the ServiceMonitor only when the Prometheus resource's serviceMonitorSelector and serviceMonitorNamespaceSelector match it. Setting both to {} selects every ServiceMonitor in every namespace. On a kube-prometheus-stack install, also set serviceMonitorSelectorNilUsesHelmValues to false, so the chart uses the empty selectors as they are.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Deploying the fluss-grafana Chart

The fluss-grafana chart installs two Grafana dashboards for the Fluss server metrics, packaged as one ConfigMap, <RELEASE>-fluss-dashboards, that Grafana's dashboard sidecar imports. Both dashboards filter every query by a namespace variable, so one Grafana instance serves several Fluss clusters:

Dashboard Prerequisites

• Grafana's dashboard sidecar, which imports every ConfigMap carrying its discovery label as dashboards. kube-prometheus-stack enables it by default; see the Grafana chart's sidecar for dashboards.
• A Prometheus datasource in Grafana that queries the Prometheus instance scraping the Fluss metrics Services.

Configurable values

Install

Install the chart in the namespace Grafana runs in, with the UID of your Prometheus datasource:

1helm install fluss-grafana \
2  oci://registry.ververica.cloud/platform-charts/fluss-grafana \
3  --version 1.0.0-vv-3 \
4  --namespace <GRAFANA_NAMESPACE> \
5  --set datasource=<PROMETHEUS_DATASOURCE_UID>

datasource is the UID of a Grafana datasource, not its display name. Grafana lists the UIDs under Connections > Data sources. With kube-prometheus-stack, the default Prometheus datasource UID is prometheus, the chart's default.

To change the discovery label for a sidecar configured differently, or to load the dashboards into a dedicated Grafana folder instead of the sidecar's default folder, write a values file, fluss-grafana-values.yaml:

1datasource: <PROMETHEUS_DATASOURCE_UID>
2labels:
3  grafana_dashboard: "<SIDECAR_LABEL_VALUE>"
4annotations:
5  k8s-sidecar-target-directory: /tmp/dashboards/Fluss

After the install, the sidecar imports both dashboards into the configured folder, or into its default folder when no k8s-sidecar-target-directory annotation is set. Set each dashboard's namespace variable to the namespace your Fluss cluster runs in.

Customizing the Dashboards

To add panels, change thresholds, or remove charts, pull the chart and unpack it:

1helm pull oci://registry.ververica.cloud/platform-charts/fluss-grafana \
2  --version 1.0.0-vv-3 --untar

Edit fluss-grafana/dashboards/fluss-overview.json and fluss-grafana/dashboards/fluss-detail.json. Keep the ${PROMETHEUS_DATASOURCE_ID} placeholder where a panel names its datasource: the chart replaces it with the datasource value. Then install your copy in place of the published chart, upgrading the existing release:

1helm upgrade --install fluss-grafana ./fluss-grafana \
2  --namespace <GRAFANA_NAMESPACE> \
3  --set datasource=<PROMETHEUS_DATASOURCE_UID>

Verifying End-to-End

Confirm the metrics Services serve metrics

Forward a local port to the tablet metrics Service. If you set fluss.metrics.prometheus.port to another value than 9249, use it in place of 9249 in this command and the next:

1kubectl port-forward --namespace <NAMESPACE> \
2  svc/<RELEASE>-tablet-server-metrics-hs 9249:9249

In a second terminal, read the reporter's output:

1curl -s http://localhost:9249/

The output is Prometheus text: one line per series, beside # HELP and # TYPE lines, with the tablet server metrics named fluss_tabletserver_…. Repeat with svc/<RELEASE>-coordinator-server-metrics-hs, whose metrics are named fluss_coordinator_….

Confirm Prometheus is scraping Apache Fluss

Forward a local port to Prometheus, which serves its web UI on port 9090:

1kubectl port-forward --namespace <PROMETHEUS_NAMESPACE> \
2  svc/<PROMETHEUS_SERVICE> 9090:9090

Open http://localhost:9090. Under Status > Targets, every coordinator and tablet server pod is a target with state UP. Query fluss_coordinator_activeCoordinatorCount: it returns a series labelled with your Fluss namespace.

Confirm the dashboards are loaded

In Grafana, search the dashboards for Fluss: Fluss Overview – $namespace and Fluss Detail – $namespace are listed. Open one and choose your Fluss namespace in its Namespace variable.

Troubleshooting

Metrics endpoint refuses connections

Read the reporter settings the chart rendered on a server pod:

1kubectl exec --namespace <NAMESPACE> <TABLET_POD> -- \
2  cat /opt/fluss/conf/server.yaml | grep metrics

Expected:

1metrics.reporters: prometheus
2metrics.reporter.prometheus.port: 9249

If the lines are missing, the reporter values did not take effect: set them under fluss.metrics, not under fluss.configurationOverrides, and upgrade the release. The Fluss servers read their configuration only at start, and a change to it alone restarts no Fluss pod, so restart the Fluss StatefulSets after the upgrade:

1kubectl rollout restart statefulset coordinator-server tablet-server --namespace <NAMESPACE>

If they are present and the port is still closed, the metrics-prometheus plugin is not installed, and the server's log warns that No reporter plugin can be found for reporter prometheus while the server keeps running without it. Check the init container's logs, then the /opt/fluss/plugins/prometheus directory on the pod, as Setup directory not populated after pod start describes.

Metrics Services are missing or have no metrics port

The chart creates the metrics Services only when fluss.metrics.reporters includes prometheus. Describe them:

1kubectl describe service --namespace <NAMESPACE> <RELEASE>-coordinator-server-metrics-hs
2kubectl describe service --namespace <NAMESPACE> <RELEASE>-tablet-server-metrics-hs

Each Service prints its port as a Port: line; look for:

1Port:  metrics  9249/TCP

If the Services do not exist, enable the reporter. If the port has another name, a ServiceMonitor with port: metrics does not match it: set fluss.metrics.prometheus.service.portName to the name the ServiceMonitor uses.

ServiceMonitor is not discovered by Prometheus

The Prometheus resource's serviceMonitorSelector or serviceMonitorNamespaceSelector filters out the ServiceMonitor. Read the active selectors:

1kubectl describe prometheus --namespace <PROMETHEUS_NAMESPACE>

Either set both selectors to {}, with serviceMonitorSelectorNilUsesHelmValues set to false on a kube-prometheus-stack install, or add the labels they require to the ServiceMonitor and widen serviceMonitorNamespaceSelector to select the Fluss namespace. Keep the ServiceMonitor in the Fluss namespace: it selects Services in its own namespace only.

Prometheus targets are DOWN

The metrics Services resolve, but Prometheus cannot scrape them:

• Traffic denied. A network policy or service mesh denies traffic from the Prometheus namespace to the Fluss namespace on port 9249. For a mesh, see how Securing Fluss treats the metrics port.
• Mismatched port name. The ServiceMonitor names a port the metrics Services do not expose (see Metrics Services are missing or have no metrics port).

Grafana shows No data on dashboards

• The datasource value does not match the UID of a Grafana datasource, or that datasource does not query the Prometheus instance scraping Fluss. Check the UID under Connections > Data sources and upgrade the fluss-grafana release with the right one.
• The dashboard's namespace variable is unset or names another namespace. Every query filters by it, so a wrong value returns no series.
• The Namespace list is empty: it is filled from the coordinator's metrics, so Prometheus scrapes no coordinator. See Confirm Prometheus is scraping Fluss.

Further Reading

• Deploying Fluss on Kubernetes: deploying the cluster this manual monitors
• Installing Fluss Setups: installing the reporter plugin through an init container
• Securing Fluss: how a service mesh treats the metrics port
• Apache Fluss Observability Quickstart: metrics
• Fluss Helm Chart Documentation: chart values
• Fluss Configuration Reference: configuration keys
• Prometheus Operator API Reference: the ServiceMonitor schema
• kube-prometheus-stack Helm Chart: serviceMonitorSelector, datasource provisioning, and the Grafana dashboard sidecar