Deploying Apache Fluss™ on Kubernetes

8 min read

Install, verify, harden, and remove an Apache Fluss cluster on Kubernetes with the Ververica bundle Helm chart.

On this page

Applies to: fluss-bundle chart, Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

The fluss-bundle Helm chart deploys a complete Fluss cluster on Kubernetes: a coordinator server, tablet servers, and a bundled ZooKeeper ensemble. Clients connect on the client listener, port 9124. Before you start, you need the image credential pair for the Ververica registry; the chart itself is public.

Prerequisites

A Kubernetes cluster at version 1.24 or later, with kubectl configured for it.
Helm 3.8 or later, for OCI chart support.
Access to the Ververica registry. See Obtaining Registry Access.
Capacity for the default pod count: 7 pods (1 coordinator server, 3 tablet servers, 3 ZooKeeper nodes), plus the resource requests you configure.
A namespace for this release alone: one Fluss release per namespace (see Managing Multiple Fluss Clusters).

Registry Credentials

The Fluss and ZooKeeper images need the image credential pair; the Helm charts are public and need none. To request the pair, see Obtaining Registry Access.

Create the Image Pull Secret

Create the release namespace and an image pull secret in it. The Fluss pods and the ZooKeeper pods use the same secret:

BASH

1kubectl create namespace <NAMESPACE>
2
3kubectl create secret docker-registry registry-credentials \
4  --docker-server=registry.ververica.cloud \
5  --docker-username=<REGISTRY_USERNAME> \
6  --docker-password=<REGISTRY_PASSWORD> \
7  --namespace <NAMESPACE>

Verify Registry Access

Verify Helm OCI access

The chart pull needs no credentials. To check it, pull the chart as shown in Pulling the Charts.

Verify Docker image pull from the cluster

Run a one-shot pod in the release namespace that pulls the Fluss image through the pull secret:

BASH

1kubectl run registry-check \
2  --namespace <NAMESPACE> \
3  --rm -it --restart=Never \
4  --image=registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3 \
5  --overrides='{"spec":{"imagePullSecrets":[{"name":"registry-credentials"}]}}' \
6  -- /bin/sh -c "echo image pulled successfully"

The pod prints image pulled successfully and exits when the pull secret works. If the pull fails, see Troubleshooting.

Install the Chart

Prepare values.yaml

Reference the pull secret for both images:

YAML

1fluss:
2  image:
3    pullSecrets:
4      - registry-credentials
5
6zookeeper:
7  image:
8    pullSecrets:
9      - registry-credentials

ZooKeeper needs its own entry: it comes from its own chart, the Bitnami ZooKeeper chart (https://charts.bitnami.com/bitnami), and its image is served from the same registry, as registry.ververica.cloud/platform-images/bitnamilegacy/zookeeper.

Do not set the Fluss image tag. The published chart already pins the Fluss image to its own release.

To add plugins through init containers, see Installing Fluss Setups.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Install

BASH

1helm install <RELEASE_NAME> oci://registry.ververica.cloud/platform-charts/fluss-bundle \
2  --version 1.0.0-vv-3 \
3  --namespace <NAMESPACE> \
4  -f values.yaml

Upgrade

To apply a changed values.yaml to the same release:

BASH

1helm upgrade <RELEASE_NAME> oci://registry.ververica.cloud/platform-charts/fluss-bundle \
2  --version 1.0.0-vv-3 \
3  --namespace <NAMESPACE> \
4  -f values.yaml

The Fluss servers read their configuration only at start, and a change to it alone restarts no Fluss pod. If only the Fluss configuration changed, restart the Fluss StatefulSets to apply it:

BASH

1kubectl rollout restart statefulset coordinator-server tablet-server --namespace <NAMESPACE>

To move to a new release, point at its version:

BASH

1helm upgrade <RELEASE_NAME> oci://registry.ververica.cloud/platform-charts/fluss-bundle \
2  --version <NEW_CHART_VERSION> \
3  --namespace <NAMESPACE> \
4  -f values.yaml

Warning

A version upgrade makes the cluster briefly unavailable. helm upgrade to a new version triggers a rolling restart of the Fluss StatefulSets. The single coordinator has a brief outage during its restart, and clients reconnect within seconds. Tablet servers restart one at a time, and each restart causes leader failover for the buckets that server leads, with short read and write latency spikes. If your workload is sensitive to this, plan upgrades during a maintenance window.

Verify the Deployment

List the pods and check that every pod is Running and ready:

BASH

1kubectl get pods --namespace <NAMESPACE>

With the chart defaults, expect:

Component	Pods
Component	Pods
Coordinator server	1
Tablet servers	3
ZooKeeper	3

List the services:

BASH

1kubectl get services --namespace <NAMESPACE>

Clients connect through the coordinator Service, coordinator-server-hs, on port 9124 (see Connecting to Fluss).

Managing Multiple Apache Fluss Clusters

Run one Fluss release per namespace. The client-facing Service names do not depend on the release name, so two releases in one namespace collide.

To run a second cluster, install a second release in its own namespace, with its own pull secret:

BASH

1kubectl create namespace <SECOND_NAMESPACE>
2
3kubectl create secret docker-registry registry-credentials \
4  --docker-server=registry.ververica.cloud \
5  --docker-username=<REGISTRY_USERNAME> \
6  --docker-password=<REGISTRY_PASSWORD> \
7  --namespace <SECOND_NAMESPACE>
8
9helm install <SECOND_RELEASE_NAME> oci://registry.ververica.cloud/platform-charts/fluss-bundle \
10  --version 1.0.0-vv-3 \
11  --namespace <SECOND_NAMESPACE> \
12  -f <SECOND_VALUES_FILE>

Upgrade and uninstall each release in its own namespace.

Connecting to Apache Fluss

Clients bootstrap from the coordinator Service on the client port:

TEXT

1coordinator-server-hs.<NAMESPACE>.svc.cluster.local:9124

The address depends only on the namespace, not on the release name.

For configuring clients against this address, see Reading and Writing Fluss.

Production Hardening

The storage, scheduling, and disruption-budget keys below sit under fluss.tablet for the tablet servers and under fluss.coordinator for the coordinator server. Only the tablet servers have the configurable readiness probe; the coordinator's readiness probe is a fixed port check on the client port that no value configures.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Persistent Storage

By default the chart creates no persistent volume for the tablet servers (fluss.tablet.storage.enabled: false), and Fluss keeps its data under in-pod paths:

YAML

1fluss:
2  configurationOverrides:
3    data.dir: /tmp/fluss/data
4    remote.data.dir: /tmp/fluss/remote-data

Enable a persistent volume for the tablet servers and persistence for ZooKeeper. The sizes and the storage class below are an example to adapt to your workload and cluster:

YAML

1fluss:
2  tablet:
3    storage:
4      enabled: true
5      size: 300Gi
6      storageClass: <STORAGE_CLASS>
7
8zookeeper:
9  persistence:
10    enabled: true
11    storageClass: <STORAGE_CLASS>
12    accessModes:
13      - ReadWriteOnce
14    size: 4Gi
15    dataLogDir:
16      size: 4Gi

List your cluster's storage classes before choosing <STORAGE_CLASS>:

BASH

1kubectl get storageclasses

To back the cluster with object storage instead of the in-pod remote data directory, see Configuring Remote Storage.

Resource Requests and Limits

The chart sets no resource requests or limits (fluss.resources: {}, zookeeper.resources: {}). Set them under fluss.resources.coordinatorServer, fluss.resources.tabletServer, and zookeeper.resources. The example below sets requests equal to limits; the values are a shape to adapt, not a recommendation:

YAML

1fluss:
2  resources:
3    coordinatorServer:
4      requests:
5        cpu: "2"
6        memory: 4Gi
7      limits:
8        cpu: "2"
9        memory: 4Gi
10    tabletServer:
11      requests:
12        cpu: "6"
13        memory: 24Gi
14      limits:
15        cpu: "6"
16        memory: 24Gi
17
18zookeeper:
19  resources:
20    requests:
21      cpu: "1"
22      memory: 2Gi
23    limits:
24      cpu: "1"
25      memory: 2Gi

Readiness Probe

The tablet readiness probe asks the local tablet server for the cluster health, which the tablet server forwards to the coordinator over the internal listener. A freshly created tablet pod is marked Ready only when the cluster is green. The probe keeps this cluster-health check for at most healthCheckTimeoutSeconds, then falls back to a plain port check, so a cluster that cannot recover does not block a rolling upgrade indefinitely. Once the cluster has been green, or the timeout has passed, the probe stays a plain port check: a running tablet pod does not turn NotReady when the cluster leaves green, and the cluster-health check runs again when the pod is recreated.

The keys sit under fluss.tablet.readinessProbe:

Key	Default	Meaning
Key	Default	Meaning
rpcTimeoutMs	5000	Timeout of the health RPC, in milliseconds
failureThreshold	360	Kubernetes probe failure threshold
timeoutSeconds	10	Kubernetes probe timeout
initialDelaySeconds	15	Kubernetes probe initial delay
periodSeconds	5	Kubernetes probe period
healthCheckTimeoutSeconds	1200	How long the cluster-health check can keep a new pod NotReady before the port-check fallback
healthCheckAuth	""	Client credentials for the probe

Kubernetes waits initialDelaySeconds + failureThreshold * periodSeconds before marking the pod NotReady: 15 + 360 * 5 = 1815 seconds with the defaults. Keep failureThreshold * periodSeconds larger than healthCheckTimeoutSeconds: otherwise Kubernetes marks the pod NotReady before the fallback takes over. If you raise healthCheckTimeoutSeconds, raise failureThreshold with it.

healthCheckAuth is needed only when the client listener enforces SASL. See Configuring Client-Facing SASL.

Scheduling and Disruption Budgets

Concern	Keys
Concern	Keys
Scheduling	affinity, nodeSelector, tolerations, topologySpreadConstraints
Disruption budget	podDisruptionBudget.enabled, with exactly one of minAvailable or maxUnavailable

The disruption budget is off by default. The example below spreads the tablet servers across availability zones and nodes and keeps at least one available during voluntary disruptions. Replace <RELEASE_NAME> with your Helm release name:

YAML

1fluss:
2  tablet:
3    podDisruptionBudget:
4      enabled: true
5      minAvailable: 1
6    affinity:
7      podAntiAffinity:
8        preferredDuringSchedulingIgnoredDuringExecution:
9          - weight: 100
10            podAffinityTerm:
11              topologyKey: topology.kubernetes.io/zone
12              labelSelector:
13                matchLabels:
14                  app.kubernetes.io/instance: <RELEASE_NAME>
15                  app.kubernetes.io/component: tablet
16          - weight: 50
17            podAffinityTerm:
18              topologyKey: kubernetes.io/hostname
19              labelSelector:
20                matchLabels:
21                  app.kubernetes.io/instance: <RELEASE_NAME>
22                  app.kubernetes.io/component: tablet

Make the ZooKeeper anti-affinity hard, so the 3 ZooKeeper pods run on different nodes: losing one node then takes down at most one ZooKeeper pod, and the other two keep a majority (quorum). The hard preset needs at least 3 nodes ZooKeeper can run on. While one of them is down, the displaced ZooKeeper pod stays Pending rather than sharing a node; quorum holds at 2 of 3:

YAML

1zookeeper:
2  podAntiAffinityPreset: hard

If you dedicate node pools to Fluss, pin each component to its own nodes with a node selector and the matching toleration. With dedicated pools sized for one tablet server per node, the sizing alone keeps the tablet servers apart:

YAML

1fluss:
2  tablet:
3    nodeSelector:
4      <NODE_LABEL_KEY>: <TABLET_NODE_LABEL_VALUE>
5    tolerations:
6      - key: <TAINT_KEY>
7        operator: Equal
8        value: <TABLET_TAINT_VALUE>
9        effect: NoSchedule
10  coordinator:
11    nodeSelector:
12      <NODE_LABEL_KEY>: <CONTROL_NODE_LABEL_VALUE>
13    tolerations:
14      - key: <TAINT_KEY>
15        operator: Equal
16        value: <CONTROL_TAINT_VALUE>
17        effect: NoSchedule
18
19zookeeper:
20  nodeSelector:
21    <NODE_LABEL_KEY>: <CONTROL_NODE_LABEL_VALUE>
22  tolerations:
23    - key: <TAINT_KEY>
24      operator: Equal
25      value: <CONTROL_TAINT_VALUE>
26      effect: NoSchedule

Replication and Bucketing

The chart sets these defaults:

YAML

1fluss:
2  configurationOverrides:
3    default.bucket.number: 3
4    default.replication.factor: 3
5  tablet:
6    numberOfReplicas: 3

default.bucket.number is the default number of buckets per table, default.replication.factor the default number of replicas of each bucket, and fluss.tablet.numberOfReplicas the number of tablet server pods. A table's replication factor must not exceed the number of tablet servers available when the table is created, so keep default.replication.factor at or below fluss.tablet.numberOfReplicas. For every server configuration key, see the Fluss configuration reference.

Service Account

The chart creates no service account by default. To have it create one:

YAML

1fluss:
2  serviceAccount:
3    create: true
4    name: <SERVICE_ACCOUNT_NAME>
5    annotations: {}

When create is true and name is empty, the name is derived from the release name. Use annotations for integrations such as workload identity.

Config Providers

Keep a sensitive configuration value, such as a storage credential, out of values.yaml: store it in a Kubernetes Secret and reference it from fluss.configurationOverrides through a config-provider marker. The values file then holds only the reference, and the server redacts the resolved values. Declare the Secret under fluss.secrets; the chart generates the matching config.providers configuration. For the providers themselves, see Secrets in the Apache Fluss documentation.

The chart offers two marker forms:

Declared under	Each entry	Marker in configurationOverrides
Declared under	Each entry	Marker in configurationOverrides
fluss.secrets.env	name, secretName, key: one Secret key as an environment variable	${env:<NAME>}
fluss.secrets.mounts	name, secretName: a Secret mounted under fluss.secrets.basePath (default /etc/fluss/secrets)	${directory:<BASE_PATH>/<NAME>:<KEY>}

The server resolves a marker only when it is the whole value: a value with text around a marker, such as a URI with a marker inside it, is kept literally. To write a value that starts with a literal ${, write $${ instead.

The mount form reads one key of a mounted Secret:

YAML

1fluss:
2  secrets:
3    mounts:
4      - name: s3-creds
5        secretName: fluss-s3-credentials
6  configurationOverrides:
7    s3.secret.key: ${directory:/etc/fluss/secrets/s3-creds:secret-key}

Example with the environment form: create the Secret in the release namespace, then reference each key through a marker:

BASH

1kubectl create secret generic fluss-storage-credentials \
2  --from-literal=s3-access-key=<S3_ACCESS_KEY> \
3  --from-literal=s3-secret-key=<S3_SECRET_KEY> \
4  --namespace <NAMESPACE>

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

For each storage backend's credentials, see Configuring Remote Storage.

Uninstall

Remove the release:

BASH

1helm uninstall <RELEASE_NAME> --namespace <NAMESPACE>

The release's PersistentVolumeClaims survive the uninstall. To delete them, list the claims in the release namespace:

BASH

1kubectl get pvc --namespace <NAMESPACE>

Then delete the release's claims, the Fluss and the ZooKeeper ones, by name. <PVC_NAMES> is their names, separated by spaces.

Warning

Deleting the PersistentVolumeClaims is irreversible and deletes all stored data.

BASH

1kubectl delete pvc <PVC_NAMES> --namespace <NAMESPACE>

Complete values.yaml Example

The pull secrets and every hardening setting except the optional node pinning, in one file. Sizes, resources, and the storage class are a shape to adapt:

YAML

1fluss:
2  image:
3    pullSecrets:
4      - registry-credentials
5
6  configurationOverrides:
7    default.bucket.number: 3
8    default.replication.factor: 3
9
10  coordinator:
11    numberOfReplicas: 1
12
13  tablet:
14    numberOfReplicas: 3
15    storage:
16      enabled: true
17      size: 300Gi
18      storageClass: <STORAGE_CLASS>
19    readinessProbe:
20      rpcTimeoutMs: 5000
21      failureThreshold: 360
22      timeoutSeconds: 10
23      initialDelaySeconds: 15
24      periodSeconds: 5
25      healthCheckTimeoutSeconds: 1200
26      healthCheckAuth: ""
27    podDisruptionBudget:
28      enabled: true
29      minAvailable: 1
30    affinity:
31      podAntiAffinity:
32        preferredDuringSchedulingIgnoredDuringExecution:
33          - weight: 100
34            podAffinityTerm:
35              topologyKey: topology.kubernetes.io/zone
36              labelSelector:
37                matchLabels:
38                  app.kubernetes.io/instance: <RELEASE_NAME>
39                  app.kubernetes.io/component: tablet
40          - weight: 50
41            podAffinityTerm:
42              topologyKey: kubernetes.io/hostname
43              labelSelector:
44                matchLabels:
45                  app.kubernetes.io/instance: <RELEASE_NAME>
46                  app.kubernetes.io/component: tablet
47
48  resources:
49    coordinatorServer:
50      requests:
51        cpu: "2"
52        memory: 4Gi
53      limits:
54        cpu: "2"
55        memory: 4Gi
56    tabletServer:
57      requests:
58        cpu: "6"
59        memory: 24Gi
60      limits:
61        cpu: "6"
62        memory: 24Gi
63
64  serviceAccount:
65    create: true
66    name: <SERVICE_ACCOUNT_NAME>
67
68zookeeper:
69  image:
70    pullSecrets:
71      - registry-credentials
72  replicaCount: 3
73  persistence:
74    enabled: true
75    storageClass: <STORAGE_CLASS>
76    accessModes:
77      - ReadWriteOnce
78    size: 4Gi
79    dataLogDir:
80      size: 4Gi
81  resources:
82    requests:
83      cpu: "1"
84      memory: 2Gi
85    limits:
86      cpu: "1"
87      memory: 2Gi
88  podAntiAffinityPreset: hard

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Troubleshooting

Pods not starting

Inspect the pod's events and logs:

BASH

1kubectl describe pod <POD_NAME> --namespace <NAMESPACE>
2kubectl logs <POD_NAME> --namespace <NAMESPACE>

Image pull fails

A pod stuck in ImagePullBackOff or ErrImagePull usually means the pull secret is missing, misnamed, or in another namespace. Check that the secret exists in the release namespace:

BASH

1kubectl get secret registry-credentials --namespace <NAMESPACE>

Then check that fluss.image.pullSecrets and zookeeper.image.pullSecrets in values.yaml name it exactly. If the secret is missing, create it as in Create the Image Pull Secret.

The pod's events, from kubectl describe pod, tell the causes apart:

Event	Cause	Fix
Event	Cause	Fix
pull access denied, unauthorized	Pull secret missing, misnamed, or in another namespace	Run the checks above
manifest unknown	The image tag does not exist	Confirm the version against the one Ververica communicated
dial tcp ... i/o timeout	The cluster nodes cannot reach registry.ververica.cloud	Check egress firewalls, NAT gateways, and any registry mirror configured on the nodes

On an authentication failure, also check that the username and password carry no leading or trailing whitespace, and that the credentials were issued for the Fluss projects on the registry, not for an unrelated Ververica product.

ZooKeeper pods fail to pull while the Apache Fluss pods start

The pull secret is set under fluss.image.pullSecrets but not under zookeeper.image.pullSecrets. Add it there too: the ZooKeeper image is served from the same registry.

Tablet pods stay NotReady after turning on client-facing SASL

The tablet pods stay NotReady until fluss.tablet.readinessProbe.healthCheckTimeoutSeconds runs out, then become Ready without a cluster-health check, and the tablet pod log shows the readiness probe failing with The connection has not completed authentication yet. With client-facing SASL on, the readiness probe needs client credentials in fluss.tablet.readinessProbe.healthCheckAuth. See Configuring Client-Facing SASL.