Securing Apache Fluss™

Applies to: fluss-bundle chart, Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

This manual covers authentication, authorization, and in-cluster encryption for a Fluss cluster deployed with the fluss-bundle chart. The Fluss servers authenticate with SASL PLAIN only and carry no TLS of their own, so transport encryption comes from a service mesh.

Prerequisites

• A Fluss cluster deployed with the fluss-bundle chart. See Deploying Fluss on Kubernetes.
• kubectl access to the namespace the cluster runs in.
• For the service mesh, a mesh control plane running in the cluster:
  • Kubernetes 1.29 or later is recommended. The mesh proxy then runs as a native sidecar, so run-to-completion pods finish instead of waiting on a proxy that never exits.
  • Recent Linkerd versions need the Gateway API CRDs installed before the control plane. In production, pick a supported Linkerd distribution and follow its installation guide.
  • Linkerd's viz extension is optional. You need it only to verify mTLS (see Verifying).

SASL Authentication

SASL can be turned on for three planes:

The planes are configured independently and can be turned on in any combination. Each plane takes the mechanism plain, or empty to leave SASL off.

Where Credentials Come From

Keep the SASL credentials in Kubernetes Secrets. The chart also accepts literal username and password pairs, but a literal is rendered as a plain environment value in the pod spec, readable by anyone who can read the pods, while a Secret reference renders only a reference to the Secret.

Create two Secrets in the release namespace: one for the Fluss credentials and one for ZooKeeper. The ZooKeeper Secret is separate because the bundled ZooKeeper comes from its own chart and reads its passwords through that chart, under the fixed key names client-password and server-password.

Create each Secret from a key=value file, so that no credential appears on the command line: the passwords stay out of the process list and your shell history. The file for fluss-sasl-credentials holds every user's key pair:

1client-0-username=<CLIENT_USERNAME>
2client-0-password=<CLIENT_PASSWORD>
3internal-username=<INTERNAL_USERNAME>
4internal-password=<INTERNAL_PASSWORD>

The file for zookeeper-sasl-credentials holds the ZooKeeper user and its password, under both key names:

1username=<ZK_USERNAME>
2client-password=<ZK_PASSWORD>
3server-password=<ZK_PASSWORD>

Then reference the Secrets from values.yaml, for the three planes and for the bundled ZooKeeper:

1fluss:
2  security:
3    client:
4      sasl:
5        mechanism: plain
6        plain:
7          users:
8            - existingSecret:
9                name: fluss-sasl-credentials
10                usernameKey: client-0-username
11                passwordKey: client-0-password
12    internal:
13      sasl:
14        mechanism: plain
15        plain:
16          existingSecret:
17            name: fluss-sasl-credentials
18            usernameKey: internal-username
19            passwordKey: internal-password
20    zookeeper:
21      sasl:
22        mechanism: plain
23        plain:
24          existingSecret:
25            name: zookeeper-sasl-credentials
26            usernameKey: username
27            passwordKey: client-password
28zookeeper:
29  auth:
30    client:
31      enabled: true
32      clientUser: <ZK_USERNAME>
33      serverUsers: <ZK_USERNAME>
34      existingSecret: zookeeper-sasl-credentials
35  extraEnvVars:
36    - name: JVMFLAGS
37      value: -Dzookeeper.extendedTypesEnabled=true -Dzookeeper.enforce.auth.enabled=true -Dzookeeper.enforce.auth.schemes=sasl -Djute.maxbuffer=104857600

The client plane lists one entry per user, each naming its own key pair in the Secret. The internal plane uses one shared credential pair; the internal listener is not reachable by external clients, so it needs no client configuration.

Without -Dzookeeper.enforce.auth.enabled=true in JVMFLAGS, ZooKeeper still accepts connections that do not authenticate. The chart's default sets it to false.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Other secrets. A Secret reference covers the SASL credentials only. Any other secret in the server configuration, such as a storage credential, comes from a Secret through config providers: see Config Providers. Do not write such a secret as a literal: the cluster configuration API returns a literal in plain text to any principal allowed to describe the cluster.

Configuring Client-Facing SASL

With SASL on the client listener, the tablet readiness probe authenticates as a client too. Give it the credentials of a listed user, or the tablet pods stay NotReady until fluss.tablet.readinessProbe.healthCheckTimeoutSeconds runs out, then become Ready anyway:

1fluss:
2  tablet:
3    readinessProbe:
4      healthCheckAuth: client.security.protocol:sasl;client.security.sasl.mechanism:PLAIN;client.security.sasl.username:<CLIENT_USERNAME>;client.security.sasl.password:<CLIENT_PASSWORD>

The probe's credentials are a literal value; they have no Secret form.

With the authorizer on (see Authorization), the probe's user needs the DESCRIBE permission on the cluster: the super user holds it, and any other listed user needs an ACL that grants it. Without it, the tablet pods stay NotReady as they do with wrong credentials.

For the client side, the keys a Flink job or a Java SDK client sets, see Reading and Writing Fluss.

Verifying SASL

• Client-facing. Connect with a client configured for SASL PLAIN with a listed user's credentials (see Reading and Writing Fluss). The connection succeeds. A client without credentials is rejected, and a client with a wrong password or an unknown user is rejected with Invalid username or password.
• Internal. Restart a tablet pod. The coordinator log shows it rejoining the cluster without authentication errors.
• ZooKeeper. Restart the Fluss pods. Their logs show neither a NoAuthException nor Will not attempt to authenticate using SASL.

Rotating Credentials

This rotates the credentials in fluss-sasl-credentials. A change made only inside a Secret, whether a SASL plane references it or config providers read it, leaves the pod template unchanged, so no pod restarts on its own.

To update the Secret, edit the key=value file you created it from with --from-env-file (see Where Credentials Come From): change the passwords in place and keep every user's key pair, including those added for further client users (see Authorization). Re-applying the Secret from the file replaces its whole data with the file's keys, so a key the file leaves out is deleted. Re-apply the whole Secret from that file, then restart the pods so they read it:

1kubectl create secret generic fluss-sasl-credentials \
2  --from-env-file=<FILE> \
3  --namespace <NAMESPACE> \
4  --dry-run=client -o yaml | kubectl apply -f -
5
6kubectl get statefulsets --namespace <NAMESPACE>
7
8kubectl rollout restart statefulset \
9  <COORDINATOR_STATEFULSET> <TABLET_STATEFULSET> <ZOOKEEPER_STATEFULSET> \
10  --namespace <NAMESPACE>

The restart makes the cluster briefly unavailable, as an upgrade does: see Upgrade.

The probe's credentials in fluss.tablet.readinessProbe.healthCheckAuth are the exception. They are a chart value, so changing them is a helm upgrade, which rolls the tablets by itself; restart only the coordinator and ZooKeeper afterwards.

Change the probe's credentials in lockstep with the client user list. Removing the probe's user from the list without updating healthCheckAuth leaves the tablet pods NotReady after the next restart, until fluss.tablet.readinessProbe.healthCheckTimeoutSeconds runs out.

To roll a client credential over without downtime:

1. Add the new user to the client user list, beside the old one, and upgrade.
2. Move the clients to the new credentials.
3. Remove the old user and upgrade again.

Renaming a user, not only changing its password, also means updating every place that names the principal, such as super.users and any ACL grant.

Authorization

The authorizer checks every operation of an authenticated principal against ACLs. Turn it on in the server configuration and name a super user, one of the listed client users:

1fluss:
2  configurationOverrides:
3    authorizer.enabled: true
4    authorizer.type: default
5    super.users: User:<SUPER_USER>

The servers read their configuration only at start, and a change to it alone restarts no Fluss pod, so after the helm upgrade that turns the authorizer on, restart the Fluss servers; until then, every operation is still allowed:

1kubectl rollout restart statefulset coordinator-server tablet-server --namespace <NAMESPACE>

For each further client user, add one entry to the client user list (see Where Credentials Come From) and one key pair to fluss-sasl-credentials, such as client-1-username and client-1-password.

The authorizer denies by default: a principal holds no permission until an ACL grants it. The super user needs none. Creating or dropping ACLs needs the ALL permission on the database; an ACL on a table is checked against the table's database.

To grant and revoke ACLs, see Authorization in the Apache Fluss documentation.

Encrypting Traffic with a Service Mesh

A service mesh runs a sidecar proxy beside each pod and wraps every pod-to-pod connection in mutual TLS (mTLS). Fluss keeps speaking plaintext on its own sockets; the proxies encrypt and authenticate the connections on its behalf. No Fluss configuration changes: you add pod annotations in the chart values.

This manual uses Linkerd as the reference mesh. The same pattern, transparent sidecar mTLS with protocol detection turned off for the binary ports, applies to other meshes, with their own annotation keys.

The mesh composes with SASL. The mesh identifies the workload and encrypts the channel; SASL identifies the Fluss principal. Run both: meshing needs no change to the SASL configuration.

What the Mesh Covers

• Covered: all pod-to-pod traffic in the cluster: client pods to Fluss, server to server, and the servers to ZooKeeper.
• Not covered: a client outside the cluster. It has no sidecar, so the mesh cannot encrypt its connection.

How Apache Fluss and a Mesh Fit Together

Two facts drive the configuration:

1. Fluss RPC and ZooKeeper speak binary protocols. A mesh proxy normally inspects the first bytes of a connection to detect its protocol. On these ports, detection adds latency, and on ZooKeeper's client port, where the server speaks first, it hangs the connection. Mark these ports opaque, so the proxy skips detection and tunnels the encrypted bytes.
2. The sidecar is added only when a pod is created. See Enabling the Mesh at Install Time.

Configuration

Add the annotations to the coordinator, the tablet, and the ZooKeeper pods:

1fluss:
2  coordinator:
3    podAnnotations:
4      linkerd.io/inject: enabled
5      config.linkerd.io/opaque-ports: "9123,9124"
6  tablet:
7    podAnnotations:
8      linkerd.io/inject: enabled
9      config.linkerd.io/opaque-ports: "9123,9124"
10zookeeper:
11  podAnnotations:
12    linkerd.io/inject: enabled
13    config.linkerd.io/opaque-ports: "2181,2888,3888"

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Each pod lists its own inbound ports. A proxy learns that an outbound connection is opaque from the destination pod's annotation, so the Fluss ports are not repeated on the ZooKeeper pods, nor the ZooKeeper ports on the Fluss pods.

Mesh the client pods too, or their traffic to Fluss stays plaintext: add linkerd.io/inject: enabled to their pod template, with no opaque-ports entry. For how jobs connect to Fluss, see Reading and Writing Fluss.

Enabling the Mesh at Install Time

Install Fluss with the annotations in your values, with the mesh control plane already running, so that every pod starts meshed.

Meshing a running cluster means a rolling restart, and that restart can stall. While the tablets roll one at a time the cluster is degraded, and a newly meshed tablet stays NotReady until the readiness probe's cluster-health timeout runs out (see Readiness Probe). The rollout completes, slowly.

Verifying

1kubectl get pods --namespace <NAMESPACE>

1linkerd viz edges pod --namespace <NAMESPACE>

Interaction with Network Policies

Between meshed pods, the bytes on the wire go to the mesh proxy port, 4143 for Linkerd, not to the Fluss or ZooKeeper ports. Network policies and security groups between pods must allow it. The kubelet's probes still reach the application ports directly from the node, so those paths stay allowed too.

Troubleshooting

Client authentication fails

The client log shows Authentication failed, an AuthenticationException, or a connection dropped right after the SASL handshake.

• Check the client's SASL settings against Reading and Writing Fluss.
• Check that the credentials match a user in fluss.security.client.sasl.plain.users.
• After a change made only inside a Secret, check that the pods were restarted (see Rotating Credentials).

Tablet pods stay NotReady after turning on client-facing SASL

The first tablet pod runs but stays NotReady until fluss.tablet.readinessProbe.healthCheckTimeoutSeconds runs out, then becomes Ready anyway, and its log shows the readiness probe failing with The connection has not completed authentication yet. Set fluss.tablet.readinessProbe.healthCheckAuth with the credentials of a listed user (see Configuring Client-Facing SASL) and run helm upgrade.

Coordinator and tablets cannot communicate after turning on internal SASL

The coordinator shows the tablet servers as offline, and the tablet logs show authentication errors against the internal listener. If the helm install or helm upgrade output warns about auto-generated SASL credentials, the explicit internal credentials were not set: set them (see Where Credentials Come From) and upgrade.

Apache Fluss cannot connect to ZooKeeper after turning on ZooKeeper SASL

The Fluss pods crash-loop with AuthFailed, or log Will not attempt to authenticate using SASL.

• Check that both sides read the same Secret and key: fluss.security.zookeeper.sasl.plain.existingSecret and zookeeper.auth.client.existingSecret name the same Secret, and the Fluss side reads its password from client-password.
• Check that the ZooKeeper pods carry -Dzookeeper.enforce.auth.enabled=true in JVMFLAGS.
• Check that fluss.security.zookeeper.sasl.mechanism is plain.

Values validation errors at install or upgrade

The chart validates the SASL configuration. It refuses a mechanism other than plain or empty, and it refuses client-facing SASL with no user in the user list. It also refuses a client user entry that sets both a literal username or password and a Secret reference, a literal client user entry without both a username and a password, ZooKeeper SASL with neither a Secret reference nor both a literal username and password, and ZooKeeper SASL with an empty login module class.

A principal is denied after turning on the authorizer

The authorizer denies by default, so a principal with no ACL can do nothing. Grant the principal the ACLs its operations need (see the Apache Fluss authorization documentation).

A meshed pod's proxy never becomes Ready

Read the proxy container's log and check the control plane:

1kubectl logs <POD_NAME> --namespace <NAMESPACE> -c linkerd-proxy
2linkerd check

Connections hang or time out after meshing

This is almost always a missing opaque port. Check the effective annotations on the pods, remembering that the list replaces Linkerd's defaults:

1kubectl get pod <POD_NAME> --namespace <NAMESPACE> -o jsonpath='{.metadata.annotations}'

The rolling restart stalls while meshing a running cluster

This is expected (see Enabling the Mesh at Install Time). Wait for the readiness probe's timeout, or install meshed from the start.

Further Reading

• Deploying Fluss on Kubernetes: config providers, the availability impact of a rolling restart, and the readiness probe's tuning
• Reading and Writing Fluss: the client side of SASL, and how client jobs connect
• Fluss Security Overview: security and SASL
• Fluss Authentication: server and client SASL properties
• Fluss Authorization: granting and revoking ACLs
• Fluss Helm Chart Documentation: chart values
• Linkerd Documentation: installing the mesh, automatic mTLS, and the viz extension
• Kubernetes Gateway API: the CRDs recent Linkerd versions need