Installing Apache Fluss™ Setups

Applies to: fluss-bundle chart, Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

The Fluss Docker image ships a setup installer, /opt/fluss/bin/setup/install.sh, that downloads and installs Fluss setups: server plugins for the Fluss coordinator and tablet servers, classpath JARs for client applications, and runnable binaries such as the lake-tiering job. The installer is Ververica's own and not part of Apache Fluss, so the Apache Fluss documentation does not describe it. It resolves JARs through an embedded Maven wrapper, so no Maven installation is needed. The Fluss image already carries everything the installer needs; to run it outside the image, you need a JDK and either curl or wget. On Kubernetes, you run it in an init container that writes the server plugins to a volume the Fluss containers mount.

Prerequisites

• A Fluss cluster deployed with the fluss-bundle chart. See Deploying Fluss on Kubernetes.
• Access to the Fluss Docker image, which the installer runs from. See Obtaining Registry Access.
• Network access from the pods that run the installer to the Maven repositories it resolves from. If they cannot reach them, see Using a private Maven repository.

Available Setups

Every option a setup declares is required and takes exactly one value. The same options install the server plugin and the client classpath. What each value adds:

• --fs s3: the S3 filesystem plugin and the lake format's storage JARs for S3. For lake-iceberg, it also covers NooBaa and other S3-compatible stores.
• --fs abs: the Azure filesystem plugin and the Iceberg storage JARs for Azure Blob Storage.
• --catalog rest: nothing beyond the base JARs. Polaris and Lakekeeper are both REST catalogs.
• --catalog jdbc-postgres: the PostgreSQL JDBC driver, on the server and the client.
• --catalog hadoop and --catalog filesystem: nothing beyond the base JARs.

On the client, the lake-iceberg install brings the Iceberg bundle for its --fs store, so one install is all a Flink lake job needs. The fs-s3 and fs-azure setups serve clients that read the remote log and snapshots without a lake.

Setups install into one of two targets:

• Server: install.sh fluss <setup> installs into /opt/fluss/plugins/<plugin-dir>, where the plugin directory is fixed per setup and shown in the table.
• Client: install.sh client <setup> installs into <flink-home>/lib/fluss-<setup> when a Flink home is given, or into <flink-home>/usrlib/fluss-lake-tiering-service for the tiering job, which is a job JAR rather than a classpath library. For a client that is not Flink, such as a Fluss Java SDK application, pass --output-dir.

To list the setups and the options each one requires, run the installer from the image:

1docker run --rm registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3 \
2  /opt/fluss/bin/setup/install.sh list

To show one setup's options on both targets:

1docker run --rm registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3 \
2  /opt/fluss/bin/setup/install.sh help lake-iceberg

list and help read a setup's options through Maven, so they need the same network access as an install. They take no Maven settings file: where Maven cannot run, they print (options unavailable: …) in place of a setup's options.

How Setup Installation Works

An init container on each Fluss server pod runs the installer from the Fluss image:

1. A shared emptyDir volume is mounted at /opt/fluss/plugins in the init container.
2. Each install.sh fluss <setup> call writes its JARs to the setup's default path, /opt/fluss/plugins/<plugin-dir>, on that volume.
3. The Fluss server container mounts each setup's subdirectory of the volume, by subPath, at the same /opt/fluss/plugins/<plugin-dir> path.

Configure the init container, the volume, and the mounts on both fluss.coordinator and fluss.tablet. The subPath of each mount is the setup's plugin directory from the Available Setups table.

Installing Setups on Kubernetes

Single setup

The following values install metrics-prometheus on the tablet and coordinator servers:

1fluss:
2  tablet:
3    extraVolumes:
4      - name: fluss-plugins
5        emptyDir: {}
6    extraVolumeMounts:
7      - name: fluss-plugins
8        mountPath: /opt/fluss/plugins/prometheus
9        subPath: prometheus
10    initContainers:
11      - name: install-plugins
12        image: registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
13        command:
14          - /bin/sh
15          - -c
16          - |
17            set -e
18            /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- -q
19        volumeMounts:
20          - name: fluss-plugins
21            mountPath: /opt/fluss/plugins
22  coordinator:
23    extraVolumes:
24      - name: fluss-plugins
25        emptyDir: {}
26    extraVolumeMounts:
27      - name: fluss-plugins
28        mountPath: /opt/fluss/plugins/prometheus
29        subPath: prometheus
30    initContainers:
31      - name: install-plugins
32        image: registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
33        command:
34          - /bin/sh
35          - -c
36          - |
37            set -e
38            /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- -q
39        volumeMounts:
40          - name: fluss-plugins
41            mountPath: /opt/fluss/plugins

For another setup, replace the setup name in the install.sh call and the plugin directory in mountPath and subPath. Add the setup's options to the install.sh call, for example install.sh fluss lake-iceberg --fs s3 --catalog rest --force -- -q.

Use the same image tag in the init container as in the Fluss deployment: the installer installs the Fluss version its image was built with.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Multiple setups in one init container

Chain the install.sh calls in one init container, and add one extraVolumeMounts entry per setup. The following values install metrics-prometheus and fs-s3:

1fluss:
2  tablet:
3    extraVolumes:
4      - name: fluss-plugins
5        emptyDir: {}
6    extraVolumeMounts:
7      - name: fluss-plugins
8        mountPath: /opt/fluss/plugins/prometheus
9        subPath: prometheus
10      - name: fluss-plugins
11        mountPath: /opt/fluss/plugins/s3
12        subPath: s3
13    initContainers:
14      - name: install-plugins
15        image: registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
16        command:
17          - /bin/sh
18          - -c
19          - |
20            set -e
21            /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- -q
22            /opt/fluss/bin/setup/install.sh fluss fs-s3 --force -- -q
23        volumeMounts:
24          - name: fluss-plugins
25            mountPath: /opt/fluss/plugins
26  coordinator:
27    extraVolumes:
28      - name: fluss-plugins
29        emptyDir: {}
30    extraVolumeMounts:
31      - name: fluss-plugins
32        mountPath: /opt/fluss/plugins/prometheus
33        subPath: prometheus
34      - name: fluss-plugins
35        mountPath: /opt/fluss/plugins/s3
36        subPath: s3
37    initContainers:
38      - name: install-plugins
39        image: registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
40        command:
41          - /bin/sh
42          - -c
43          - |
44            set -e
45            /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- -q
46            /opt/fluss/bin/setup/install.sh fluss fs-s3 --force -- -q
47        volumeMounts:
48          - name: fluss-plugins
49            mountPath: /opt/fluss/plugins

The init container keeps its single mount at /opt/fluss/plugins; only the server container gets a mount per setup. To configure the storage backend that fs-s3 or fs-azure serves, see Configuring Remote Storage.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Using a private Maven repository

By default, the installer resolves the Fluss JARs from the public Ververica Maven repository, https://ververica.jfrog.io/artifactory/fluss-release, which needs no credentials, and their other dependencies from Maven Central, repo.maven.apache.org. No Maven settings file is needed when the pods can reach both.

If the pods must resolve through a mirror or a private Maven repository instead, pass a Maven settings file to the installer. The settings file changes where the JARs resolve from, not where the Maven wrapper downloads Maven: the pods still need repo.maven.apache.org. With no public egress at all, pre-bake the setups instead. If your repository needs credentials, put them in the settings file.

1kubectl create secret generic mvn-settings \
2  --from-file=settings.xml=<PATH_TO_SETTINGS_XML> \
3  --namespace <NAMESPACE>

1fluss:
2  tablet:
3    extraVolumes:
4      - name: fluss-plugins
5        emptyDir: {}
6      - name: mvn-settings
7        secret:
8          secretName: mvn-settings
9    extraVolumeMounts:
10      - name: fluss-plugins
11        mountPath: /opt/fluss/plugins/prometheus
12        subPath: prometheus
13    initContainers:
14      - name: install-plugins
15        image: registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
16        command:
17          - /bin/sh
18          - -c
19          - |
20            set -e
21            /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- --settings=<SETTINGS_PATH> -q
22        volumeMounts:
23          - name: fluss-plugins
24            mountPath: /opt/fluss/plugins
25          - name: mvn-settings
26            mountPath: <SETTINGS_PATH>
27            subPath: settings.xml

For a setup that takes options, the installer also runs maven-help-plugin to read them, so your mirror must serve it. If it does not, the installer fails and names the plugin.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Setups with a client target

install.sh client <setup> installs a setup's client JARs into a classpath directory. It takes the same options as the server install.

For a Flink installation, pass --flink-home, or set $FLINK_HOME. The installer writes into the Flink home:

1/opt/fluss/bin/setup/install.sh client lake-iceberg --fs s3 --catalog rest --flink-home <FLINK_HOME>

For any other client, such as a Fluss Java SDK application or a job on Ververica Platform 3, pass the directory explicitly. --output-dir also overrides the Flink default. To get the client JARs into a local directory, run the installer from the image with that directory mounted, one command per setup:

1mkdir -p <LOCAL_DIR>
2
3docker run --rm -v "$PWD/<LOCAL_DIR>":/output \
4  registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3 \
5  /opt/fluss/bin/setup/install.sh client <SETUP> <OPTIONS> --output-dir /output/<SETUP>

Each command ends with Installed to: /output/<SETUP> and lists the setup's JARs, which land in <LOCAL_DIR>/<SETUP>/.

For the client setups a job needs to write, read, or read tiered tables, see Reading and Writing Fluss.

The client JARs are now in a local directory. To run a job with them on Ververica Platform 3, see Running Fluss Jobs on Ververica Platform 3.

Pre-baking Setups into a Custom Image

The init container downloads the setups each time a pod starts. To avoid that, for example where egress is restricted or startup time matters, install the setups when you build an image.

1FROM registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3
2
3RUN /opt/fluss/bin/setup/install.sh fluss metrics-prometheus --force -- -q && \
4    /opt/fluss/bin/setup/install.sh fluss fs-s3 --force -- -q

1fluss:
2  image:
3    registry: <YOUR_REGISTRY>
4    repository: <YOUR_REPOSITORY>
5    tag: <YOUR_TAG>
6    pullSecrets:
7      - <YOUR_PULL_SECRET>

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Command Reference

The installer is /opt/fluss/bin/setup/install.sh in the Fluss image.

1install.sh list
2install.sh help   <setup-name>
3install.sh fluss  <setup-name> [--<option> <value>]... [--output-dir <dir>] [--force] [-- <maven-args>...]
4install.sh client <setup-name> [--<option> <value>]... [--flink-home <dir>] [--output-dir <dir>] [--force] [-- <maven-args>...]

Default output directories:

• fluss: /opt/fluss/plugins/<plugin-dir>.
• client: <flink-home>/lib/fluss-<setup>, or <flink-home>/usrlib/fluss-lake-tiering-service for lake-tiering-service. Without --flink-home or $FLINK_HOME, --output-dir is required.

The installer checks the options before it changes anything. A missing option, an empty or unknown value, a repeated option, or an option the setup does not declare fails the install with an error naming the problem. For a missing option or an empty or unknown value, the error lists the valid values; for an option the setup does not declare, it lists the setup's options, or prints the usage when the setup takes none. An existing install stays in place even with --force.

A successful install writes a .fluss-setup file into the output directory, recording how it was installed:

1setup=<SETUP_NAME>
2side=<fluss|client>
3options=<OPTION>=<VALUE>,...
4profiles=<PROFILES>
5fluss_version=1.0.0-vv-3
6installed_at=<UTC_TIMESTAMP>

The file is not a JAR, so neither the Fluss plugin loader nor a Flink classpath picks it up.

Troubleshooting

Init container fails or hangs

Read the init container's logs:

1kubectl logs -n <NAMESPACE> <POD_NAME> -c install-plugins

Common causes:

• Egress blocked. Over HTTPS, the installer must reach ververica.jfrog.io for the Fluss JARs and repo.maven.apache.org, Maven Central, for their dependencies and for the Maven wrapper's own download of Maven. If your cluster cannot reach ververica.jfrog.io, use a private Maven repository; if it cannot reach repo.maven.apache.org, pre-bake the setups.
• Settings file not found. Check that the Secret exists in the release namespace, and that the mount path and the --settings path are the same.
• Options unavailable through a mirror. For a setup that takes options, the mirror must serve maven-help-plugin; the error names it.
• Missing or invalid option. The error names the problem and, for a missing option or a bad value, lists the valid values. See Available Setups.
• Image tag does not match the deployment. The installer installs the Fluss version of the image it runs from. Use the deployment's image tag in the init container.

Setup directory not populated after pod start

List the plugin directories on a server pod:

1kubectl exec -n <NAMESPACE> <POD_NAME> -- ls /opt/fluss/plugins/

Each installed setup appears as its plugin directory. If one is missing, check that the server container's extraVolumeMounts entry for it ends its mountPath with the setup's plugin directory and uses it as the subPath, on both coordinator and tablet. The image already carries the plugin directories of metrics-prometheus, fs-s3, lake-iceberg, and lake-paimon, so for these setups a wrong mount leaves the image's directory in place: check for the setup's .fluss-setup stamp instead, as in Check what was installed. Only an installed directory carries one.

Output directory already exists

Without --force, the installer refuses to install into an existing directory, for example when an init container is retried. Always pass --force in init container commands.

Check what was installed

Read the install stamp of a setup on a server pod:

1kubectl exec -n <NAMESPACE> <POD_NAME> -- cat /opt/fluss/plugins/<PLUGIN_DIR>/.fluss-setup

It shows the setup, its options, and the Fluss version installed.

Further Reading

• Deploying Fluss on Kubernetes: the cluster these setups install into
• Obtaining Registry Access: credentials to pull the image the installer runs from
• Configuring Remote Storage: configuring the storage backends whose plugins you install here
• Reading and Writing Fluss: the client setups a job needs to write, read, or read tiered tables
• Running Fluss Jobs on Ververica Platform 3: running a job with the client JARs on Ververica Platform 3
• Fluss Helm Chart Documentation: chart values