Configuring Remote Storage

Applies to: fluss-bundle chart, Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

Remote storage backs an Apache Fluss™ cluster with durable object storage. By default the chart keeps the remote data directory under an in-pod path (remote.data.dir: /tmp/fluss/remote-data), which does not survive the pod. Each backend needs two things on the Fluss servers: its storage keys in fluss.configurationOverrides, and its filesystem setup installed on both the coordinator and the tablet servers.

Prerequisites

• A running Fluss cluster deployed with the fluss-bundle chart. See Deploying Fluss on Kubernetes.
• A storage location the Kubernetes cluster can reach: an S3 bucket, an ADLS Gen2 container, or a bucket on an S3-compatible store.
• The credentials of the chosen backend: an access key pair for S3 and S3-compatible stores; for ADLS Gen2, the storage account key and a service principal.

Supported Backends

The scheme of remote.data.dir must match the installed filesystem setup.

Installing the Filesystem Plugin

Install the backend's filesystem setup, fs-s3 or fs-azure, on both the coordinator and the tablet servers; Supported Backends gives each backend's setup and plugin directory. An init container or a custom image built with docker build both work: see Installing Fluss Setups.

Storing Credentials in a Secret

Every backend section below keeps its credentials in a Kubernetes Secret in the release namespace. The values file references each key through a config-provider marker, ${env:<NAME>}, backed by a fluss.secrets.env entry. For the markers themselves and their mount form, see Config Providers.

The env names, the Secret name, and its keys are yours to choose; every section uses the Secret fluss-storage-credentials.

Amazon S3

The server authenticates to S3 with an IAM user's access key. The tablet servers mint short-lived delegation tokens for clients, so clients need no S3 credentials of their own.

Cloud infrastructure requirements

You will need:

• An S3 bucket with public access blocked.
• An IAM user with an access key, and the policy below attached.

Attach this policy to the IAM user:

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
7      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
8    },
9    {
10      "Effect": "Allow",
11      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
12      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
13    }
14  ]
15}

See the AWS S3 documentation and the IAM documentation for setup instructions.

Helm values

Create the Secret from a key=value file, so that no credential appears on the command line. The file holds the IAM user's access key:

1s3-access-key=<AWS_ACCESS_KEY_ID>
2s3-secret-key=<AWS_SECRET_ACCESS_KEY>

Then reference it from values.yaml, next to the setup installation (see Installing the Filesystem Plugin):

1fluss:
2  configurationOverrides:
3    remote.data.dir: s3://<BUCKET_NAME>/<PREFIX>
4    s3.access.key: "${env:FLUSS_S3_ACCESS_KEY}"
5    s3.secret.key: "${env:FLUSS_S3_SECRET_KEY}"
6    s3.region: <AWS_REGION>
7  secrets:
8    env:
9      - name: FLUSS_S3_ACCESS_KEY
10        secretName: fluss-storage-credentials
11        key: s3-access-key
12      - name: FLUSS_S3_SECRET_KEY
13        secretName: fluss-storage-credentials
14        key: s3-secret-key

Instead of relying on the access key alone, the servers can assume an IAM role, set with s3.assumed.role.arn. This option cannot be combined with a custom s3.aws.credentials.provider.

1fluss:
2  configurationOverrides:
3    s3.assumed.role.arn: <ROLE_ARN>

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Azure Blob Storage (ADLS Gen2)

The server authenticates to ADLS Gen2 with the storage account key and a service principal. The tablet servers use the service principal to mint short-lived delegation tokens for clients, as on Amazon S3, so clients need no ABS credential of their own. The account key stays configured: it is the credential the servers themselves use for storage.

Cloud infrastructure requirements

You will need:

• An Azure storage account with hierarchical namespace enabled, which the abfs:// scheme requires.
• An ADLS Gen2 container (filesystem) in the storage account.
• The storage account access key.
• A service principal with the Storage Blob Data Contributor role on the storage account, and its client ID, client secret, and tenant ID.

The service principal's OAuth token endpoint has the form:

1https://login.microsoftonline.com/<TENANT_ID>/oauth2/token

See the Azure ADLS Gen2 documentation and the service principal documentation for setup instructions.

Helm values

The remote data directory has the form abfs://<CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows.net/<PATH>.

Create the Secret from a key=value file, so that no credential appears on the command line. The file holds the account key and the service principal's client secret:

1fs-azure-account-key=<STORAGE_ACCOUNT_KEY>
2fs-azure-account-oauth2-client-secret=<SP_CLIENT_SECRET>

Then reference it from values.yaml, next to the fs-azure plugin installation:

1fluss:
2  configurationOverrides:
3    remote.data.dir: abfs://<CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows.net/<PATH>
4    fs.azure.account.key: "${env:FLUSS_FS_AZURE_ACCOUNT_KEY}"
5    fs.azure.account.oauth.provider.type: org.apache.fluss.fs.azure.token.DynamicTemporaryAzureCredentialsProvider
6    fs.azure.account.oauth2.client.id: <SP_CLIENT_ID>
7    fs.azure.account.oauth2.client.secret: "${env:FLUSS_FS_AZURE_ACCOUNT_OAUTH2_CLIENT_SECRET}"
8    fs.azure.account.oauth2.client.endpoint: https://login.microsoftonline.com/<TENANT_ID>/oauth2/token
9  secrets:
10    env:
11      - name: FLUSS_FS_AZURE_ACCOUNT_KEY
12        secretName: fluss-storage-credentials
13        key: fs-azure-account-key
14      - name: FLUSS_FS_AZURE_ACCOUNT_OAUTH2_CLIENT_SECRET
15        secretName: fluss-storage-credentials
16        key: fs-azure-account-oauth2-client-secret

Without a service principal, the server runs on the account key alone, and every client then needs the account key in its own configuration. See Reading and Writing Fluss.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

OpenShift Data Foundation (ODF) and other S3-compatible stores

OpenShift Data Foundation serves S3 through its Multicloud Object Gateway. Fluss uses ODF's S3 endpoint as remote storage with the fs-s3 setup. Any other S3-compatible store reachable from the cluster (MinIO, Ceph RGW, and others) is configured the same way, with its own endpoint and credentials.

Clients need storage credentials of their own. An S3-compatible store without an STS endpoint distributes no delegation tokens, so Flink and SDK clients must be given storage credentials on their side; without them, reads return zero rows or fail with a missing-credentials error. See Reading and Writing Fluss before deploying.

Cloud infrastructure requirements

You will need:

• OpenShift Data Foundation installed in the cluster. See the Red Hat OpenShift Data Foundation documentation.
• An ObjectBucketClaim, requesting a bucket. The claim creates two objects of the same name:
  • a Secret with the keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY;
  • a ConfigMap with BUCKET_NAME, BUCKET_HOST, and BUCKET_PORT.

ODF's S3 endpoint requires path-style addressing.

Helm values

An S3-compatible store adds three keys to the Amazon S3 values: s3.endpoint, the store's endpoint; s3.path-style-access: "true", for the path-style addressing the store requires; and s3.aws.credentials.provider, set as shown below. s3.region is required by the S3 client, but ODF does not enforce its value; us-east-1 works.

Create the storage Secret from a key=value file holding the claim Secret's keys, so that no credential appears on the command line:

1s3-access-key=<AWS_ACCESS_KEY_ID>
2s3-secret-key=<AWS_SECRET_ACCESS_KEY>

Then the values:

1fluss:
2  configurationOverrides:
3    remote.data.dir: s3://<BUCKET_NAME>/<PREFIX>
4    s3.endpoint: <S3_ENDPOINT>
5    s3.access.key: "${env:FLUSS_S3_ACCESS_KEY}"
6    s3.secret.key: "${env:FLUSS_S3_SECRET_KEY}"
7    s3.region: <REGION>
8    s3.path-style-access: "true"
9    s3.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
10  secrets:
11    env:
12      - name: FLUSS_S3_ACCESS_KEY
13        secretName: fluss-storage-credentials
14        key: s3-access-key
15      - name: FLUSS_S3_SECRET_KEY
16        secretName: fluss-storage-credentials
17        key: s3-secret-key

Take the bucket name from the claim's ConfigMap (BUCKET_NAME), and the endpoint from its host and port (BUCKET_HOST, BUCKET_PORT). Install the fs-s3 setup as Installing the Filesystem Plugin describes.

For the full set of configurable fields under fluss:, refer to the Fluss Helm chart documentation.

Verify Remote Storage

After applying the values, if you install the setup with an init container, check that it completed on each server pod:

1kubectl logs -n <NAMESPACE> <POD_NAME> -c install-plugins

Check that the plugin directory, s3 or azure, is populated:

1kubectl exec -n <NAMESPACE> <POD_NAME> -- ls /opt/fluss/plugins/<PLUGIN_DIR>/

To read which setup and version were installed, see Installing Fluss Setups.

The first configuration of remote storage restarts the Fluss pods, because it changes their pod template: the init container, its volume, and the Secret entries. The Fluss servers read their configuration only at start, and a change to it alone restarts no Fluss pod. If a later change touches only configuration keys, such as another remote.data.dir or s3.region, restart the Fluss StatefulSets before the check:

1kubectl rollout restart statefulset coordinator-server tablet-server --namespace <NAMESPACE>

Check that the server configuration carries your remote data directory, on a coordinator pod and on a tablet server pod:

1kubectl exec -n <NAMESPACE> <POD_NAME> -- grep remote.data.dir /opt/fluss/conf/server.yaml

The value matches the remote.data.dir you configured. Credential keys show their markers, never the resolved values.

Troubleshooting

Init container fails

Read the init container's log:

1kubectl logs -n <NAMESPACE> <POD_NAME> -c install-plugins

For the causes and fixes, see Installing Fluss Setups.

Config-provider markers do not resolve

Pod in CreateContainerConfigError. A fluss.secrets.env entry names a Secret or key that does not exist in the namespace. kubectl describe pod -n <NAMESPACE> <POD_NAME> names the missing Secret or key. Create it, or fix secretName or key.

Server exits at startup with an IllegalConfigurationException. A marker names an env var or mount that is not declared under fluss.secrets, or is malformed. Check the server log:

1kubectl logs -n <NAMESPACE> <POD_NAME> | grep IllegalConfiguration

Every ${env:<NAME>} needs a fluss.secrets.env entry with the same name.

A key reaches the backend as the literal text ${env:...}. No fluss.secrets entry is declared, so the server has no config provider to resolve the marker. It starts and logs a warning for each such key:

1kubectl logs -n <NAMESPACE> <POD_NAME> | grep 'looks like a config provider marker'

Declare a fluss.secrets.env entry for every marker.

Azure: authentication errors

Check that the key is the storage account's access key, that it has not been rotated since it was configured, and that the account name in remote.data.dir is the account the key belongs to. Also check that the service principal's client secret has not expired or been rotated, that the tenant in fs.azure.account.oauth2.client.endpoint is the principal's own tenant, and that the principal holds Storage Blob Data Contributor on the storage account itself, not only on a container.

Further Reading

• Deploying Fluss on Kubernetes: the chart the storage values go into, and config-provider markers
• Installing Fluss Setups: installing the filesystem setups, and a private Maven repository for them
• Reading and Writing Fluss: client-side storage credentials for backends without delegation tokens
• Securing Fluss: creating Secrets from a key=value file, and the restart a changed Secret needs
• Apache Fluss Documentation: upstream reference
• Fluss Configuration Reference: configuration keys
• Fluss Helm Chart Documentation: chart values
• Creating an S3 bucket and IAM policies: AWS documentation
• Creating an ADLS Gen2 storage account and creating a service principal: Azure documentation
• Red Hat OpenShift Data Foundation Documentation