Ververica DocsVerverica Docs
Docs Home

Running Apache Fluss™ Jobs on Ververica Platform 3

7 min read

Upload client JARs, create secret values, and run an Apache Fluss job on Ververica Platform 3 as a JAR or SQL deployment or a Debug query.

On this page

Applies to: Ververica Platform 3, with client setups from Docker image registry.ververica.cloud/platform-images/fluss:1.0.0-vv-3

Overview

This manual covers running a Fluss job on Ververica Platform 3: uploading the job's JARs as artifacts, creating secret values for its credentials, preparing deployment targets and session clusters, and running the job as a JAR deployment, a SQL deployment, or a query in Debug mode. Each job's own manual says what the job contains: its SQL, its job arguments, and the setups it needs.

Prerequisites

  • A Ververica Platform 3 workspace and namespace, and access to the Ververica Platform 3 console. See Ververica Platform 3 Getting Started.
  • The client JARs of every setup the job needs, installed into a local directory. See Installing Fluss Setups.
  • Network access from the Kubernetes namespace of the job's deployment target to the Fluss bootstrap address.
  • Optionally, vvctl, the Ververica CLI, installed and configured for that namespace. See the Ververica CLI documentation. Every step below can also be done in the console.

Uploading the JARs as Artifacts

Ververica Platform 3 deployments and drafts reference JARs uploaded to the namespace's artifact store. Upload each JAR once, by its full file name. For example, with the client JARs of your setups installed under <LOCAL_DIR>, one directory per setup (see Installing Fluss Setups), upload all of them:

BASH
1for jar in <LOCAL_DIR>/*/*.jar; do
2  vvctl create artifact --filename "$jar" \
3    --workspace <WORKSPACE> --namespace <NAMESPACE>
4done

The loop uploads the Fluss Flink connector JAR, fluss-flink-<FLINK_VERSION>-1.0.0-vv-3.jar, too. Upload it even though the VERA engine ships its own Fluss connector: the one a setup installs comes with your Fluss release, so it might be more recent than the engine's, and the engine's built-in Fluss catalog rejects some of the lake credential options, such as iceberg.token.

Each command prints the artifact with its URI, s3i://<VVP_BUCKET>/artifacts/namespaces/<NAMESPACE>/<JAR_FILE>. The console's Artifacts page uploads JARs too; see the Ververica Platform 3 artifacts documentation.

The Artifacts page lists the uploaded JARs, for example:

Artifacts page listing uploaded JARs

The console's JAR URI and Additional Dependencies fields offer them by name, so you pick them from a list rather than typing the URI.

Creating Secret Values

A secret value keeps a credential, such as a SASL password, out of the job's SQL and arguments. Create one for each credential the job references:

BASH
1vvctl create secret <SECRET_NAME> <SECRET_VALUE>

To keep the value off the command line, create the secret value in the console instead, under Configurations -> Secret Values; see the Ververica Platform 3 secret values documentation.

Configurations -> Secret Values lists the secret values by name; the console never shows their values, for example:

Secret Values list showing the secret names

A job reads a secret value as ${secret_values.<SECRET_NAME>}, in SQL:

SQL
1'client.security.sasl.password' = '${secret_values.<SECRET_NAME>}'

and in a JAR deployment's main arguments, as in Running a JAR Deployment.

Deployment Targets and Session Clusters

Every deployment and session cluster runs on a deployment target. Debug queries run on a session cluster.

Creating a Deployment Target

A deployment target binds a deployment or session cluster to a Kubernetes namespace. A fresh Ververica Platform 3 namespace has none. Open Deployment Targets -> Create Deployment Target and set:

  • Deployment Target Name: <DEPLOYMENT_TARGET>
  • Kubernetes Namespace: <WORKLOADS_NAMESPACE>

Confirm with OK. The console shows a "Deployment Target created." notice, and the list shows <DEPLOYMENT_TARGET> with its Kubernetes namespace. The Ververica Platform 3 getting started guide also creates a deployment target.

Creating a Session Cluster

A session cluster stays up between queries, so Debug runs need no deployment of their own. Open Session Clusters -> Create Session Cluster and set:

  • Name: <SESSION_CLUSTER>
  • Deployment Target: <DEPLOYMENT_TARGET>
  • State: RUNNING
  • Engine Version: an engine whose Flink version is the one the installed setups were built for.

Confirm with Create Session Cluster; see the Ververica Platform 3 session clusters documentation. A session cluster has no Additional Dependencies field: a Debug run takes its JARs from the draft. The session cluster page shows Starting, then Running, for example:

Session cluster overview, Running

Running a JAR Deployment

A job whose entry point is a JAR runs as a JAR deployment. Open Deployments -> Create Deployment (hover) -> JAR Deployment and set:

  • Deployment Mode: Stream Mode
  • Deployment Name: <DEPLOYMENT_NAME>
  • Engine Version: an engine whose Flink version is the one the installed setups were built for.
  • JAR URI: pick the job's entry JAR from the list.
  • Entry Point Class: leave it empty when the JAR declares its entry point class.
  • Entry Point Main Arguments: the job arguments, with credentials as secret values.
  • Additional Dependencies: every other JAR of the setups the job needs.
  • Deployment Target: <DEPLOYMENT_TARGET>

The arguments are shown one key or value per line for reading; Ververica Platform 3 takes them joined on one line. For example, a SASL user whose password is a secret value:

TEXT
1--fluss.client.security.sasl.username
2<SASL_USERNAME>
3--fluss.client.security.sasl.password
4${secret_values.<SECRET_NAME>}

Confirm with Deploy; see the Ververica Platform 3 JAR deployment documentation. The deployment is created CANCELLED: click Start in its row, keep Initial Mode, and click Start. The deployment goes STARTING, then RUNNING, and stays there, for example:

Deployments list with a JAR deployment Running

Running a SQL Deployment

A job written in SQL runs from a draft in the SQL editor.

Creating a Draft

A streaming job runs from a draft, a batch query from a batch draft. Save As on an existing draft creates a copy that keeps its type, engine version, and Additional Dependencies.

  1. Open SQL Editor -> New (hover) -> New Blank Draft for a streaming job, or -> New Blank Batch Draft for a batch query.
  2. In the draft's Configurations panel (right rail), set Engine Version to an engine whose Flink version is the one the installed setups were built for, and select the job's JARs as Additional Dependencies.
  3. Write or paste the job's SQL.

See the Ververica Platform 3 SQL draft documentation.

Running Statements in the SQL Editor

To run statements in the editor without a deployment, select them, right-click -> Run; the Results tab prints Query has been executed. For example, a Fluss catalog with SASL credentials, and a table in it:

SQL
1CREATE CATALOG `<FLUSS_CATALOG>` WITH (
2    'type'                           = 'fluss',
3    'bootstrap.servers'              = '<BOOTSTRAP_ADDRESS>',
4    'client.security.protocol'       = 'sasl',
5    'client.security.sasl.mechanism' = 'PLAIN',
6    'client.security.sasl.username'  = '<SASL_USERNAME>',
7    'client.security.sasl.password'  = '${secret_values.<SECRET_NAME>}'
8);
9
10CREATE DATABASE IF NOT EXISTS `<FLUSS_CATALOG>`.`<DATABASE>`;
11
12CREATE TABLE IF NOT EXISTS `<FLUSS_CATALOG>`.`<DATABASE>`.`<TABLE>` (
13    `<COLUMN>` <TYPE>,
14    `<KEY_COLUMN>` <TYPE>,
15    PRIMARY KEY (`<KEY_COLUMN>`) NOT ENFORCED
16) WITH (
17    'bucket.num' = '<BUCKET_COUNT>'
18);

For example, a draft that creates a Fluss catalog, with its engine version and Additional Dependencies:

SQL draft with Engine Version and Additional Dependencies in the Configurations panel

Deploying a Draft

Click Deploy, pick Deployment Target <DEPLOYMENT_TARGET> (under Per-Job Clusters), and Confirm. A streaming deployment is created CANCELLED: on Deployments, click Start in its row, keep Initial Mode, and click Start. For a batch deployment, switch the list on Deployments to BATCH, click Start in its row, and click Start again in the "Start Job" dialog.

A streaming deployment goes STARTING, then RUNNING, and keeps running. A batch job ends FINISHED. See the Ververica Platform 3 documentation on deploying a SQL draft.

Reading a Job's Output

A job that writes to a print sink prints its rows to the TaskManager's standard output. Open the deployment, then Diagnostics -> Running Task Managers, the TaskManager, and Stdout, for example:

TaskManager Stdout of a running SQL deployment

The console keeps no logs of a finished job, so a batch job's printed rows are visible only while it runs; after it ends, check its FINISHED state. To stop a deployment, click Cancel, then OK.

Running a Query in Debug Mode

Debug mode runs a draft's query on a session cluster and shows the rows in a grid under the editor, without a deployment.

  1. On Session Clusters, check that <SESSION_CLUSTER> is RUNNING; click Start if it shows STOPPED.
  2. Open the draft and click Debug.
  3. In the Debug dialog, pick Session Cluster <SESSION_CLUSTER> and click Next.
  4. Keep Online data for the source table, and click Confirm.

The console shows a "SQL task is running" notice, then a grid under the editor that fills with rows and pauses. Click Resume (▶ left of the grid) to keep it updating, and stop the Debug job with the red square left of the grid; see the Ververica Platform 3 Debug documentation. For example:

Debug grid under the SQL editor

Troubleshooting

Session cluster is stopped

A session cluster stops after 30 idle minutes. Before a Debug run, check that it shows RUNNING on Session Clusters, and click Start if it shows STOPPED.

Debug fails with OutOfMemoryError: Metaspace

Each Debug run of a lake query loads the Iceberg JARs again, and repeated runs exhaust the TaskManager's metaspace. Restart the session cluster: on Session Clusters, click Stop, tick "I am aware and confirm", click OK, then click Start once it shows STOPPED.

Planning fails with No LakeStoragePlugin available

The draft reads a table's lake side without the lake setup's JARs. Add the lake setup's uploaded JARs to the draft's Additional Dependencies.

Further Reading

Was this helpful?