Version: 2.14

Configuration

This section contains example configurations for Ververica Platform. The configuration can be passed to Ververica Platform during the installation with helm via the values.yaml file under the vvp key.

Persistence Configuration

note

Remote RDBMS persistence is only available in Ververica Platform Stream Edition and above.

caution

Ververica Platform persists its metadata using JDBC, either in a remote RDBMS or locally using SQLite.

note

This refers to data owned by the Platform itself and accessed via its API such as Namespaces, Deployments, Jobs, and Savepoint metadata, but does not include artifacts, Apache Flink® checkpoint and savepoint data etc.

Currently, the following remote RDBMSs are supported:

MariaDB/MySQL Persistence
PostgreSQL Persistence
Microsoft SQL Server Persistence

The simplest mode of operation is using the preset configuration vvp.persistence.type: local which uses a Kubernetes PVC to store a SQLite database.

note

This preset also configures SQLite with some important settings. If you use SQLite with vvp.persistence.type: jdbc, we recommend using the following JDBC URL parameters: journal_mode=WAL&synchronous=FULL&busy_timeout=10000

caution

Please do not use a network-backed filesystem for backing your SQLite database. There is no guarantee that in the case of a network partition, the data in SQLite is not corrupted. We recommend using a dedicated remote RDBMS for production workloads.

Instead, if you wish to use a remote datastore, specify vvp.persistence.type: jdbc and provide an appropriate Spring datasource configuration under vvp.persistence.datasource.

MariaDB/MySQL Persistence

important

Ververica Platform supports MySQL persistence using the MySQL-compatible MariaDB JDBC connector. Therefore, if you wish to use MySQL with Ververica Platform, you must use mariadb in your JDBC connection URL.

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:mariadb://mysql.internal:3306/vvp
      username: vvp
      password: password

PostgreSQL Persistence

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:postgresql://postgresql.internal:5432/vvp
      username: vvp
      password: password

Microsoft SQL Server Persistence

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:sqlserver://mssql.internal;databaseName=vvp
      username: vvp
      password: password

Configuring Offline DDL Updates

Ververica Platform uses the Liquibase database schema change management tool to run database updates.

If your environment disallows automatic DDL updates, disable Liquibase at startup and configure Ververica Platform to generate an offline changeset against the target database. Ververica Platform generates the changeset and creates or updates the database.

To disable Liquibase at startup, please add the following configuration to your values.yaml configuration file:

env:
  - name: LIQUIBASE_ENABLED
    value: false

To configure Ververica Platform to generate an offline changeset and create or update the target database, run the following command:

docker run --rm <vvp-image-registry>/vvp-persistence:<vvp-version> --url="<JDBC-connection-string>" --username=<your-db-user-name> --password=<your-db-password>

where the parameters are:

--url The database URL, a JDBC connection string using the following syntax: --url="jdbc-format-dependent-on-db" -- please see examples below
--username The username for database access: --username=db-user-name
--password The password for database access: --password=db-pass

The database URL is a JDBC connection string which varies depending on the database. Some common examples are:

MS SQL Server: jdbc:sqlserver://<IP OR HOSTNAME>:1433;database=<DATABASE>
PostgreSQL: jdbc:postgresql://<IP OR HOSTNAME>:5432/<DATABASE>?currentSchema=<SCHEMA NAME>
MySQL: jdbc:mysql://<IP OR HOSTNAME>:3306/<DATABASE>
SQLite: jdbc:sqlite:/tmp/<DB FILE NAME>.db

note

The Liquibase documentation includes useful additional information including details of additional parameters that can be passed as a Docker running command and JDBC connection strings for a number of popular databases.

Authentication Configuration

Bootstrap Token

Ververica Platform supports a "bootstrap token", specified during installation or upgrade, which can be used as an API token with administrator privileges. This is useful for performing certain bootstrapping tasks such as creating an initial Namespace and assigning its members.

The token can be any non-empty string and is set by assigning vvp.auth.bootstrapToken.token in a Helm values file or on the command line.

For example, if you install or upgrade Ververica Platform using Helm and include the CLI option: --set vvp.auth.bootstrapToken.token=my-secret-token, you (or an automated task) could create a Namespace by running:

curl \
  -X POST \
  -H 'Authorization: Bearer my-secret-token' \
  -H 'Content-Type: application/json' \
  https://vvp.internal/namespaces/v1/namespaces \
  -d '
{
  "name": "namespaces/my-namespace",
  "roleBindings": [{
    "role": "owner",
    "members": ["group:vvp-users"]
  }]
}
'

Google Authentication Configuration Example

(Scope under vvp.auth)

# Google auth does not support groups, so administrators must be specified manually in this
# list, or with an environment variable: vvp.admins=user:admin1@example.com,admin2@example.com
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
  #groupsClaim:  # Google auth does not support groups
  registrationId: google
  registration:
    clientId: 1009242745340-9piji4g84vkrzbp2qyp19asrk8p2ug2s.apps.googleusercontent.com
    clientSecret: 4wHQZc_KHN0u8QqgpmV6TY86
  provider:
    userNameAttribute: email  # Required to correctly identify users

Azure Authentication Configuration Example

(Scope under vvp.auth)

admins:
  - user:admin1@example.com
  - user:admin2@example.com
oidc:
  # NOTE: see the following Azure doc on how to enable groupsClaim at the Azure side:
  # https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims
  groupsClaim: groups
  # NOTE: Your Azure application needs a redirect URI of <baseUrl>/login/oauth2/code/vvp
  registrationId: vvp
  registration:
    clientId: xxxxxx-your-client-id-xxxxxx
    clientSecret: xxxxxx-your-client-secret-xxxxxx
    redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
    clientAuthenticationMethod: client_secret_basic
    authorizationGrantType: authorization_code
    scope:
      - openid
      - profile
  provider:
    # Let Spring Boot figure out parameters itself from
    # https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0/.well-known/openid-configuration
    # Note: External users may not be able to sign in if you use the 'common'
    #       tenant ID. Instead, find your Azure AD's tenant ID and use that.
    issuerUri: https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0  # No trailing slash!
    # make sure, spring-boot does not fetch user info
    # see https://github.com/spring-projects/spring-security/issues/7679
    userInfoUri:
    userNameAttribute: preferred_username  # Required to correctly identify users

Amazon AWS Authentication Configuration Example

(Scope under vvp.auth)

# This uses AWS Cognito User Pool as an identity provider. You need to create a user pool, 
# create and configure an app client in the user pool, and create a Cognito domain.
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
  groupsClaim:  cognito:groups
  registrationId: vvp
  registration:
    clientId: <ClientId>
    clientSecret: <ClientSecret>
    redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
    clientAuthenticationMethod: client_secret_basic
    authorizationGrantType: authorization_code
    scope:
      - openid
      - profile
  provider:
    issuerUri: https://cognito-idp.<AWSRegion>.amazonaws.com/<UserPoolID>
    userNameAttribute: username
    # LogoutURL is the logout URL of the created app client in the User Pool
    endSessionEndpoint: "https://<UserPoolDomainPrefix>.auth.<AWSRegion>.amazoncognito.com/logout?client_id=<ClientId>&logout_uri=<LogoutURL>"

External License Configuration Example

Ververica Platform supports the usage of an external license which does not need to be part of your values.yaml.

First, create a Secret in the same namespace as your Ververica Platform installation.

apiVersion: v1
kind: Secret
metadata:
  name: vvp-license
  namespace: vvp
type: Opaque
stringData:
  license.yaml: |-
    vvp:
      license:
        data: <your-license>

Once the Secret has been created you have to add the following entries in your values.yaml file without the need to specify vvp.license.data.

licenseConfigPath: file:/etc/vvp-licenses/license.yaml

# This volume is added to the Ververica Platform Pod
volumes:
  - name: license
    secret:
      secretName: vvp-license

# The added volume is mounted to the respective Ververica Platform containers
volumeMounts:
  - name: license
    mountPath: /etc/vvp-licenses/

note

It is important to follow this schema <https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.files>__ for the path provided via licenseConfigPath.

Custom Pod Annotations

Custom annotations can be added to the Ververica Platform pod by adding them to the values.yaml file as shown in the example below.

# Annotations 'inject-sidecar' and 'environment', with values 'enabled' and 'test', are added to the Ververica Platform pod
podAnnotations:
  inject-sidecar: enabled
  environment: test

Full Example Configuration

This full example demonstrates most of the options needed to configure Ververica Platform for your environment.

vvp:
  auth:
    enabled: true

    admins:
    - group:vvp-admins  # The OIDC-supplied group which indicates an administrator

    bootstrapToken:
      token: dmVydmVyaWNhLmNvbS9jYXJlZXJz

    oidc:
      groupsClaim: roles  # The OIDC ID token claim containing a list of a user's groups
      registrationId: my-oidc-provider
      registration:
        clientId: vvp
        clientSecret: secret
        redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
        clientAuthenticationMethod: client_secret_basic
        authorizationGrantType: authorization_code
        scope:
        - openid
      provider:
        authorizationUri: http://my-oidc-provider.internal/openid-connect/auth
        tokenUri: http://my-oidc-provider.internal/openid-connect/token
        userInfoUri: http://my-oidc-provider.internal/openid-connect/userinfo
        jwkSetUri: http://my-oidc-provider.internal/openid-connect/certs
        userNameAttribute: email  # Required to correctly identify users
      endSessionEndpoint: http://my-oidc-provider.internal/openid-connect/logout

  persistence:
    type: jdbc
    datasource:
      url: jdbc:postgresql://postgresql.internal:5432/vvp
      username: vvp
      password: password

  blobStorage:
    baseUri: s3://my-bucket/vvp

  # Add additional custom Flink images to the UI, optionally setting them as the default image
  # for a particular Flink minor version
  flinkVersionMetadata:
  - flinkVersion: 1.9.0  # The full Flink version this image supplies
    imageTag: 1.9.0-custom1  # The Docker image tag for the Flink repository specified below
    defaultFor:
    - 1.9  # Make this the default image for deployments on Flink 1.9

  flinkDeploymentDefaults:
    registry: my-custom-registry.internal/vvp
    repository: flink

  license:
    # Ververica Platform License (www.ververica.com/enterprise-trial)
    data: {
      "kind": "License",
      "apiVersion": "v1",
      "metadata": {
        "id": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
        "createdAt": "2020-02-21T12:56:52.407899Z",
        "annotations": {
          "signature": "<omitted>",
          "licenseSpec": "ewogICJsaWNlbnNlSWQiIDogIjUzYjhjZjIyLTFhZjItNDRiZC1hN2JhLTc0MjA0MThmNjU3MiIsCiAgImxpY2Vuc2VkVG8iIDogInRlc3QiLAogICJleHBpcmVzIiA6ICIyMDIwLTAzLTIyVDEyOjU2OjUxLjg3MzU1M1oiLAogICJwYXJhbXMiIDogewogICAgInF1b3RhLnR5cGUiIDogIlVOTElNSVRFRCIsCiAgICAidHJpYWwiIDogInRydWUiCiAgfQp9"
        }
      },
      "spec": {
        "licenseId": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
        "licensedTo": "My Company Inc.",
        "expires": "2020-03-22T12:56:51.873553Z",
        "params": {
          "quota.type": "UNLIMITED",
          "trial": "true"
        }
      }
    }

Persistence Configuration​

MariaDB/MySQL Persistence​

PostgreSQL Persistence​

Microsoft SQL Server Persistence​

Configuring Offline DDL Updates​

Authentication Configuration​

Bootstrap Token​

Google Authentication Configuration Example​

Azure Authentication Configuration Example​

Amazon AWS Authentication Configuration Example​

External License Configuration Example​

Custom Pod Annotations​

Full Example Configuration​

Persistence Configuration

MariaDB/MySQL Persistence

PostgreSQL Persistence

Microsoft SQL Server Persistence

Configuring Offline DDL Updates

Authentication Configuration

Bootstrap Token

Google Authentication Configuration Example

Azure Authentication Configuration Example

Amazon AWS Authentication Configuration Example

External License Configuration Example

Custom Pod Annotations

Full Example Configuration