Skip to main content
Version: 2.13

Configuration

This section contains example configurations for Ververica Platform. The configuration can be passed to Ververica Platform during the installation with helm via the values.yaml file under the vvp key.

Persistence Configuration

note

Remote RDBMS persistence is only available in Ververica Platform Stream Edition and above.

caution

Ververica Platform persists its metadata using JDBC, either in a remote RDBMS or locally using SQLite.

note

This refers to data owned by the Platform itself and accessed via its API such as Namespaces, Deployments, Jobs, and Savepoint metadata, but does not include artifacts, Apache Flink® checkpoint and savepoint data etc.

Currently, the following remote RDBMSs are supported:

  • MariaDB/MySQL Persistence
  • PostgreSQL Persistence
  • Microsoft SQL Server Persistence

The simplest mode of operation is using the preset configuration vvp.persistence.type: local which uses a Kubernetes PVC to store a SQLite database.

note

This preset also configures SQLite with some important settings. If you use SQLite with vvp.persistence.type: jdbc, we recommend using the following JDBC URL parameters: journal_mode=WAL&synchronous=FULL&busy_timeout=10000

caution

Please do not use a network-backed filesystem for backing your SQLite database. There is no guarantee that in the case of a network partition, the data in SQLite is not corrupted. We recommend using a dedicated remote RDBMS for production workloads.

Instead, if you wish to use a remote datastore, specify vvp.persistence.type: jdbc and provide an appropriate Spring datasource configuration under vvp.persistence.datasource.

MariaDB/MySQL Persistence

important

Ververica Platform supports MySQL persistence using the MySQL-compatible MariaDB JDBC connector. Therefore, if you wish to use MySQL with Ververica Platform, you must use mariadb in your JDBC connection URL.

vvp:
persistence:
type: jdbc
datasource:
url: jdbc:mariadb://mysql.internal:3306/vvp
username: vvp
password: password

PostgreSQL Persistence

vvp:
persistence:
type: jdbc
datasource:
url: jdbc:postgresql://postgresql.internal:5432/vvp
username: vvp
password: password

Microsoft SQL Server Persistence

vvp:
persistence:
type: jdbc
datasource:
url: jdbc:sqlserver://mssql.internal;databaseName=vvp
username: vvp
password: password

Configuring Offline DDL Updates

Ververica Platform uses the Liquibase database schema change management tool to run database updates.

If your environment disallows automatic DDL updates, disable Liquibase at startup and configure Ververica Platform to generate an offline changeset against the target database. Ververica Platform generates the changeset and creates or updates the database.

To disable Liquibase at startup, please add the following configuration to your values.yaml configuration file:

env:
- name: LIQUIBASE_ENABLED
value: false

To configure Ververica Platform to generate an offline changeset and create or update the target database, run the following command:

docker run --rm <vvp-image-registry>/vvp-persistence:<vvp-version> --url="<JDBC-connection-string>" --username=<your-db-user-name> --password=<your-db-password>

where the parameters are:

  • --url The database URL, a JDBC connection string using the following syntax: --url="jdbc-format-dependent-on-db" -- please see examples below
  • --username The username for database access: --username=db-user-name
  • --password The password for database access: --password=db-pass

The database URL is a JDBC connection string which varies depending on the database. Some common examples are:

  • MS SQL Server: jdbc:sqlserver://<IP OR HOSTNAME>:1433;database=<DATABASE>
  • PostgreSQL: jdbc:postgresql://<IP OR HOSTNAME>:5432/<DATABASE>?currentSchema=<SCHEMA NAME>
  • MySQL: jdbc:mysql://<IP OR HOSTNAME>:3306/<DATABASE>
  • SQLite: jdbc:sqlite:/tmp/<DB FILE NAME>.db
note

The Liquibase documentation includes useful additional information including details of additional parameters that can be passed as a Docker running command and JDBC connection strings for a number of popular databases.

Authentication Configuration

Bootstrap Token

Ververica Platform supports a "bootstrap token", specified during installation or upgrade, which can be used as an API token with administrator privileges. This is useful for performing certain bootstrapping tasks such as creating an initial Namespace and assigning its members.

The token can be any non-empty string and is set by assigning vvp.auth.bootstrapToken.token in a Helm values file or on the command line.

For example, if you install or upgrade Ververica Platform using Helm and include the CLI option: --set vvp.auth.bootstrapToken.token=my-secret-token, you (or an automated task) could create a Namespace by running:

curl \
-X POST \
-H 'Authorization: Bearer my-secret-token' \
-H 'Content-Type: application/json' \
https://vvp.internal/namespaces/v1/namespaces \
-d '
{
"name": "namespaces/my-namespace",
"roleBindings": [{
"role": "owner",
"members": ["group:vvp-users"]
}]
}
'

Google Authentication Configuration Example

(Scope under vvp.auth)

# Google auth does not support groups, so administrators must be specified manually in this
# list, or with an environment variable: vvp.admins=user:admin1@example.com,admin2@example.com
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
#groupsClaim: # Google auth does not support groups
registrationId: google
registration:
clientId: 1009242745340-9piji4g84vkrzbp2qyp19asrk8p2ug2s.apps.googleusercontent.com
clientSecret: 4wHQZc_KHN0u8QqgpmV6TY86
provider:
userNameAttribute: email # Required to correctly identify users

Azure Authentication Configuration Example

(Scope under vvp.auth)

admins:
- user:admin1@example.com
- user:admin2@example.com
oidc:
# NOTE: see the following Azure doc on how to enable groupsClaim at the Azure side:
# https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims
groupsClaim: groups
# NOTE: Your Azure application needs a redirect URI of <baseUrl>/login/oauth2/code/vvp
registrationId: vvp
registration:
clientId: xxxxxx-your-client-id-xxxxxx
clientSecret: xxxxxx-your-client-secret-xxxxxx
redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
clientAuthenticationMethod: client_secret_basic
authorizationGrantType: authorization_code
scope:
- openid
- profile
provider:
# Let Spring Boot figure out parameters itself from
# https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0/.well-known/openid-configuration
# Note: External users may not be able to sign in if you use the 'common'
# tenant ID. Instead, find your Azure AD's tenant ID and use that.
issuerUri: https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0 # No trailing slash!
# make sure, spring-boot does not fetch user info
# see https://github.com/spring-projects/spring-security/issues/7679
userInfoUri:
userNameAttribute: preferred_username # Required to correctly identify users

Amazon AWS Authentication Configuration Example

(Scope under vvp.auth)

# This uses AWS Cognito User Pool as an identity provider. You need to create a user pool, 
# create and configure an app client in the user pool, and create a Cognito domain.
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
groupsClaim: cognito:groups
registrationId: vvp
registration:
clientId: <ClientId>
clientSecret: <ClientSecret>
redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
clientAuthenticationMethod: client_secret_basic
authorizationGrantType: authorization_code
scope:
- openid
- profile
provider:
issuerUri: https://cognito-idp.<AWSRegion>.amazonaws.com/<UserPoolID>
userNameAttribute: username
# LogoutURL is the logout URL of the created app client in the User Pool
endSessionEndpoint: "https://<UserPoolDomainPrefix>.auth.<AWSRegion>.amazoncognito.com/logout?client_id=<ClientId>&logout_uri=<LogoutURL>"

External License Configuration Example

Ververica Platform supports the usage of an external license which does not need to be part of your values.yaml.

First, create a Secret in the same namespace as your Ververica Platform installation.

apiVersion: v1
kind: Secret
metadata:
name: vvp-license
namespace: vvp
type: Opaque
stringData:
license.yaml: |-
vvp:
license:
data: <your-license>

Once the Secret has been created you have to add the following entries in your values.yaml file without the need to specify vvp.license.data.

licenseConfigPath: file:/etc/vvp-licenses/license.yaml

# This volume is added to the Ververica Platform Pod
volumes:
- name: license
secret:
secretName: vvp-license

# The added volume is mounted to the respective Ververica Platform containers
volumeMounts:
- name: license
mountPath: /etc/vvp-licenses/
note

It is important to follow this schema <https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.files>__ for the path provided via licenseConfigPath.

Custom Pod Annotations

Custom annotations can be added to the Ververica Platform pod by adding them to the values.yaml file as shown in the example below.

# Annotations 'inject-sidecar' and 'environment', with values 'enabled' and 'test', are added to the Ververica Platform pod
podAnnotations:
inject-sidecar: enabled
environment: test

Full Example Configuration

This full example demonstrates most of the options needed to configure Ververica Platform for your environment.

vvp:
auth:
enabled: true

admins:
- group:vvp-admins # The OIDC-supplied group which indicates an administrator

bootstrapToken:
token: dmVydmVyaWNhLmNvbS9jYXJlZXJz

oidc:
groupsClaim: roles # The OIDC ID token claim containing a list of a user's groups
registrationId: my-oidc-provider
registration:
clientId: vvp
clientSecret: secret
redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
clientAuthenticationMethod: client_secret_basic
authorizationGrantType: authorization_code
scope:
- openid
provider:
authorizationUri: http://my-oidc-provider.internal/openid-connect/auth
tokenUri: http://my-oidc-provider.internal/openid-connect/token
userInfoUri: http://my-oidc-provider.internal/openid-connect/userinfo
jwkSetUri: http://my-oidc-provider.internal/openid-connect/certs
userNameAttribute: email # Required to correctly identify users
endSessionEndpoint: http://my-oidc-provider.internal/openid-connect/logout

persistence:
type: jdbc
datasource:
url: jdbc:postgresql://postgresql.internal:5432/vvp
username: vvp
password: password

blobStorage:
baseUri: s3://my-bucket/vvp

# Add additional custom Flink images to the UI, optionally setting them as the default image
# for a particular Flink minor version
flinkVersionMetadata:
- flinkVersion: 1.9.0 # The full Flink version this image supplies
imageTag: 1.9.0-custom1 # The Docker image tag for the Flink repository specified below
defaultFor:
- 1.9 # Make this the default image for deployments on Flink 1.9

flinkDeploymentDefaults:
registry: my-custom-registry.internal/vvp
repository: flink

license:
# Ververica Platform License (www.ververica.com/enterprise-trial)
data: {
"kind": "License",
"apiVersion": "v1",
"metadata": {
"id": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
"createdAt": "2020-02-21T12:56:52.407899Z",
"annotations": {
"signature": "<omitted>",
"licenseSpec": "ewogICJsaWNlbnNlSWQiIDogIjUzYjhjZjIyLTFhZjItNDRiZC1hN2JhLTc0MjA0MThmNjU3MiIsCiAgImxpY2Vuc2VkVG8iIDogInRlc3QiLAogICJleHBpcmVzIiA6ICIyMDIwLTAzLTIyVDEyOjU2OjUxLjg3MzU1M1oiLAogICJwYXJhbXMiIDogewogICAgInF1b3RhLnR5cGUiIDogIlVOTElNSVRFRCIsCiAgICAidHJpYWwiIDogInRydWUiCiAgfQp9"
}
},
"spec": {
"licenseId": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
"licensedTo": "My Company Inc.",
"expires": "2020-03-22T12:56:51.873553Z",
"params": {
"quota.type": "UNLIMITED",
"trial": "true"
}
}
}