Configuration

This section contains example configurations for Ververica Platform. The configuration can be passed to Ververica Platform during the installation with helm via the values.yaml file under the vvp key.

Persistence Configuration

Note

Remote RDBMS persistence is only available in Ververica Platform Stream Edition and above.

Ververica Platform persists its metadata[1] using JDBC, either in a remote RDBMS or locally using SQLite.

Currently, the following remote RDBMSs are supported:

The simplest mode of operation is using the preset configuraiton vvp.persistence.type: local which uses a Kubernetes PVC to store a SQLite database[2].

Warning

Please do not use a network-backed filesystem for backing your SQLite database. There is no guarantee that in the case of a network partition, the data in SQLite is not corrupted. We recommend using a dedicated remote RDBMS for production workloads.

Instead, if you wish to use a remote datastore, specify vvp.persistence.type: jdbc and provide an appropriate Spring datasource configuration under vvp.persistence.datasource.

MariaDB/MySQL Persistence

Important

Ververica Platform supports MySQL persistence using the MySQL-compatible MariaDB JDBC connector. Therefore, if you wish to use MySQL with Ververica Platform, you must use mariadb in your JDBC connection URL.

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:mariadb://mysql.internal:3306/vvp
      username: vvp
      password: password

PostgreSQL Persistence

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:postgresql://postgresql.internal:5432/vvp
      username: vvp
      password: password

Microsoft SQL Server Persistence

vvp:
  persistence:
    type: jdbc
    datasource:
      url: jdbc:sqlserver://mssql.internal;databaseName=vvp
      username: vvp
      password: password
[1]This refers to data owned by the Platform itself and accessed via its API such as Namespaces, Deployments, Jobs, and Savepoint metadata, but does not include artifacts, Apache Flink® checkpoint and savepoint data etc.
[2]This preset also configures SQLite with some important settings. If you use SQLite with vvp.persistence.type: jdbc, we recommend using the following JDBC URL parameters: journal_mode=WAL&synchronous=FULL&busy_timeout=10000

Authentication Configuration

Bootstrap Token

Ververica Platform supports a “bootstrap token”, specified during installation or upgrade, which can be used as an API token with administrator privileges. This is useful for performing certain bootstrapping tasks such as creating an initial Namespace and assigning its members.

The token can be any non-empty string and is set by assigning vvp.auth.bootstrapToken.token in a Helm values file or on the command line.

For example, if you install or upgrade Ververica Platform using Helm and include the CLI option: --set vvp.auth.bootstrapToken.token=my-secret-token, you (or an automated task) could create a Namespace by running:

$ curl \
    -X POST \
    -H 'Authorization: Bearer my-secret-token' \
    -H 'Content-Type: application/json' \
    https://vvp.internal/namespaces/v1/namespaces \
    -d '
  {
    "name": "namespaces/my-namespace",
    "roleBindings": [{
      "role": "owner",
      "members": ["group:vvp-users"]
    }]
  }
  '

Google Authentication Configuration Example

(Scope under vvp.auth)

# Google auth does not support groups, so administrators must be specified manually in this
# list, or with an environment variable: vvp.admins=user:admin1@example.com,admin2@example.com
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
  #groupsClaim:  # Google auth does not support groups
  registrationId: google
  registration:
    clientId: 1009242745340-9piji4g84vkrzbp2qyp19asrk8p2ug2s.apps.googleusercontent.com
    clientSecret: 4wHQZc_KHN0u8QqgpmV6TY86
  provider:
    userNameAttribute: email  # Required to correctly identify users

Azure Authentication Configuration Example

(Scope under vvp.auth)

admins:
  - user:admin1@example.com
  - user:admin2@example.com
oidc:
  # NOTE: see the following Azure doc on how to enable groupsClaim at the Azure side:
  # https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims
  groupsClaim: groups
  # NOTE: Your Azure application needs a redirect URI of <baseUrl>/login/oauth2/code/vvp
  registrationId: vvp
  registration:
    clientId: xxxxxx-your-client-id-xxxxxx
    clientSecret: xxxxxx-your-client-secret-xxxxxx
    redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
    clientAuthenticationMethod: basic
    authorizationGrantType: authorization_code
    scope:
      - openid
      - profile
  provider:
    # Let Spring Boot figure out parameters itself from
    # https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0/.well-known/openid-configuration
    # Note: External users may not be able to sign in if you use the 'common'
    #       tenant ID. Instead, find your Azure AD's tenant ID and use that.
    issuerUri: https://login.microsoftonline.com/xxxxxx-your-tenant-id-xxxxxx/v2.0  # No trailing slash!
    # make sure, spring-boot does not fetch user info
    # see https://github.com/spring-projects/spring-security/issues/7679
    userInfoUri:
    userNameAttribute: preferred_username  # Required to correctly identify users

Amazon AWS Authentication Configuration Example

(Scope under vvp.auth)

# This uses AWS Cognito User Pool as an identity provider. You need to create a user pool, 
# create and configure an app client in the user pool, and create a Cognito domain.
admins:
- user:admin1@example.com
- user:admin2@example.com

oidc:
  groupsClaim:  cognito:groups
  registrationId: vvp
  registration:
    clientId: <ClientId>
    clientSecret: <ClientSecret>
    redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
    clientAuthenticationMethod: basic
    authorizationGrantType: authorization_code
    scope:
      - openid
      - profile
  provider:
    issuerUri: https://cognito-idp.<AWSRegion>.amazonaws.com/<UserPoolID>
    userNameAttribute: username
    # LogoutURL is the logout URL of the created app client in the User Pool
    endSessionEndpoint: "https://<UserPoolDomainPrefix>.auth.<AWSRegion>.amazoncognito.com/logout?client_id=<ClientId>&logout_uri=<LogoutURL>"
    

Full Example Configuration

This full example demonstrates most of the options needed to configure Ververica Platform for your environment.

vvp:
  auth:
    enabled: true

    admins:
    - group:vvp-admins  # The OIDC-supplied group which indicates an administrator

    bootstrapToken:
      token: dmVydmVyaWNhLmNvbS9jYXJlZXJz

    oidc:
      groupsClaim: roles  # The OIDC ID token claim containing a list of a user's groups
      registrationId: my-oidc-provider
      registration:
        clientId: vvp
        clientSecret: secret
        redirectUri: "{baseUrl}/{action}/oauth2/code/{registrationId}"
        clientAuthenticationMethod: basic
        authorizationGrantType: authorization_code
        scope:
        - openid
      provider:
        authorizationUri: http://my-oidc-provider.internal/openid-connect/auth
        tokenUri: http://my-oidc-provider.internal/openid-connect/token
        userInfoUri: http://my-oidc-provider.internal/openid-connect/userinfo
        jwkSetUri: http://my-oidc-provider.internal/openid-connect/certs
        userNameAttribute: email  # Required to correctly identify users
      endSessionEndpoint: http://my-oidc-provider.internal/openid-connect/logout

  persistence:
    type: jdbc
    datasource:
      url: jdbc:postgresql://postgresql.internal:5432/vvp
      username: vvp
      password: password

  blobStorage:
    baseUri: s3://my-bucket/vvp

  # Add additional custom Flink images to the UI, optionally setting them as the default image
  # for a particular Flink minor version
  flinkVersionMetadata:
  - flinkVersion: 1.9.0  # The full Flink version this image supplies
    imageTag: 1.9.0-custom1  # The Docker image tag for the Flink repository specified below
    defaultFor:
    - 1.9  # Make this the default image for deployments on Flink 1.9

  flinkDeploymentDefaults:
    registry: my-custom-registry.internal/vvp
    repository: flink

  license:
    # Ververica Platform License (www.ververica.com/enterprise-trial)
    data: {
      "kind": "License",
      "apiVersion": "v1",
      "metadata": {
        "id": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
        "createdAt": "2020-02-21T12:56:52.407899Z",
        "annotations": {
          "signature": "<omitted>",
          "licenseSpec": "ewogICJsaWNlbnNlSWQiIDogIjUzYjhjZjIyLTFhZjItNDRiZC1hN2JhLTc0MjA0MThmNjU3MiIsCiAgImxpY2Vuc2VkVG8iIDogInRlc3QiLAogICJleHBpcmVzIiA6ICIyMDIwLTAzLTIyVDEyOjU2OjUxLjg3MzU1M1oiLAogICJwYXJhbXMiIDogewogICAgInF1b3RhLnR5cGUiIDogIlVOTElNSVRFRCIsCiAgICAidHJpYWwiIDogInRydWUiCiAgfQp9"
        }
      },
      "spec": {
        "licenseId": "53b8cf22-1af2-44bd-a7ba-7420418f6572",
        "licensedTo": "My Company Inc.",
        "expires": "2020-03-22T12:56:51.873553Z",
        "params": {
          "quota.type": "UNLIMITED",
          "trial": "true"
        }
      }
    }