Skip to main content
Version: 2.12

Ververica Platform 2.1.0

Release Date: 2020-03-09

Changelog​

General​

  • The Getting Started Guide now builds upon the Ververica Platform Playground repository. The repository contains Helm configuration files for Ververica Platform and related components (e.g. MinIO) and serves as a blueprint for integrating Ververica Platform with third-party services such as logging and metrics systems.

  • This release adds support for storing Ververica Platform in an external datastore. For new installations, please refer to the Persistence Configuration section of the docs for information on configuring this. For information about migrating your data to an external datastore, see the Migration section below.

Helm Charts​

  • The Helm charts of Ververica Platform are no longer available as a download in the customer portal. Instead, they are now shared via a Helm charts Repository hosted under charts.ververica.com. Please see Installation using Helm for details.

  • Ververica Platform Helm charts now support Helm 3 as well as Helm 2. Support for Helm 2 will be dropped in the Ververica Platform release following the EOL of Helm 2 (November 2020).

  • Until now, Ververica Platform's Helm charts have included a set of so called demo components, a logging and a metrics stack, enabled via demoComponentsEnabled.[logging|metrics]. These demo components are no longer distributed in our Helm charts.

  • Until now, the Helm chart version has always coincided with the Ververica Platform release (so called appVersion). For Ververica Platform 2.1 onwards, Ververica Platform and its Helm charts are versioned independently. For example, for Ververica Platform 2.1, the corresponding chart version is 3.0.0.

  • Ververica Platform 2.1.0 includes our distribution of Apache Flink® 1.10.0 and Apache Flink® 1.9.2.

  • Please check Ververica Platform Docker Images for all available Apache Flink® images of this release.

  • Starting with Flink 1.10, Ververica Platform uses a dedicated classloader for user code, eliminating the risk of classloading conflicts between your application code and framework code.

  • For Flink 1.9 and Flink 1.10, we now use Flink's stop command when suspending a Deployment. The stop command performs a savepoint, atomically shuts down the Apache Flink application, and guarantees persistence to external storage systems for exactly-once sinks by committing all outstanding transactions. The resulting savepoint is thus fully self-contained. Please see FLIP-34 for details.

  • From Flink 1.10 onwards, FLIP-49 has changed the way memory is configured for Taskmanagers. This allows us to apply smaller safety margins when configuring Flink's memory pools on Kubernetes leading to better overall memory utilization, while increasing robustness at the same time.

    Specifically when using the RocksDBStatebackend, RocksDB will allocate memory outside of the JVM on your Taskamanger. This memory consumption is now controlled and limited by Flink. Therefore, you can use Flink configuration taskmanager.memory.managed.fraction to configure the fraction of memory you would like to dedicate to RocksDB.

    • taskmanager.memory.managed.fraction: 0.0 means no memory for RocksDB (applicable if you are using state.backend: filesystem)
    • taskmanager.memory.managed.fraction defaults to 0.4.

    Please see the Flink documentation for details.

Deployment Defaults​

You can now configure global default values for your Deployments in the Ververica Platform configuration. You can further configure namespaced default values for your Deployments, which are applied on top of the global default values. See Deployment Defaults for more details.

Logging Profiles​

The logging configuration (Log4J2) of your Apache Flink Deployments is now fully configurable: in the Ververica configuration you can define different Logging Profiles, which you can then choose from for each of your Deployments. Furthermore, you can customize any of the logging profiles on a Deployment basis.

Extended Job and Deployment Status​

Ververica Platform 2.1 exposes more information about the status of a Deployment and Job. Once a Job is STARTED, its status now includes the live status of the underlying Flink job as well as the observed number of job restarts. Based on this information, Ververica Platform derives a set of conditions for the RUNNING Deployment, which are exposed via the REST API and web user interface.

OpenSSL Support​

Ververica Platform now supports OpenSSL as the SSL Provider for Apache Flink® intra-cluster communication as well as Apache Flink®'s REST API out-of-the-box. Benchmarks show that openSSL has a significantly lower performance overhead than the default JDK SSL provider. Please check out SSL Provider for the details.

Web User Interface​

  • When creating or configuring a Deployment, you can now choose and switch between three different forms:

    • Standard, which only shows a minimal set of configurations
    • Advanced, which shows the full set of configurations (comparable to the form in previous versions of Ververica Platform)
    • YAML, an enhanced editor to work on the raw YAML representation of the Deployment resource

  • The "Snapshots" tab now also lists retained checkpoints of the currently running Apache Flink® job, including information like checkpoint size and duration.

Additional Features & Changes​

  • You can now configure a Deployment to add custom Kubernetes labels to all Apache Flink pods (see Labels) .

  • You can now use placeholders in your Flink Configuration (e.g. for the namespace or Deployment ID), which are resolved during Job creation time.

  • Newly created Deployment resources now have defaults for the Upgrade Strategy (Deployment.spec.upgradeStrategy) and Restore Strategy (Deployment.spec.restoreStrategy) attributes. Previously, both of these attributes were required and did not have a default. If desired, you can modify these values with the newly introduced Deployment Defaults APIs.

Migration​

Preparation​

Persistence Configuration​

Until now, Ververica Platform has always stored Platform data locally on a Persistent Volume. Ververica Platform 2.1 makes the persistence backend pluggable. When upgrading from Ververica Platform 2.0 to 2.1 you need to choose a persistence type. If you do not provide this setting, the upgrade will be cancelled and no changes made to the running Platform.

As an operator of Ververica Platform, you have three options. Please note that you must perform the migration to the pluggable persistence backend (Option 1 or Option 2) prior to upgrading to Ververica Platform 2.2.

Option 1: Migrate to an External Database

In order migrate all Platform data to the configured MariaDB/MySQL or PostgreSQL database add the following configurations to your values.yaml file prior to upgrading.

  vvp:
    persistence:
      type: legacyToJdbc
      datasource:
        url: jdbc:..
        username: <omitted>
        password: <omitted>

Please see the section on Persistence Configuration for more details.

The migration will be performed during the first startup of Ververica Platform 2.1. It is safe to keep this option afterwards, but recommended to change vvp.persistence.type to jdbc once the migration is complete.

Option 2: Keep Using a Local SQLite Database

In order migrate all Platform data to a local SQLite database file in the same data directory where the Persistent Volume is mounted add the following configuration to your values.yaml file prior to upgrading. In essence, this persistence type has the same characteristics as the legacy persistence layer (see Option 3).

  vvp:
    persistence:
      type: legacyToLocal

The migration will be performed during the first startup of Ververica Platform 2.1. It is safe to keep this option afterwards, but recommended to change vvp.persistence.type to local once the migration is complete.

The Platform cannot automatically migrate your data from local persistence to an external database. If you migrate to local, and later wish to migrate to an external database, you will have to do this manually.

Option 3: Defer the Decision

In order to defer the decision until a later time add the following configuration to your values.yaml file prior to upgrading.

  vvp:
    persistence:
      type: legacy
caution

If you configure vvp.persistence.type to jdbc or local without previously migrating your data via legacyToJdbc or legacyToLocal and you do not use a persistent volume created out-of-band, Helm will delete the persistent volume holding your metadata during the upgrade to Ververica Platform 2.1.

Minor Breaking Changes​

  • Minimum Memory Requirements: From this release onwards, Ververica Platform validates that a minimum amount of memory has been configured for all Flink processes: Taskmanagers require at least 1G of memory, Jobmanagers require at least 500m of memory. If you have an existing Deployment from before this release with a configured memory size below this threshold, you will not be able to make changes to it until you raise the memory size.

  • Deployment Patch Sets: The experimental API for Deployment patch sets has been removed from the DeploymentTarget resource. There is no action required if you were not using this API. If you were, please migrate to the newly introduced DeploymentDefaults API. Existing DeploymentTarget resources will drop the spec.deploymentPatchSet attribute and we no longer apply any JSON patches during Deployment resource creations. Requests to the DeploymentTarget API that include the dropped attribute will fail.

  • Job Logging: The Platform-wide configuration vvp.appmanager.jobLogging has been removed and is superseded by Logging Profiles and custom logging configuration templates on a per-Deployment basis. There is no action required if you were not using this configuration. If you were using the removed configuration, you can modify the default logging profile to additionally use the GELF appender and add the GELF appender to your Flink cluster class path by building a custom Docker image. For reference, the logging configuration in earlier releases used the following appender configuration (where gelfAppenderServer equals the previously configured vvp.appmanager.jobLogging.graylogHost):

    <Appender name="Gelf" type="GELF" includeExceptionCause="true" protocol="udp" server="{{ gelfAppenderServer }}">
      <KeyValuePair key="jobId" value="{{ jobId }}"/>
      <KeyValuePair key="deploymentId" value="{{ deploymentId }}"/>
    </Appender>
note

Please reach out to your Ververica account manager or contact us prior to upgrading if you are currently using the Ververica Platform Enterprise Trial and would like to upgrade to this release.

Upgrade​

As usual, we recommend upgrading via Helm after setting a persistence type in your values.yaml.

      helm repo add ververica https://charts.ververica.com
      helm upgrade [RELEASE] ververica/ververica-platform --version 4.1.0 --values values.yaml