Ververica Platform 2.2.0

Release Date: 2020-08-04

Changelog

Apache Flink® 1.11.1

• With the addition of our distribution of Apache Flink® 1.11.1 Ververica Platform 2.2.0 adds support for Apache Flink® 1.11. Apache Flink® versions 1.10 and 1.11 are supported under SLA. Apache Flink® 1.9 is deprecated in this version and supported on a best-effort basis.

• Please check the release notes of Flink 1.11.0 and Flink 1.11.1 for a complete list of features, improvements and fixes.

• For Stream Edition the following Apache Flink® Docker images are available. Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

  • 1.11.1-stream1-scala_2.11
  • 1.11.1-stream1-scala_2.12

• For Spring Edition the following / archives are available

  • flink-1.11.1-spring1-scala_2.11.tgz
  • flink-1.11.1-spring1-scala_2.12.tgz

Vulnerability Fixes

The following security vulnerabilities have been fixed compared to 1.10.1-[stream|spring]2:

CVE-2019-0201, CVE-2019-12399, CVE-2019-12402

Apache Flink® 1.10.1

• This update to our distribution of Apache Flink® 1.10.1 fixes a bug in the Kubernetes-based Flink Jobmanager failover.

  When using high-availability: vvp-kubernetes in conjunction with state.backend.incremental: true some checkpoint files were cleaned up prematurely leading to FileNotFoundExceptions during recovery.

• For Stream Edition the following Apache Flink® Docker images are available. Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

  • 1.10.1-stream2-scala_2.11
  • 1.10.1-stream2-scala_2.12

• For Spring Edition the following Apache Flink® archives are available

  • flink-1.10.1-spring2-scala_2.11.tgz
  • flink-1.10.1-spring2-scala_2.12.tgz

Vulnerability Fixes

The following security vulnerabilities have been fixed compared to 1.10.1-[stream|spring]1:

CVE-2017-5645, CVE-2020-9488

Apache Flink® 1.9.3

• This update to our distribution of Apache Flink® 1.9.3 fixes a bug in the Kubernetes-based Flink Jobmanager failover.

  When using high-availability: vvp-kubernetes together with with incremental checkpoints state.backend.incremental: true some checkpoint files were cleaned up prematurely leading to FileNotFoundExceptions during recovery.

• For Stream Edition the following Apache Flink® Docker images are available. Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

  • 1.9.3-stream2-scala_2.11
  • 1.9.3-stream2-scala_2.12

• For Spring Edition the following Apache Flink® archives are available

  • flink-1.9.3-spring2-scala_2.11.tgz
  • flink-1.9.3-spring2-scala_2.12.tgz

Vulnerability Fixes

None

Autopilot

Ververica Platform 2.2.0 introduces an autopilot for Apache Flink® streaming applications. When activated for a Deployment the autopilot will adjust the parallelism of your Deployment to varying load in order to maintain a back-pressure free configuration that matches a specified target utilization of your pipeline. Please check out Autopilot for configuration options, assumptions and future work.

Other Changes

Universal Blob Storage with Apache Hadoop®

You can now use HDFS 2 or HDFS 3 for Universal Blob Storage including authentication via Kerberos Keytabs. Besides HDFS, AWS S3, and Azure Blob Storage, as well as any S3-compatible object storage services like MinIO have been supported for a while.

Support for SQL Server

You can configure Ververica Platform to store its metadata in Microsoft SQL Server as well as Azure SQL.

The full list of supported remote RDBMS now includes: MariaDB, Microsoft SQL Server, MySQL, and PostgreSQL.

Please see Persistence Configuration for more details.

Suspend with Draining

When a Deployment is suspended or a stateful upgrade is triggered, Ververica Platform submits a stop command to Apache Flink®. This command will atomically trigger a savepoint and stop the job. You can now instruct Ververica Platform to additionally drain the pipeline. Draining will result in all future event time timers of the Flink job to fire before the stop command is executed. Checkout Draining for details.

Bootstrap Tokens

Ververica Platform now supports a Bootstrap Token, specified during installation or upgrade, which can be used as an API token with administrator privileges. This is useful for performing certain bootstrapping tasks such as creating an initial Namespace and assigning its members.

Checkpointing in flink-conf.yaml

Apache Flink® 1.10+ allows to set checkpointing configurations via the Deployment. The following entries have been added to the default Global Deployment Defaults.

flinkConfiguration:
  execution.checkpointing.interval: 10s
  execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION

In addition, execution.checkpointing.min-pause defaults to the value of execution.checkpointing.interval if not configured otherwise.

Attention

For existing Deployments that configure the checkpointing interval in application code but do not set the execution.checkpointing.min-pause, execution.checkpointing.min-pause will be configured to 10s effectively prohibiting checkpointing intervals lower than 10 seconds. In this case, explicitly setting execution.checkpointing.min-pause to the desired value on the Deployment resolves the issue.

In particular, Ververica Platform now warns users if their checkpointing configuration does not support the selected upgrade strategy.

UBS with AWS S3 on EKS

For Universal Blob Storage on AWS S3, we do not require hard coded credentials anymore if the credentials are inherited from the roles attached to EC2 instances underlying your Kubernetes Cluster.

If you have instance-level authentication configured, you can skip providing additional credentials during installation.

Bundled Apache Hadoop® S3 File System

From Flink 1.10 onwards all Flink images include both the HadoopS3Filesystem and the PrestoS3FileSystem. In our experience the PrestoS3FileSystem should be used by default unless you are using the StreamingFileSink, which requires the HadoopS3Filesystem. You can switch the file system implementation on a use case basis via the scheme.

Scheme	File System
s3	PrestoS3FileSystem
s3p	PrestoS3FileSystem
s3a	HadoopS3Filesystem

Deployment Overview Page

We have slightly reworked the Deployment page in the web user interface. It now displays the full Deployment configuration including logging configuration and pod templates.

Community Edition Feedback Widget

For community edition, the web user interface now contains a widget that allows to directly submit feedback & questions from within the application.

Vulnerability Fixes (outside of Apache Flink®)

CVE-2017-18640, CVE-2018-10237, CVE-2019-12402, CVE-2019-12418, CVE-2019-17563, CVE-2019-20444, CVE-2019-20445, CVE-2020-1935, CVE-2020-1938, CVE-2020-1938, CVE-2020-5397, CVE-2020-5398, CVE-2020-8022, CVE-2020-8022, CVE-2020-8203, CVE-2020-9484, CVE-2020-9484, CVE-2020-9488, CVE-2020-11612, CVE-2020-11996, CVE-2020-13692

Upgrade

We recommend upgrading via Helm using the following commands:

$ helm repo add ververica https://charts.ververica.com
$ helm upgrade [RELEASE] ververica/ververica-platform --version 4.2.0 --values custom-values.yaml