Ververica Platform 2.4.0

Release Date: 2021-03-10

Changelog

Charts

  • Ververica Platform 2.4 has dropped support for Helm 2, following its Deprecation Timeline. Please see the Upgrade section for more details.
  • The Helm chart now supports running custom initContainers and containers in the Ververica Platform Pod, via the parameters extraInitContainers and extraContainers, respectively.

Session Cluster Deployments

So far, Ververica Platform only supported Deployments in application mode. With this release, Deployments can also be executed in session mode. In session mode, a Deployment is deployed into a shared Session Clusters instead of a dedicated cluster. With session mode, there is no per Deployment resource overhead. Therefore, session mode is often used to run many, small, similar Deployments in a shared cluster.

Please see Deployment Modes for a comparison of application mode and session mode.

Improvements to REST API

  • Deployments are now name-addressable, thus names must now follow certain requirements.
  • Deployments’ metadata.name fields now serve as a unique identifier.
  • Deployments now have a new field for human-readable names, which is used for displaying Deployments in the frontend (metadata.displayName).
  • Deployments can now reference Deployment Targets by name (spec.deploymentTargetName).
  • Deployments, Deployment Targets, and Session Clusters now support PUT operations for idempotent upserts.

Migration & Compatibility

  • For existing Deployments, metadata.displayName will be created using the initial value of metadata.name. Additionally, metadata.name will be converted into a valid version if the initial name was invalid, adding a random suffix (5 alpha-numeric characters) to ensure uniqueness. In case of a an upgrade to Ververica Platform 2.4 and subsequent rollback to 2.3, the original name will have been replaced by the converted name. For example, Top Speed Windowing would have been converted to top-speed-windowing-oy20u.
  • Deployments can still be accessed by ID to maintain backward-compatibility.
  • Deployment Targets can still be referenced in Deployments by ID to maintain backward-compatibility.
  • Creating a Deployment via a POST operation will silently convert invalid names as long as a metadata.displayName is not provided.

Fully-Customizable Kubernetes Pod Templates

The two Kubernetes Pod Templates used for Flink JobManagers and Flink TaskManager are now fully customizable.

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        jobManagerPodTemplate: <V1PodTemplateSpec>
        taskManagerPodTemplate: <V1PodTemplateSpec>

With these new options you can now, for example, …

  • …add custom sidecars & init containers
  • …specify separate tolerations, labels, or node selectors for JobManager and TaskManager
  • …specify different resource limits and requests for TaskManagers

Please see Flink Pod Templates (Recommended) for details and examples.

Other Changes

  • From this release onwards, the minimum memory required for Flink JobManagers is 650m, which reflects the increased Apache Flink® requirements in newer versions. If you have an existing Deployment from before this release with a configured memory size below this threshold, you will not be able to make changes to it until you increase the memory request.
  • The Apache Flink® UI is now accessible in read-only mode for all viewers of a Namespace. Please check Authorization for more details.
  • legacy persistence has now been dropped. It was deprecated in Ververica Platform 2.1.0 and has been unsupported since Ververica Platform 2.2.0. If you were still using legacy persistence, upgrade your persistence mode prior to upgrading to Ververica Platform 2.4.0 as described in the Release Notes of Ververica Platform 2.1.0.
  • The Deployments page now has an additional Overview tab that shows a summary of the most important information of all the other tabs.

Upgrade

Warning

Due to Helm breaking changes and limitations, upgrading must be done with Helm version 3.2 or later. If you are on a version less that 3.2, please read the below notes before upgrading. For more information, see helm/helm#6850 and helm/helm#7649.

We recommend upgrading via Helm using the following commands:

$ helm repo add ververica https://charts.ververica.com
$ helm upgrade [RELEASE] ververica/ververica-platform --version 5.0.0 --values custom-values.yaml

If you are upgrading from Helm 2:

  • You’ll first need to migrate the release to Helm 3. It is recommended to use the helm-2to3 plugin to migrate the release metadata.
  • Once migrated, continue with the below section.

If you are upgrading from Helm < 3.2:

  • You’ll need to manually annotate two RBAC resources with Helm metadata in each namespace the Ververica Platform uses. The complete set of those namespaces is the namespace the Helm chart is installed in plus all the namespaces where jobs are managed (set in the Helm chart values field rbac.additionalNamespaces).

    • For both the role/$RELEASE_NAME-ververica-platform and rolebinding/$RELEASE_NAME-ververica-platform resources as RESOURCE, in each of the above namespaces as NAMESPACE:

      $ RELEASE_NAMESPACE=vvp # or wherever you installed the Ververica Platform
$ RELEASE_NAME=vvp # or whatever release name you installed the Ververica Platform under
$ NAMESPACE=vvp
$ RESOURCE=role/$RELEASE_NAME-ververica-platform
$ kubectl -n $NAMESPACE annotate $RESOURCE meta.helm.sh/release-name=$RELEASE_NAME
$ kubectl -n $NAMESPACE annotate $RESOURCE meta.helm.sh/release-namespace=$RELEASE_NAMESPACE
$ kubectl -n $NAMESPACE label $RESOURCE meta.helm.sh/release-namespace=$RELEASE_NAMESPACE
$ kubectl -n $NAMESPACE label $RESOURCE app.kubernetes.io/managed-by=Helm

  • Finally, you should be able to upgrade via the above-mentioned recommended method using Helm 3.2+.