Ververica Platform supports several alternatives for High Availability
Flink Kubernetes supports High-Availability (HA) Kubernetes
clusters out of the box. Ververica Platform with Flink Kubernetes
therefore supports Kubernetes HA out of the box.
Ververica Platform Kubernetes was introduced to enable HA
services at a time when Flink Kubernetes did not directly support
HA. While Ververica Platform Kubernetes is still available in the
platform, it is deprecated since Ververica Platform 2.10 and will be
removed in Ververica Platform 2.12. For discussion see below.
Other HA options include HA support via e.g. Zookeeper.
HA services based on Kubernetes do not require multiple Job
Managers bacause Kubernetes itself will restart the Job Manager pod
as required. However, running more than one Job Manager will
optimize recovery time.
Apache Flink® before v1.12 did not support Flink Kubernetes HA out
of the box. Instead HA support was based on Zookeeper. Ververica
therefore developed its own HA capable Kubernetes service, Ververica
Platform Kubernetes, to run HA Flink clusters on Kubernetes.
Ververica Platform 2.8.0 introduced support for Flink Kubernetes with
out of the box HA support. Since that time both Flink Kubernetes
and Ververica Platform Kubernetes have been available in the
platform. Both support HA Flink clusters on Kubernetes but with
different implementations which impact the behaviour of Kubernetes
TaskManagers, which in turn can impact the JobManager:
Flink Kubernetes: TaskManagers use Watchers to monitor for
changes to ConfigMaps, which in theory puts less pressure on the
API server at large scale compared to querying for changes.
Ververica Platform Kubernetes: TaskManagers periodically query
the Kubernetes API server for ConfigMaps. Therefore, the number of
requests scales with the number of TaskManagers and puts added
pressure on the API server at large scale, making JobManager a
potential single point of failure.
From Ververica Platform 2.10.0 Ververica Platform Kubernetes is
deprecated, and it will be removed in Ververica Platform 2.12. We
recommend that Flink applications that use Ververica Platform
Kubernetes are migrated to use Flink Kubernetes instead.