Kubernetes High-Availability Service¶
Background¶
Flink provides the High-availability(HA) of JobManager to prevent it from being the single point of failure.
Before Flink-1.12, the Flink community only provides the Zookeeper HA out of the box.
Therefore, Ververica developed a HA service, VVP Kubernetes, for Flink clusters running on Kubernetes.
The Ververica Platform starts to support Flink kubernetes for flink distribution 1.15.2-stream3 and above since VVP-2.8.0.
VVP Kubernetes vs. Flink Kubernetes¶
For both VVP Kubernetes and Flink Kubernetes, the JobManager leader of a Flink cluster stores its information in Kubernetes ConfigMaps. The difference is how the TaskManager gets the leader’s information.
- VVP Kubernetes: the TaskManagers periodically query the Kubernetes API server for ConfigMaps. Therefore, the number of requests scales with the number of TaskManagers and puts pressure on the API server under large scale.
- Flink Kubernetes: the TaskManagers only “WATCH” the ConfigMaps for changes. Therefore, in theory, there will be less pressure on the API server on large scale compared to Flink clusters using VVP Kubernetes.