Scaling Deployments

The scale of Deployments is configured via the parallelism attribute of the Deployment Template. In application mode, it is possible to additionally configure the number of deployed TaskManagers and the number of task slots per TaskManager.

Overview

The scale of your Deployments is configured via the parallelism attribute of the Deployment template.

1kind: Deployment
2spec:
3  template:
4    spec:
5      parallelism: p

Depending on the Deployment Mode, you have additional control over how the Deployment is executed.

Application Mode

In application mode, you have control over the parallelism of your Flink job and the number of deployed TaskManager instances, including the number of task slots per TaskManager. By default, each created TaskManager instance will be configured with a single slot, resulting in p TaskManager instances for parallelism p.

Session Mode

In session mode, you only have control over the parallelism of your Flink job. The number of TaskManager instances and the number of task slots per TaskManager is configured in the SessionCluster referenced by the Deployment.

Parallelism, Number of Taskmanagers, and Slots

In application mode, you have fine-grained control over the number of available TaskManager instances and the number of task slots per TaskManager.

By default, each created TaskManager instance will be configured with a single slot, resulting in p TaskManager instances for parallelism p. It is highly recommended to adjust this configuration for your use case.

1kind: Deployment
2spec:
3  template:
4    spec:
5      parallelism: p
6      numberOfTaskManagers: n
7      flinkConfiguration:
8        taskmanager.numberOfTaskSlots: s

In total, there will be n * stask slots available. With a default slot sharing configuration, p must be lower or equal to the total the number of available slots. Otherwise the job cannot be executed.

Typically, the number of slots per TaskManager should be adjusted hand in hand with the allocated compute resources per TaskManager.

Examples

The following examples assume application modeand a default slot sharing configuration, ignoring compute resource allocation.

Run on Single TaskManager instance

In the following example, the Flink job will be deployed with parallelism of 8 on a single TaskManager instance.

1kind: Deployment
2spec:
3  template:
4    spec:
5      parallelism: 8
6      numberOfTaskManagers: 1
7      flinkConfiguration:
8        taskmanager.numberOfTaskSlots: 8

Run on Fixed Number of TaskManager instances

In the following example, the Flink job will be deployed with parallelism of 8 on two TaskManager instances.

1kind: Deployment
2spec:
3  template:
4    spec:
5      parallelism: 8
6      numberOfTaskManagers: 2
7      flinkConfiguration:
8        taskmanager.numberOfTaskSlots: 4

Run with Standby TaskManager

In the following example, the Flink job will be deployed with a standby TaskManager that can recover the job without waiting for a new TaskManager to be available.

1kind: Deployment
2spec:
3  template:
4    spec:
5      parallelism: 8
6      numberOfTaskManagers: 2
7      flinkConfiguration:
8        taskmanager.numberOfTaskSlots: 8