Skip to main content
Version: 2.13

Scaling Deployments

The scale of Deployments is configured via the parallelism attribute of the Deployment Template. In application mode, it is possible to additionally configure the number of deployed TaskManagers and the number of task slots per TaskManager.

note

See also: Deployments running in application mode can configure allocated compute resources and support autoscaling.

Overview​

The scale of your Deployments is configured via the parallelism attribute of the Deployment template.

kind: Deployment
spec:
  template:
    spec:
      parallelism: p

Depending on the Deployment Mode, you have additional control over how the Deployment is executed.

Application Mode

In application mode, you have control over the parallelism of your Flink job and the number of deployed TaskManager instances, including the number of task slots per TaskManager. By default, each created TaskManager instance will be configured with a single slot, resulting in p TaskManager instances for parallelism p.

Session Mode

In session mode, you only have control over the parallelism of your Flink job. The number of TaskManager instances and the number of task slots per TaskManager is configured in the SessionCluster referenced by the Deployment.

Parallelism, Number of Taskmanagers, and Slots​

In application mode, you have fine-grained control over the number of available TaskManager instances and the number of task slots per TaskManager.

By default, each created TaskManager instance will be configured with a single slot, resulting in p TaskManager instances for parallelism p. It is highly recommended to adjust this configuration for your use case.

kind: Deployment
spec:
  template:
    spec:
      parallelism: p
      numberOfTaskManagers: n
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: s

In total, there will be n * s task slots available. With a default slot sharing configuration, p must be lower or equal to the total the number of available slots. Otherwise the job cannot be executed.

Typically, the number of slots per TaskManager should be adjusted hand in hand with the allocated compute resources per TaskManager.

Examples​

The following examples assume application mode and a default slot sharing configuration, ignoring compute resource allocation.

Run on Single TaskManager instance​

In the following example, the Flink job will be deployed with parallelism of 8 on a single TaskManager instance.

kind: Deployment
spec:
  template:
    spec:
      parallelism: 8
      numberOfTaskManagers: 1
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: 8

Run on Fixed Number of TaskManager instances​

In the following example, the Flink job will be deployed with parallelism of 8 on two TaskManager instances.

kind: Deployment
spec:
  template:
    spec:
      parallelism: 8
      numberOfTaskManagers: 2
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: 4

Run with Standby TaskManager​

In the following example, the Flink job will be deployed with a standby TaskManager that can recover the job without waiting for a new TaskManager to be available.

kind: Deployment
spec:
  template:
    spec:
      parallelism: 8
      numberOfTaskManagers: 2
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: 8