Skip to main content
Version: 2.14

Resources (CPU, Memory)

The physical resources allocated for SessionClusters are configured per JobManager and TaskManager container and allocated CPU cores may be counted against a global CPU quota.

Overview​

You can configure requested compute resources for CPU and memory via the resources section:

kind: SessionCluster
spec:
  resources:
    jobmanager:
      cpu: 1
      memory: 1G
    taskmanager:
      cpu: 1
      memory: 2G

The keys jobmanager and taskmanager configure the respective Flink components. By default, the above values are used. You can overwrite each value selectively.

Note that resources are configured per component instance, e.g. if you use 10 TaskManager instances, Ververica Platform will in total try to acquire 10 times the configured taskmanager resources.

CPU​

You can specify CPU as a decimal number, e.g. 1, 1.0, or 0.5, which always needs to be greater than or equal to 0.1.

By default, the provided value will be set as the Kubernetes container CPU request and limit.

Memory​

You can specify memory as a decimal number with an optional unit as indicated by the suffix (ignoring case):

  • Gigabytes g, e.g. 1g or 1.5g
  • Megabytes m, e.g. 1000m or 1500m

Each memory unit is interpreted as a power of ten, e.g. 1k equals 1000 bytes. The memory unit is translated to base 2 before being forwarded to Flink. If no unit is specified, the provided number is interpreted as bytes.

Flink clusters require at least 1g of memory for the TaskManager and 650m of memory for the JobManager. By default, the provided value will be set as the Kubernetes container memory request and limit.

The memory configuration will be forwarded to the processes running the Flink TaskManager and JobManager, defined in Flink’s memory model, as described in the following sections.

TaskManager​

The configuration in resources["taskmanager"].memory is forwarded to Flink's taskmanager.memory.process.size configuration, relying on Flink's automatic configuration of overall TaskManager memory.

For low memory configurations with less than 1000m of total process memory, the managed memory fraction taskmanager.memory.managed.fraction is reduced to 0.2 to accommodate the minimum required by Flink.

JobManager​

The configuration in resources["jobmanager"].memory is forwarded to Flink's jobmanager.memory.process.size configuration, relying on Flink's automatic configuration of overall JobManager memory.

CPU Quota​

Ververica Platform Enterprise Edition may enforce a quota on the total number of CPU cores used by running, production Flink clusters. Community Edition does not enforce a resource quota.

The quota check is performed prior to launching a Flink cluster. This check considers CPU cores allocated by JobManager and TaskManager containers, as configured by the respective Kubernetes CPU limit.

Ververica Platform will never stop a running Flink cluster because of a quota limitation. If a Flink cluster can't be launched due to a quota limit, there will be a message in the event log.