Deployment Templates

Deployment templates are specified as part of Deployment under spec.template and configure the Apache Flink® job to execute, including Flink cluster-level configuration as well as Flink job-specific configuration.

Overview

The relationship between the overall Deployment specification and its Deployment template is as follows:

  • The Deployment template specifies which Apache Flink® job is executed and how to execute it, including its configuration.
  • The Deployment specification defines how job instances are managed over time, for instance how to perform upgrades or which Savepoint to restore from.

On a high-level, each Deployment template consists of two parts:

metadata
Optional annotations used to provide additional configuration options, used to enable additional features.
spec
The spec specifies which Flink job to execute and how to configure it.

The main part of a Deployment template is its spec which is described in the following sections. At the end of this page you will find a full example.

Application Mode vs. Session Mode

The Deployment template specifies both Flink cluster-level configuration as well as Flink job-specific configuration.

In application mode, the Deployment will be executed in a separate Flink cluster and all template attributes are available for configuration.

In session mode, the Deployment will be executed in a shared Flink cluster and some template attribute have to be omitted. The cluster-level configuration happens as part of the SessionCluster resource referenced by the Deployment. Therefore, cluster-level configuration must be ommitted from the Deployment template.

The following table summarizes the available attributes and whether they can be specified in the respective Deployment mode.

Application vs. Session Mode
Attribute See Also Application Mode Session Mode
Metadata (template.metadata)      
annotations["flink.queryable-state.enabled"] Flink Configuration ✓ !
annotations["flink.security.ssl.enabled"] Flink Configuration ✓ !
Spec (template.spec)      
artifact["kind"="jar"] Artifacts ✓ ✓
artifact["kind"="sqlscript"] Artifacts ✓ ✓
flinkConfiguration Flink Configuration ✓ ✓
parallelism Scaling Deployments ✓ ✓
numberOfTaskManagers Scaling Deployments ✓ !
resources Scaling Deployments ✓ !
logging Logging ✓ !
kubernetes Kubernetes Pod Templates ✓ !
  • ✓ indicates that the attribute can be configured as part of the Deployment Template.
  • ! indicates that the attribute cannot be configured as part of the Deployment Template. In this case, the attribute affects the Flink cluster and should be configured as part of the SessionCluster referenced by the Deployment.

Full Example

Application Mode
Session Mode
kind: Deployment
apiVersion: v1
metadata:
  name: top-speed-windowing-example
  displayName: TopSpeedWindowing Example
  labels:
    env: testing
spec:
  state: RUNNING
  deploymentTargetName: default
  restoreStrategy:
    kind: LATEST_STATE
  upgradeStrategy:
    kind:  STATEFUL
  maxSavepointCreationAttempts: 4
  maxJobCreationAttempts: 4
  jobFailureExpirationTime: 0
  template:
    metadata:
      annotations:
        flink.queryable-state.enabled: 'false'
        flink.security.ssl.enabled: 'false'
    spec:
      artifact:
        kind: jar
        jarUri: https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.12/1.16.1/flink-examples-streaming_2.12-1.16.1-TopSpeedWindowing.jar
        additionalDependencies:
        - s3://mybucket/some_additional_library.jar
        - s3://mybucket/some_additional_resources
        mainArgs: --windowSize 10 --windowUnit minutes
        entryClass: org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
        flinkVersion: 1.16
        flinkImageRegistry: registry.ververica.com/v2.10
        flinkImageRepository: flink
        flinkImageTag: 1.16.1-stream2-scala_2.12
      flinkConfiguration:
        execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
        execution.checkpointing.interval: 10s
        high-availability: vvp-kubernetes
        state.backend: filesystem
      parallelism: 2
      numberOfTaskManagers: 2
      resources:
        jobManager:
          cpu: 1
          memory: 1g
        taskManager:
          cpu: 1.0
          memory: 2g
      logging:
        loggingProfile: default
        log4jLoggers:
          "": INFO
          org.apache.flink.streaming.examples: DEBUG
      kubernetes:
        pods:
          envVars:
          - name: KEY
            value: VALUE

kind: Deployment
apiVersion: v1
metadata:
  name: top-speed-windowing-example
  displayName: TopSpeedWindowing Example
  labels:
    env: testing
spec:
  state: RUNNING
  sessionClusterName: default
  restoreStrategy:
    kind: LATEST_STATE
  upgradeStrategy:
    kind:  STATEFUL
  maxSavepointCreationAttempts: 4
  maxJobCreationAttempts: 4
  jobFailureExpirationTime: 0
  template:
    spec:
      artifact:
        kind: jar
        jarUri: https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.12/1.16.1/flink-examples-streaming_2.12-1.16.1-TopSpeedWindowing.jar
        mainArgs: --windowSize 10 --windowUnit minutes
        entryClass: org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
      flinkConfiguration:
        execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
        execution.checkpointing.interval: 10s
        high-availability: vvp-kubernetes
        state.backend: filesystem
      parallelism: 2