Deployments

Deployments are the core resource abstraction within Ververica Platform to manage Apache Flink® jobs. A Deployment specifies the desired state of a Flink job and its configuration. Ververica Platform tracks and reports each Deployment’s status and derives other resources from it. Whenever the Deployment specification is modified, Ververica Platform will ensure that the corresponding Flink job will eventually reflect this change.

Overview

Deployments tie together a sequence of deployed Flink Jobs, their accumulated state, an event log, policies to execute upgrades, and a template for creating Flink jobs. A Deployment will always be backed by zero or one active Flink job(s). Each Flink job is either executed in application mode or session mode.

On a high-level, each Deployment consists of three parts:

metadata
Auxiliary information (e.g. name or modifcation timestamps) to manage the Deployment. This part is mostly user-configurable.
spec
Configuration and behaviour of the Deployment. This part is fully user-configurable.
status
Current status of the Deployment. This part is read-only.

The main part of a Deployment is its spec which is described in the following sections. At the end of this page you will find a full example.

Specification

The spec section is the main part of each Deployment and consists of:

When creating or modifying a Deployment, optional fields will be expanded with their default values.

Desired State

A Deployment resource has a desired state (specified in spec.state). A Deployment’s desired state can be one of the following:

RUNNING
This indicates that the user would like to start a Flink job as defined in the spec.template section.
CANCELED
This indicates that the user would like to terminate any currently running Flink jobs.
SUSPENDED
This indicates that the user would like to gracefully terminate the currently running job and take a snapshot of its state.

When the desired state does not match the current state, Ververica Platform will try to perform a state transition to achieve the desired state.

Note

The desired state will only be eventually achieved. The behavior of modifications happening concurrently to a state transition is undefined. Ververica Platform only guarantees that eventually the latest modification will be acted upon. Intermediate states may not be observed. This means that an ongoing state transition will not necessarily be immediately interrupted if the desired state changes while an operation is still ongoing.

Please refer to the Deployment Lifecycle page for more details.

Deployment Mode

Deployments can either be executed in application or session mode. This can be configured by providing one of the follwowing attributes:

deploymentTargetName
Execute the Deployment in application mode. The name references an existing Deployment Target in the same Namespace. In this mode, the deployed Flink job will have exclusive access to the Flink cluster.
sessionClusterName
Execute the Deployment in session mode. The name references an existing Session Cluster resource in the same Namespace. In this mode, deployed Flink jobs share resources with other Flink jobs running on the same Flink cluster.

Please refer to the Deployment Modes page for more details.

Deployment Template

The Deployment template section specifies which Apache Flink® job to execute and how to execute it, including its configuration.

You can think of settings in the template being directly applied to your job instances, whereas the overall Deployment specification section defines how to control these jobs over time (for instance, how to do upgrades).

Please refer to the Deployment Templates page for more details.

Deployment Upgrades

When Ververica Platform discovers that a running Flink job deviates from specified Deployment template, it will perform an upgrade of the Flink job to reconciliate the situation. The behaviour of Ververica Platform is defined by an upgrade strategy (spec.upgradeStrategy) and a restore strategy (spec.restoreStrategy).

Please refer to the Deployment Upgrades page for more details.

Deployment Defaults

When a Deployment is created, default values for attributes in the spec section will be filled in as specified in the global and namespace Deployment defaults configuration. Therefore, a Deployment will be always be fully specified after it has been created.

Please refer to the Deployment Defaults page for more details.

Limits

During state transitions of a Deployment, Ververica Platform creates Job resource and potentially triggers a Savepoint.

These operations might fail due to transient reasons, like a network issue, or misconfiguration. For these cases you can limit the number of attempts that Ververica Platform tries to create a Job or Savepoint before transitioning to a “FAILED” state:

Key Description Default
maxSavepointCreationAttempts Maximum attempted Savepoints before failing. 4
maxJobCreationAttempts Maximum Job creation attempts before failing. 4

Both of these limits are reset to zero after a successful Savepoint/Job creation.

Full Example

The following snippets are a complete example of a Deployment in application mode and session mode, including all optional keys and a Deployment Template.

kind: Deployment
apiVersion: v1
metadata:
  name: top-speed-windowing-example
  displayName: TopSpeedWindowing Example
  labels:
    env: testing
spec:
  state: RUNNING
  deploymentTargetName: default
  restoreStrategy:
    kind: LATEST_STATE
  upgradeStrategy:
    kind:  STATEFUL
  maxSavepointCreationAttempts: 4
  maxJobCreationAttempts: 4
  template:
    metadata:
      annotations:
        flink.queryable-state.enabled: 'false'
        flink.security.ssl.enabled: 'false'
    spec:
      artifact:
        kind: jar
        jarUri: https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.12/1.12.5/flink-examples-streaming_2.12-1.12.5-TopSpeedWindowing.jar
        additionalDependencies:
        - s3://mybucket/some_additional_library.jar
        - s3://mybucket/some_additional_resources
        mainArgs: --windowSize 10 --windowUnit minutes
        entryClass: org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
        flinkVersion: 1.12
        flinkImageRegistry: registry.ververica.com/v2.4
        flinkImageRepository: flink
        flinkImageTag: 1.12.5-stream1-scala_2.12
      flinkConfiguration:
        execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
        execution.checkpointing.interval: 10s
        high-availability: vvp-kubernetes
        state.backend: filesystem
      parallelism: 2
      numberOfTaskManagers: 2
      resources:
        jobManager:
          cpu: 1
          memory: 1g
        taskManager:
          cpu: 1.0
          memory: 2g
      logging:
        loggingProfile: default
        log4jLoggers:
          "": INFO
          org.apache.flink.streaming.examples: DEBUG
      kubernetes:
        pods:
          envVars:
          - name: KEY
            value: VALUE
kind: Deployment
apiVersion: v1
metadata:
  name: top-speed-windowing-example
  displayName: TopSpeedWindowing Example
  labels:
    env: testing
spec:
  state: RUNNING
  sessionClusterName: default
  restoreStrategy:
    kind: LATEST_STATE
  upgradeStrategy:
    kind:  STATEFUL
  maxSavepointCreationAttempts: 4
  maxJobCreationAttempts: 4
  template:
    spec:
      artifact:
        kind: jar
        jarUri: https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.12/1.12.5/flink-examples-streaming_2.12-1.12.5-TopSpeedWindowing.jar
        mainArgs: --windowSize 10 --windowUnit minutes
        entryClass: org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
      flinkConfiguration:
        execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
        execution.checkpointing.interval: 10s
        high-availability: vvp-kubernetes
        state.backend: filesystem
      parallelism: 2