Logo
1.2
  • dA Platform Setup
  • Application Manager
    • Quick Start
    • Concepts
    • Namespaces
    • Access Control
    • Deployments
      • Deployment Templates
      • Configure Apache Flink
      • Configure Kubernetes
      • Deployment Targets
      • Jobs
      • Savepoints
      • Secret Values
      • Event Log
      • CPU Quota
    • REST API
    • Web User Interface
    • Troubleshooting Application Manager
    • Releases
  • Streaming Ledger
dA Platform
  • Docs »
  • Application Manager »
  • Deployments »
  • Configure Apache Flink

Configure Apache Flink¶

In this page we will cover how to configure common Flink features with Application Manager.

The Deployment Template section of your Deployment provides a flinkConfiguration attribute that allows you to specify the Flink configuration for created jobs. The following examples should be nested under Deployment.spec.template.spec.

  • Savepoints
  • State in S3
  • High Availability

Savepoints¶

All stateful life-cycle operations (such as suspending a deployment or executing a stateful upgrade) require savepoints to be configured. Please provide an entry in the flinkConfiguration map with the key state.savepoints.dir:

flinkConfiguration:
  state.savepoints.dir: s3://flink/savepoints

The provided directory needs to be accessible by all nodes of your cluster. For instance, you can use a custom volume mount (Volume Mounts) or S3 bucket (State in S3).

Please consult the official Flink documentation on savepoints for more details.

State in S3¶

In order to store Flink application state such as checkpoints or savepoints in AWS S3, you have to provide credentials as part of the flinkConfiguration map.

flinkConfiguration:
  s3.access-key: yourAccessKey
  s3.secret-key: yourSecretKey

The default Flink images of Application Manager ship with PrestoS3FileSystem. Please refer to the Flink documentation on AWS deployment for more details.

High Availability¶

High availability (HA) of Flink applications requires a ZooKeeper installation and a persistent storage backend as described in the official Flink documentation on High Availability.

For convenience, we repeat the required options here as part of the flinkConfiguration map.

flinkConfiguration:
  high-availability: zookeeper
  high-availability.zookeeper.quorum: zk-node1:2181, zk-node2:2181, zk-node3:2181
  high-availability.zookeeper.storageDir: s3://flink/haStorage

Application Manager will automatically scope all state to the executing job by setting the high-availability.cluster-id to the respective job ID. Currently, you cannot overwrite this behaviour as it can lead to undefined side effects between jobs. Therefore, the above options are sufficient to configure HA with Application Manager.

Next Previous

© Copyright 2018, data Artisans GmbH.

Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.

Report an issue with this documentation page | Imprint

Other Versions v: v1.2
Tags
v2.10
v2.9
v2.8
v2.7
v2.6
v2.5
v2.4
v2.3
v2.2
v2.1
v2.0
v1.4
v1.3
v1.2
v1.1
v1.0
sql-eap