Configure Apache Flink¶
In this page we will cover how to configure common Flink features with Application Manager.
The Deployment Template section of your Deployment provides a flinkConfiguration attribute that allows you to specify the Flink configuration for created jobs. The following examples should be nested under Deployment.spec.template.spec.
Savepoints¶
All stateful life-cycle operations (such as suspending a deployment or executing a stateful upgrade) require savepoints to be configured. Please provide an entry in the flinkConfiguration map with the key state.savepoints.dir:
flinkConfiguration:
state.savepoints.dir: s3://flink/savepoints
The provided directory needs to be accessible by all nodes of your cluster. For instance, you can use a custom volume mount (Volume Mounts) or S3 bucket (State in S3).
Please consult the official Flink documentation on savepoints for more details.
State in S3¶
In order to store Flink application state such as checkpoints or savepoints in AWS S3, you have to provide credentials as part of the flinkConfiguration map.
flinkConfiguration:
s3.access-key: yourAccessKey
s3.secret-key: yourSecretKey
The default Flink images of Application Manager ship with PrestoS3FileSystem. Please refer to the Flink documentation on AWS deployment for more details.
High Availability¶
High availability (HA) of Flink applications requires a ZooKeeper installation and a persistent storage backend as described in the official Flink documentation on High Availability.
For convenience, we repeat the required options here as part of the flinkConfiguration map.
flinkConfiguration:
high-availability: zookeeper
high-availability.zookeeper.quorum: zk-node1:2181, zk-node2:2181, zk-node3:2181
high-availability.zookeeper.storageDir: s3://flink/haStorage
Application Manager will automatically scope all state to the executing job by setting the high-availability.cluster-id to the respective job ID. Currently, you cannot overwrite this behaviour as it can lead to undefined side effects between jobs. Therefore, the above options are sufficient to configure HA with Application Manager.