Configure Kubernetes

In this page we will cover how to configure Kubernetes as part of your Deployment.

The Deployment Template section of your Deployment provides a kubernetes attribute that allows you to specify Kubernetes-specific options for your Deployment.

Overview

Currently, you can specify these options for pods created for Flink jobs.

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        pods:
          annotations:
            key: value
          nodeSelector:
            key: value
          affinity:
            KubernetesAffinity
          tolerations:
            - KubernetesToleration
          volumeMounts:
            - name: name
              volume: KubernetesVolume
              volumeMount: KubernetesVolumeMount
          envVars:
            - name: name
              value: value

Annotations

You can attach annotations to pods created for Flink jobs via pods.annotations.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            annotations:
              key: value

Provided annotations will be added to the metadata section of created pods.

Node Selector

You can attach a node selector to pods created for Flink jobs in order to constrain pods to run on particular nodes via pods.nodeSelector.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            nodeSelector:
              key: value

Affinity

You can attach an affinity to pods created for Flink jobs in order to constrain pods to run on particular nodes via pods.affinity.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            affinity:
              KubernetesAffinity

KubernetesAffinity is expected to be of type Affinity. The provided affinity will only be validated when the actual job is created and not when creating/modifying the resource.

Tolerations

You can attach tolerations to pods created for Flink jobs in order ensure that pods are not scheduled onto inappropriate nodes via pods.tolerations.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            tolerations:
              - KubernetesToleration

KubernetesToleration is expected to be of type Toleration. The provided tolerations will only be validated when the actual job is created and not when creating/modifying the resource.

Volume Mounts

You can attach volumes and volume mounts to pods created for Flink jobs, for instance an NFS mount for use as a state backend via pods.volumeMounts.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            volumeMounts:
              - name: name
                volume: KubernetesVolume
                volumeMount: KubernetesVolumeMount

KubernetesVolume is expected to be of type Volume and KubernetesVolumeMount is expected to be of type VolumeMount. The provided volumes and volume mounts will only be validated when the actual job is created and not when creating/modifying the resource.

Example: Mounting an NFS and Secret

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        pods:
          volumeMounts:
            - name: my-volume
              volume:
                name: my-volume
                nfs:
                  server: 10.1.2.3
                  path: /daplatform-foo
              volumeMount:
                name: my-volume
                mountPath: /foo/bar
            - name: my-secret
              volume:
                name: my-secret
                secret:
                  secretName: my-secret
              volumeMount:
                name: my-secret
                mountPath: /var/run/secrets/some-secret

Environment Variables

You can set environment variables for pods created for Flink jobs. These can be useful to dynamically configure image-specific behaviour, such as using a custom log42j.xml or overwriting Flink-specific variables.

kind: Deployment
  spec:
    template:
      spec:
        kubernetes:
          pods:
            envVars:
              - name: name
                value: value

Example: Using a Custom Log Configuration File

In this example, we will mount a custom log4j2.xml configuration and use it in our Deployment.

For this example, we have the following content in a file called custom-log4j2.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
  <Appenders>
    <Appender name="STDOUT" type="Console">
      <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
    </Appender>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="STDOUT"/>
    </Root>
  </Loggers>
</Configuration>

We create a Kubernetes ConfigMap with the contents of this file as follows:

kubectl create configmap custom-log4j-config --from-file=log4j2.xml=custom-log42j.xml

We can now mount the ConfigMap in our Flink Deployment and use use the mounted file by setting the environment variable LOG4J_CONF.

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        pods:
          volumeMounts:
            - name: logconfig
              volume:
                name: logconfig
                configMap:
                  name: custom-log4j-config
              volumeMount:
                name: logconfig
                mountPath: /opt/flink/conf
          envVars:
            - name: LOG4J_CONF
              value: /opt/flink/conf/log42j.xml