Draining

When a Deployment is suspended or a stateful upgrade is triggered, Ververica Platform will submit a Stop-with-Savepoint command to Apache Flink®. This command will atomically trigger a Savepoint and stop the job.

You can instruct Ververica Platform to additionally drain the pipeline. Draining will result in all future event time timers of the Flink job to fire before the Stop-with-Savepoint command is executed.

The following sections describe how to enable draining and provide example usage scenarios in which you would want to enable draining.

Usage Scenarios

Draining emits the maximum watermark before stopping the job. When the watermark is emitted, all event time timers will fire, allowing you to process events that depend on this timer (e.g. time windows or process functions).

This is useful when you want to fully shut down your job without leaving any unhandled events or state.

Another common scenario is triggering an incompatible job upgrade. Draining allows you to preserve important state such as Apache Kafka® offsets while you flush out incompatible state. Without draining you would typically need to start your upgraded job from a clean slate.

Configuration

Annotation

You can enable draining by setting the following annotation:

kind: Deployment
metadata:
  annotations:
    "com.dataartisans.appmanager.controller.deployment.stop-with-drain": "true"

As long as the annotation is set to true, suspensions and stateful upgrades will drain the pipeline.

Web Frontend

In the web frontend, you can enable draining by setting the Stop with Drain toggle as part in the Upgrade Strategy section (in the Behaviour tab of the Configure Deployment page).

When the toggle is activated, stateful upgrades and suspensions will drain the pipeline.