Jobs

A Job resource represents a single Apache Flink® job. The Job resource is derived from a Deployment Template whenever Ververica Platform detects that a new Flink job is needed.

Overview

The Job resource is exclusively managed by Ververica Platform and it is not possible to create it manually. Once the Job resource is created, its spec section is effectively immutable and represents a point in time of the spec section of the owning Deployment resource.

The status.state field of the Job resource has the following states:

• STARTING The Job is in the process of starting. This includes requesting resources from the Deployment Target and submitting the specified job to it.
• STARTED The Job was successfully started.
• TERMINATING The Job is in the process of terminating.
• TERMINATED The Job has terminated.
• FAILED The Deployment Target or Flink have reported an unrecoverable failure. Ververica Platform may report additional failure details.
• FINISHED The Job has terminated and finished successfully, e.g. a finite streaming or batch job.

Below state machine shows the possible transitions of a Job’s status:

Ververica Platform keeps all jobs in a terminal state. This allows users to trace back the evolution of the Deployment specification, such as parallelism and other configuration parameters.

Job Status

When a Job reaches status STARTED, additional information about the Job is provided under status.started. This section of the Job API is considered experimental and no guarantees are made about future availability.

1kind: Job
2status:
3  state: started
4  started:
5    # The Flink Job ID. Currently always equal to Job.metadata.id without
6    # dashes, e.g. 3f1da3c8-6f22-430f-9200-eda014ea5319 becomes
7    # 3f1da3c86f22430f9200eda014ea5319.
8    flinkJobId: string
9    # Time that the Job transitioned to the STARTED state.
10    transitionTime: string
11    # Time of last update to {status.started}.
12    lastUpdateTime: string
13    # The last observed number of Flink job restarts, including process
14    # restarts. The number is a lower bound on the actual number of Flink
15    # job restarts. On the happy path, it should be close to the actual
16    # number. During failures, there are no guarantees except that it is
17    # lower than or equal to the actual number of restarts.
18    observedFlinkJobRestarts: number
19    # The Flink JobStatus observed in the last probe. Either a Flink Job
20    # status string (CREATED, RUNNING, FAILING, FAILED, CANCELLING,
21    # CANCELED, FINISHED, RESTARTING, SUSPENDED, RECONCILING) or UNKNOWN
22    # if the Flink API was not reachable.
23    observedFlinkJobStatus: string
24  # ***SQL Deployments only*** a list of all sink tables of this query
25  # including schema and properties
26  sinkTables:
27    - catalogName: string
28      databaseName: string
29      name: string
30      temporary: boolean
31      columns:
32        - name: string
33          type: string
34      properties: object
35  # ***SQL Deployments only*** a list of all source tables of this query
36  # including schema and properties
37  sourceTables:
38    - catalogName: string
39      databaseName: string
40      name: string
41      temporary: boolean
42      columns:
43        - name: string
44          type: string
45      properties: object

Job storage configuration

Ververica Platform maintains a record of deployment jobs in persistent storage, as well as in AppManager memory. You can view the information regarding all the terminal jobs for a specific deployment. Terminal jobs refer to jobs that have any of these Ververica Platform specific statuses: FAILED, TERMINATED, and FINISHED.

By default, Ververica Platform stores an unlimited number of terminal jobs. However, you can configure the deletion of terminal jobs based on specific criteria such as time interval or the maximum number of jobs to be kept in history. This configuration can be done optionally during the installation of Ververica Platform.

Configuration

During the Ververica Platform installation, you can configure the following two optional properties:

• vvp.job-service.max-number-jobs-in-terminal-state - defines the maximum number of terminal jobs per deployment that should be always kept in jobs history.
• vvp.job-service.jobs-cleanup-interval - defines the duration of cleanup interval. Specify the interval using the Spring Duration conversion format. When these properties are provided, the AppManager scheduled task repeatedly searches for outdated deployment jobs and deletes them from both memory and persistent storage.