dA Platform 1.1.0

Changelog

  • General API: Deployment and Job resources are expanded eagerly with default values for optional fields. Prior to this version, optional fields of resources were lazily expanded which made it hard to understand the full state of a given resource. With this change, resources will always be fully specified.
  • Support for Flink 1.5: Flink 1.5 is the new default version for Deployments. You can specify which Flink version to deploy via the newly introduced flinkVersion attribute in Deployment.spec.template.spec.artifact. Note that the specified Flink version and Flink image have to match. If they do not match, deployed jobs will fail with a corresponding error message.
  • Custom Kubernetes options: We allow to pass Kubernetes-specific options to created Flink pods via Deployment.spec.template.spec.kubernetes. This gives you more control about deployed Flink jobs, for instance by attaching volume mounts or a node selector to deployed pods.
  • Specify Flink image by digest: In addition to image tags we allow to specify Flink images by image digest in Deployment.spec.template.spec.artifact.flinkImageTag. In order to specify a digest prefix the Flink image tag with @, e.g. @sha256:....
  • Support for finite Flink jobs: Finite Flink jobs such streaming applications consuming from finite sources or batch jobs transition transition to a new terminal state FINISHED. Prior to this release, any terminated job was treated as a failure and led to a job recovery (re-running the finite job).
  • Web UI: There have been additions to the web UI to support newly introduced features and multiple minor fixes.

Migration

  • When modifying an existing Deployment resource, default values will be expanded eagerly as described above. The newly introduced flinkVersion attribute will default to 1.5 which will lead to your Flink jobs to be upgraded to Flink 1.5. If you don’t want please update the Deployment accordingly with the first modification (PATCH) and set flinkVersion: 1.4.

Deprecation

  • Volume mount annotations with key alpha/k8s-volume-mounts have been deprecated with this release. Please migrate these annotations to the newly introduced Kubernetes options. Support for the annotations may be dropped in future versions.