dA Platform 1.3.0

Changelog

  • End-to-end Flink SSL/TLS allows you to auto provision Flink 1.6 and 1.7 jobs with SSL/TLS support, including mutual authentication.
  • New restoreStrategy API for more flexible state restore operations. This include a new mode of operation that allows you to restore your application from both savepoints as well as checkpoints. You have to enable Flink High Availability and checkpoint retention in order to enable restoring from latest checkpoints.
  • Application Manager supports Flink 1.7 and manages Flink applications using the container entrypoint. This makes the deployment of applications more robust and less time consuming. Instead of requiring Application Manager to access and submit your JAR artifact to Flink, we let Flink fetch it.
  • You can now configure custom imagePullSecrets for each Deployment, making it possible to access custom Docker registries.
  • The Settings page of the web user interface now allows you to manage more resources, such as namespaces, API tokens, and secret values.

Migration

  1. Important: Flink 1.7 jobs require entryClass to be configured in Deployment.spec.template.spec.artifact. If you are running Application Manager version greater or equal to 1.3.2 and Flink image tagged as 1.7.0-dap2 or 1.7.1-dap2 (or more recent) this requirement has been dropped. See release notes of version 1.3.2.
  2. If you are using Flink 1.6, please set your Flink applications to use the image with tag: registry.platform.data-artisans.net/trial/v1.3/flink:1.6.2-dap2-scala_2.11. The dap2 image fixes the built-in InfluxDB metrics reporter that was included in earlier versions. For Flink 1.7, all images already contain the fixed metrics reporter.
  3. In order to use an existing data volume of an earlier installation please add the following securityContext entry to your Application Manager Deployment:
kind: Deployment
spec:
  template:
    spec:
      securityContext:
      fsGroup: 102
  1. Add the following at the root level of your Application Manager ConfigMap for the appmanager.yaml entry:
cluster:
  kubernetes.artifact-fetcher.image-registry: registry.platform.data-artisans.net/trial/v1.3
  kubernetes.artifact-fetcher.image-tag: 1.0.1
  1. Update the image and args for the Application Manager Kubernetes Deployment:
kind: Deployment
metadata:
  name: daplatform-appmanager
spec:
  containers:
  - name: appmanager
    image: registry.platform.data-artisans.net/trial/v1.3/appmanager:1.3.0
    args:
      - server
      - /etc/daplatform/appmanager.yaml
  1. Give access to jobs and secrets Kubernetes resources for Role daplatform-appmanager.
kind: Role
metadata:
  name: daplatform-appmanager
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - "batch"
  resources:
  - jobs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Deprecation

  • Deployment.spec.startFromSavepoint has been deprecated in favor of Deployment.spec.restoreStrategy. Operations against startFromSavepoint are reflected in restoreStrategy and vice versa. Only when using the new LATEST_STATE restore strategy will startFromSavepoint be not available.