Skip to main content

Architecture Overview

Architecture Overview​

The Bring-Your-Own-Cloud (BYOC) deployment model lets you run Apache Flink workloads in your own Kubernetes environment (such as Amazon EKS), while centrally managing and monitoring those workloads through the Ververica Cloud Control Plane.

image

In this architecture, you install Ververica Agent (also referred to as “Pyxis”) on your Kubernetes cluster to communicate with Ververica Cloud. The diagram above illustrates how the various components interact:

  1. Install Agent
    You begin by installing the Ververica BYOC Agent (Pyxis) onto your Kubernetes cluster. The agent runs alongside any workloads you plan to deploy.

  2. Initiate Connection
    Once installed, the agent connects outbound to the Ververica Cloud Control Plane over HTTPS. This channel ensures that all communication from your cluster to Ververica Cloud is encrypted and unidirectional, reducing your infrastructure’s exposure.

  3. Send Instructions
    The Ververica Cloud Control Plane sends instructions - such as creating a new workspace - to the Ververica Agent using the established secure connection. The agent then executes these instructions in your cluster.

  4. Create and Manage Workspace Agents
    Each new workspace is managed by its own Workspace Agent, which the Ververica Agent creates on your behalf. The workspace agent is responsible for tasks like deploying Flink jobs and managing the underlying Kubernetes resources.

  5. Create and Manage Flink Jobs
    Within a workspace, the workspace agent creates and manages the Flink jobs. These jobs run in your infrastructure and access data in your data stores (e.g., S3, Kafka, relational databases) depending on your internal network setup. For example, the jobs access the data stores in the same VPC directly, or access other data stores via VPC peering, private links, or other secure networking methods.

  6. Report Status
    At every step, the Workspace Agent reports real-time status and metrics back to the Ververica Cloud Control Plane. This feedback loop provides a centralized view of your Flink applications across multiple workspaces.

Key Components​

  • Ververica Cloud Control Plane: The centralized service that handles user interactions, job orchestration, and monitoring.
  • Ververica Agent (Pyxis): A specialized controller deployed in your Kubernetes cluster. It maintains a secure, outbound-only connection to Ververica Cloud and orchestrates Workspace Agents.
  • Workspace Agent: A dedicated agent within each workspace. It manages lifecycle operations (create, update, delete) for Flink jobs and monitors their status.
  • Flink Jobs: The actual stream processing or batch jobs running on Kubernetes infrastructure. These jobs access data sources, sinks, or other resources within your own cloud environment.
  • Data Stores: This includes systems like Amazon S3, Kafka, relational databases, or other services reachable from your Kubernetes cluster. Connectivity may be established via direct access, VPC peering, or private links.
On this page