Create AWS resources

To prepare your AWS environment for deploying the Ververica Agent on an Amazon EKS cluster, complete the following steps:

Watch this video for an overview about creating AWS resources for Ververica Cloud: Bring Your Own Cloud.

• Set up resources, including IAM roles, policies, and S3 buckets.
• Create and configure your S3 bucket to store runtime artifacts and Flink checkpoints.
• Enable Kubernetes OIDC (OpenID Connect) for IAM Roles for Service Accounts (IRSA).
• Set up IAM roles and policies that securely manage the Ververica Agent.

Once these steps are complete, your environment will be configured for a deployment of the Ververica Agent.

Set up AWS Resources

You can set up these AWS resources using the provided CloudFormation stack, the Terraform module, or directly in the AWS Console.

For most cases on existing AWS EKS infrastructure, Ververica recommends the CloudFormation stack below. It creates all required resources in a single click and needs no customization.

If your setup needs more flexibility and stricter security, for example a custom encryption key, trust policy hardening, or additional IAM policies, use the Terraform module instead.

If you want full control over each resource, or want to understand every component before you create it, follow the manual steps further down this page.

For production deployments, Ververica recommends creating all resources with Terraform. If you don't have existing infrastructure and want to deploy everything from scratch, including the VPC and EKS cluster, see Deploy a full BYOC infrastructure with Terraform below.

Open the AWS console with the CloudFormation template preloaded.

Use the Terraform module

Ververica publishes an open source Terraform module, `byoc-agent`, that provisions the same S3 bucket, OIDC provider, and IAM roles as the CloudFormation stack, with more room for customization.

This module assumes you already have an EKS cluster and its supporting infrastructure (VPC, subnets, node groups). If you're starting from scratch, see Deploy a full BYOC infrastructure with Terraform below instead.

Use this module if you manage your infrastructure as code, or if you need to customize a resource beyond what the CloudFormation stack supports, for example a KMS-encrypted bucket, a restricted OIDC trust policy, or an existing OIDC provider.

Prerequisites

• Terraform 1.5.0 or later
• The AWS provider 5.0 or later
• An existing Amazon EKS cluster

Example usage

1data "aws_eks_cluster" "this" {
2  name = "my-eks-cluster"
3}
4
5module "byoc_agent" {
6  source = "ververica/ververica-cloud/aws//modules/byoc-agent"
7
8  bucket_name       = "my-vvc-agent-bucket"
9  oidc_provider_url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
10
11  # Restrict which Kubernetes service account can assume the admin role.
12  admin_role_subject_claims = [
13    "system:serviceaccount:ververica:pyxis-admin",
14  ]
15
16  tags = {
17    Environment = "production"
18  }
19}

You can further customize the module through inputs for CORS, bucket encryption (SSE-S3 or SSE-KMS), and extra managed policies. For the full list of inputs, outputs, and runnable examples, see the `byoc-agent` module reference on GitHub.

Deploy a full BYOC infrastructure with Terraform

If you don't have an EKS cluster yet, use the complete reference example. It provisions the entire BYOC infrastructure layer from scratch, in a single terraform apply:

• A VPC with public and private subnets across three availability zones, and a NAT gateway.
• An EKS cluster with OIDC/IRSA enabled and a managed node group.
• An EKS access entry that grants an existing IAM role cluster-admin access.
• The same byoc-agent resources described above (S3 bucket, OIDC trust, IAM roles), wired to the new cluster automatically.

1terraform init
2terraform apply \
3  -var="access_role_arn=arn:aws:iam::123456789012:role/my-team-role" \
4  -var="agent_bucket_name=my-vvc-agent-bucket"

For the full variable reference and architecture diagram, see the example's README on GitHub.

Create an S3 Bucket

Step 1

Use the AWS CLI or the AWS Management Console to create an S3 bucket that stores runtime artifacts and Flink checkpoints (https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints/). You can also use an existing bucket if you prefer. S3 bucket names must be globally unique. A common practice is to include your AWS account ID in the bucket name. For example:

BASHCopy

1aws s3api create-bucket --bucket vv-agent-bucket-1234567890 --region eu-central-1

Step 2

After creating the bucket, write down its Amazon Resource Name (ARN) for later use.

Step 3

Configure Cross-Origin Resource Sharing (CORS) for the bucket to ensure that only allowed origins can access the bucket, enforcing least privilege principles. The following configuration permits access from a Ververica website endpoint and required methods: a. In the AWS Management Console, open your S3 bucket.
b. Go to Permissions.
c. Scroll down to Cross-origin resource sharing (CORS) and click Edit.
d. Add the following JSON code shown below, then click Save changes.

JSONCopy

1  [
2      {
3          "AllowedHeaders": ["*"],
4          "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"],
5          "AllowedOrigins": ["https://app.ververica.cloud"],
6          "ExposeHeaders": ["ETag"],
7          "MaxAgeSeconds": 3000
8      }
9  ]
10

Enable Kubernetes OIDC

The agent uses IAM Roles for Service Accounts (IRSA) to authorize API calls to AWS. Your Amazon EKS cluster provides an OpenID Connect (OIDC (https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)) provider URL that you can use to configure pod permissions with IAM Roles.

Step 1

To obtain the cluster's OIDC provider URL, run the following AWS CLI command, replacing $cluster_name with your cluster's name. You can also find it in the AWS Console on the EKS page:

BASHCopy

1aws eks describe-cluster --name $cluster_name --query "cluster.identity.oidc.issuer" --output text

Step 2

Configure Provider: a. Navigate to Identity Providers and choose Add provider.
b. Set Provider type to OpenID Connect.
c. For Provider URL, paste the OIDC Provider URL you obtained in the previous step.
d. For Audience, add sts.amazonaws.com.
e. (Optional) Add tags.
f. Click Add provider.

Set Up IAM Roles and Policies

You need to create two IAM roles and their corresponding policies for the Ververica Agent (vv-agent):

• VVCTenantRole: Manages the S3 bucket.
• VVCAdminRole: Serves as the IRSA role for the Agent pod.

Create the Tenant Policy (VVCTenantPolicy)

1{
2    "Version": "2012-10-17",
3    "Statement": [
4        {
5            "Effect": "Allow",
6            "Action": [
7                "s3:PutObject",
8                "s3:GetObject",
9                "s3:DeleteObject"
10            ],
11            "Resource": "arn:aws:s3:::vv-agent-bucket-*/*"
12        },
13        {
14            "Effect": "Allow",
15            "Action": [
16                "s3:ListBucket",
17                "s3:GetBucketLocation"
18            ],
19            "Resource": "arn:aws:s3:::vv-agent-bucket-*"
20        }
21    ]
22}

Create the Tenant Role (VVCTenantRole)

1. In the IAM console, go to Roles and choose Create role.
2. On Select trusted entity, choose AWS account as the trusted entity type.
3. Select This account, then click Next.
4. On Add permissions, search for VVCTenantPolicy and select it.
5. Click Next and then review the role settings.
6. Name the role VVCTenantRole.
7. Click Create role.

Create the Admin Policy (VVCAdminPolicy)

1{
2    "Version": "2012-10-17",
3    "Statement": [
4        {
5            "Effect": "Allow",
6            "Action": [
7                "sts:AssumeRole"
8            ],
9            "Resource": "arn:aws:iam::<REPLACE_WITH_ACCOUNT_ID>:role/VVCTenantRole"
10        }
11    ]
12}

Create the Admin Role (VVCAdminRole)

1. In the IAM console, go to Roles and choose Create role.
2. Select Web identity as the trusted entity type.
3. For Identity provider, select the EKS cluster OIDC provider.
4. For Audience, select sts.amazonaws.com.
5. Click Next.
6. On Add permissions, search for VVCAdminPolicy and select it.
7. Click Next, then name the role VVCAdminRole.
8. Confirm that the trust policy and permissions are correct.
9. Click Create role.