Amazon S3

Amazon S3 (Simple Storage Service) is a highly scalable and durable object storage service offered by AWS. It's designed to store and retrieve data from anywhere on the web, making it a great fit for web applications, backup and restore solutions, archiving, and big data analytics. S3 provides easy-to-use management features, fine-tuned access controls, and robust security capabilities, including rest and transit encryption.

Prerequisites

To set up the integration, ensure an IAM user with the requisite permissions is available. This user should be permitted to create the IAM role and Amazon S3 bucket. To create a S3 bucket, follow the guidelines provided in the Amazon S3 documentation. Upon successful creation, retrieve the Amazon Resource Name (ARN) of the S3 bucket. Flink jobs will utilize this ARN for seamless data processing and analysis.

Step 1: Create a S3 Bucket

1. Navigate to the Amazon S3 service console by searching ‘S3’ from the search bar.
2. Click the ‘Create bucket’ button on the top-right corner.
3. Type in the name of the bucket, select the region, and leave everything else to default, then click the 'Create bucket' button at the bottom of the page.

Step 2: Create a Permissions Policy

Create an IAM Permissions Policy with the needed permissions that Ververica Cloud will assume for this role and then accept the endpoint connection request. Refer to AWS doc to learn more information about the policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
            "Resource": "arn:aws:s3:::my-vvc-bucket-victor/*"
        },
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": "arn:aws:s3:::my-vvc-bucket-victor"
        }
    ]
}

Step 3: Create an IAM Role

Create an IAM Role to grant third parties like Ververica Cloud to access your AWS resources with the previous permission policy.

info

The ‘External ID’ is actually the Workspace ID where you will configure this private connection. You can copy it from the top-right corner of the Private Connection page of that workspace. You can also find it in the URL when you enter the Ververica Console.

794031221915 is Ververica's AWS account ID.

Please create a dedicated IAM Role for each Ververica Cloud workspace. If the workspace is already bound with an IAM role, please update the policy instead of creating a new one. Refer to the AWS docs for how to create/update the IAM Role.

Step 4: Bind the IAM Role to Ververica Cloud

1. Log in to Ververica Cloud and display your Dashboard.
2. Click on the menu button (triple dots) at the end of the workspace entry and choose Private Connection.
3. Bind the IAM Role at the first section of the page and click Save.
4. Click New Private Connection, fill in the Name, set the Connector Type to Amazon S3 and click Save.