Skip to main content

Amazon Kinesis

Amazon Kinesis is a fully managed, scalable, and durable real-time streaming service offered by AWS, capable of handling large amounts of streaming data per second. It is designed to ingest, process, and analyze data in real-time, allowing users to gain timely insights and react quickly to new information. Kinesis is highly suitable for applications that require real-time analytics and monitoring.

Prerequisites​

To set up the integration, ensure an IAM user with the requisite permissions is available. This user should be permitted to create the IAM role and establish Kinesis streams. To create a stream, follow the guidelines provided in the Amazon Kinesis documentation. Upon successful creation, retrieve the Amazon Resource Name (ARN) of the Kinesis stream. Flink jobs will utilize this ARN for seamless data processing and analysis.

Step 1: Create a Kinesis data stream​

  1. Navigate to the Amazon Kinesis service console by searching ‘Kinesis’ from the search bar.
  2. Select Kinesis Data Streams and click Create data stream.
  3. Type in the name of the data stream, select On-Demand, and then click Create data stream at the bottom of the page.

image

Step 2: Create a permissions policy​

Create an IAM Permissions Policy with the needed permissions that Ververica Cloud will assume for this role and then accept the endpoint connection request. Refer to AWS docs to learn more information about the policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kinesis:*",
            "Resource": [
                "arn:aws:kinesis:us-west-1:<placeholder>:stream/kinesis-victor"
            ]
        }
    ]
}

image

Step 3: Create an IAM role​

Create an IAM Role to grant third parties like Ververica Cloud to access your AWS resources with the previous permission policy. To access the IAM configuration dashboard, type 'IAM' in the search bar of the AWS console.

info

The ‘External ID’ is actually the Workspace ID where you will configure this private connection. You can copy it from the top-right corner of the Private Connection page of that workspace. You can also find it in the URL when you enter the Ververica Console.

794031221915 is Ververica's AWS account ID.

Please create a dedicated IAM Role for each Ververica Cloud workspace. If the workspace is already bound with an IAM role, please update the policy instead of creating a new one. Refer to the AWS docs for how to create/update the IAM Role.

image

image

image

Get the ARN of this IAM Role; you will need it in the private connection configuration.

note

Please create a dedicated IAM Role for each workspace. If the workspace is already bound with an IAM role, please update the corresponding IAM role permissions policies instead of creating a new one. Refer to the AWS doc for how to create/update the IAM Role.

Step 4: Bind the IAM role to Ververica Cloud​

  1. Log in to Ververica Cloud and display your Dashboard.
  2. Click on the menu button (triple dots) at the end of the workspace entry and choose Private Connection.
  3. Bind the IAM Role at the first section of the page and click Save.
  4. Click New Private Connection, fill in the Name, set the Connector Type to Amazon Kinesis and click Save.

Private connection