Reference Guide: Built-In Connectors
Apache Flink provides a variety of built-in connectors to facilitate the integration of Flink with different data sources and sinks (also called destinations). These connectors make it easy to read and write data from/to various systems in a scalable and fault-tolerant manner. In this section, we will introduce some of the most commonly used built-in connectors in Apache Flink.
Using the Console Network Detection feature, an IP address or a domain name can be used to check whether the running environment of a fully managed Flink deployment is connected to the upstream and downstream systems. See the FAQ section for more information.
Apache Paimon
Apache Paimon is a dynamic data lake storage format that supports high-throughput data writing and low-latency data querying. It is compatible with Flink-based Ververica Cloud and can be deployed on Hadoop Distributed File System (HDFS) or Ververica Cloud. Additionally, you can configure Paimon with Azure Workload Identity to securely access Azure Blob Storage.
Delta Lake
The Delta Lake connector allows you to read and write Delta Lake tables in Flink. Delta Lake is an open-source storage layer that brings reliability to data lakes with ACID transactions, schema enforcement, and time travel.
Apache Kafka
Apache Kafka is a distributed streaming platform designed for high-throughput, fault-tolerant, and scalable data streaming. Flink’s Kafka connector allows you to consume and produce data from and to Kafka topics.
Upsert Kafka SQL Connector
The Upsert Kafka SQL Connector allows Apache Flink to integrate with Apache Kafka for reading and writing data using upsert semantics. This is particularly useful when working with changelog streams or streaming upserts, where each record represents an update or deletion of a previous record based on a primary key.
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a managed, real-time data streaming service provided by Amazon Web Services (AWS). Flink’s Kinesis connector enables you to consume and produce data from and to Kinesis data streams.
DataGen
The DataGen connector in Apache Flink allows you to create tables with in-memory data generation, which is particularly useful for developing and testing queries locally without the need to access external systems such as Kafka. DataGen tables can include computed column syntax for flexible record generation.
Faker
The Faker connector leverages the popular Java Faker library to generate random data based on predefined patterns. This allows you to create tables with data that closely resembles real-world data, enabling you to develop and test your Flink applications more effectively.
Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. Flink’s Elasticsearch connector enables you to write data to Elasticsearch indices and perform real-time search and analytics operations on the stored data.
Fluss
Apache Fluss is a lightweight, high-performance log storage and streaming system within the Apache Software Foundation. It is a cloud-native alternative to traditional event streaming platforms that aims to simplify data infrastructure while maintaining strong integration with stream processing engines like Apache Flink. Fluss provides durable, ordered, and low-latency log storage to serve as the backbone for real-time analytics and event-driven applications.
Milvus
The Milvus sink connector writes streaming and batch data, including vector embeddings, into a Milvus collection. It supports upsert semantics based on a primary key, making it ideal for real-time AI and vector search applications.
MySQL & MySQL CDC
Apache Flink provides built-in connectors for MySQL to enable both batch processing and real-time change data capture (CDC) from MySQL databases. This allows you to read and write data from and to MySQL databases and capture changes in real time as they occur.
The MySQL connector allows you to read and write data from and to MySQL databases using Flink’s JDBC connector.
Oracle
The Oracle connector allows you to read from and write to Oracle databases in both streaming and batch modes. The connector is a direct extension of the JDBC Connector, sharing some of the capabilities and configuration options.
PostgreSQL
You can connect Ververica to PostgreSQL using three dedicated options. Use the standard Postgres connector to read from or write to tables, the CDC connector to stream real-time data changes, or the Postgres catalog to make your entire database available for queries. Read more here
Redis
Redis is an open-source, in-memory data structure store that can be used as a database, cache, and message broker. Flink’s Redis connector provides seamless integration with Redis, enabling you to read and write data from/to Redis data structures.
Snowflake
The Snowflake cloud-based data warehousing service designed to manage, share, and analyze large volumes of data with ease and efficiency. This sink-only connector is specifically engineered for scenarios where data is processed in Apache Flink before being transferred and persisted in Snowflake.
StarRocks
StarRocks is a next-gen, high-performance analytical data warehouse that enables real-time, multi-dimensional, and highly concurrent data analysis.
Most documentation about built-in connectors comes from the official Apache Flink® documentation.
Refer to the Credits page for more information.