Skip to main content

Custom connectors

Background information​

When using Flink SQL to develop drafts, you need to use SQL connectors to connect your source tables, result tables, and dimension tables. Fully managed Flink supports multiple types of common connectors in the cloud. However, compared with the technology stacks that are commonly used for big data, fully managed Flink covers only a small part of connectors. If you want to use more types of connectors, you can customize connectors and upload them for use. You can edit and delete existing custom connectors.

Precautions​

  • You must develop custom connectors based on the connector standards that are defined by the Apache Flink community. This way, the custom connectors can be recognized and used by fully managed Flink. You must explicitly define connector metadata files and declare factory classes. For more information about how to develop custom connectors, see User-defined Sources & Sinks.

  • Connectors of the same type can only be uploaded once.

  • To avoid JAR package dependency conflicts, you also need to pay attention to the following points:

    • Keep the Flink version that the Flink image and the Pom depend on consistent.
    • Provided in the dependencies.
    • Use Shade to package other third-party dependencies. For details about Shade packaging, see Apache Maven Shade Plugin

  • If a deployment uses a custom connector and the custom connector is updated when the deployment is running, the deployment still uses the original custom connector. If you restart the deployment, the deployment uses the new connector.

note

See Connecting with external file systems for information on the s3i and s3 schemas when communicating with external S3 services via your own S3 connector (e.g. for sources and sinks).

Create custom connector​

  1. On the Dashboard page, open the console for the workspace you want to manage.

  2. In the Console navigation pane, click Connectors.

  3. Display the Connectors tab, and click Create Custom Connector.

  4. Upload a custom connector JAR file.

    You can upload a custom connector JAR file in any of the following ways:

    • Upload File: After clicking Click to select, select your target connector JAR file.
    • External URL: Enter the external URL address.
    note

    The external URL support only accessible HTTP links. You need to make sure that the network between the Flink service and the external HTTP service is connected.

  5. After the upload is complete, click Next. The system will parse the custom connector content you uploaded. If the parsing is successful, you can proceed to the next step. If the parsing fails, please confirm whether the custom connector code you uploaded complies with the Flink community standard.

  6. Click Finish.

Edit custom connector​

  1. On the Dashboard page, open the console for the workspace you want to manage.

  2. In the Console navigation pane, click Connectors.

  3. Display the Connectors > Custom connectors panel.

  4. Hover the pointer over the custom connector name and click Edit.

  5. If necessary, upload a new JAR file of the custom connector. Otherwise click Skip and go to the next step.

    You can use one of the following methods to upload the JAR file of a custom connector:

    • Upload File: Click Click to Select and select the desired JAR file.

    • Use External URL: Enter an external URL.

      note
      • If the external URL is the endpoint of an S3 bucket, the JAR file of the custom connector must be stored in the sql-artifacts/namespaces/{namespace} directory.
      • If the JAR file is stored in a service other than S3 and is not in the same VPC as Ververica Cloud, enter the public endpoint. Before you use the public endpoint, you must establish a connection between Ververica Cloud and the destination service. For more information, see How does the fully managed Flink service access the Internet?.

  6. Click Next.

    The system parses the content of the JAR file that you uploaded. If file parsing is successful, proceed to the next step. If file parsing fails, check whether the code of your custom connector complies with the standards that are defined by the Apache Flink community.

  7. Edit the properties of the JAR file.

  8. Click Finish.

Delete custom connector​

To delete a custom connector that is no longer used:

  1. On the Dashboard page, open the console for the workspace you want to manage.
  2. In the Console navigation pane, click Connectors.
  3. Display the Connectors > Custom connectors panel.
  4. Hover the pointer over the custom connector and click Delete.