Variables and secret value keys

Ververica Cloud allows you to configure variables and keys to prevent security risks that are caused by the use of information such as plaintext AccessKey pairs and passwords. You can reuse variables and keys to avoid repeatedly writing the same code or values and simplify configuration management. You can reference a variable or a key in various scenarios, such as the development of SQL, JAR, or Python drafts, log output configuration, and UI-based parameter configuration. This topic describes how to create and reference a variable or a key.

Background information

Variables and secret value keys are both composed of a name and a corresponding value. The name should reflect the meaning of value and is completely customizable. The value is the actual data referenced by a deployment.

There are some important distinctions between a variable and a secret value key:

• Variable: A variable can be referenced in only one deployment. You can reference variables in the DDL statements of SQL drafts and the input parameters of a Main function of JAR and Python drafts. Variables must be in the ${Variable name} format.
• Key: A key can be referenced in only one namespace. can reference keys when you write DDL statements of SQL drafts and create catalogs on the UI. Keys must be in the ${secret_values.Key_name} format.

Variables

Before getting started with Variables, note that within a single draft, each variable name must be unique.

Reference a variable in an SQL draft

You can reference a variable when you develop an SQL draft and change the variable value when you start the deployment.

1. In the development console, click the SQL Editor in the left-side navigation pane.
2. In the Drafts tab, select your desired draft and input variables in the ${Variable name} format in the SQL Editor. The following code provides an example using variables in several places within a SQL draft.

info

• If you reference a variable in the WITH clause, you must enclose the variable with a pair of single quotation marks (').
• If you reference a variable as a table name or field name in an SQL statement, you must enclose the variable with a pair of grave accents (`).

create temporary table `${source_name}`(
    id varchar,
    name varchar
) with (
    'connector' = 'datagen'
);


create temporary table blackhole(
    id varchar,
    `${test_name}` varchar
) with (
    'connector' = '${blackhole}'
);

insert into blackhole select * from `${source_name}`;

3. Ververica Cloud automatically identifies variables within the draft that are in the required ${Variable name} format. You can assign the variables' default values in one of two methods:

• Method 1: Enter the variable value in the Variables panel of the SQL Editor.

• Method 2: Click the Configurations tab on the right side of the SQL Editor. Enter the default variable values in the Variables section.

4. After deploying the draft, view the deployment variables in the Configuration tab of the Deployments page.

5. On the Deployments page, find the desired deployment and click Start in the Actions column to change variable values.

Reference a variable in a JAR or Python draft

In addition to SQL drafts, you can also reference a variable in the parameters of a Main function of a JAR or Python draft.

1. In the Ververica Cloud console, navigate to Deployments on the left side pane, followed then by clicking Create Deployment.
2. Enter variable names in the Entry Point Main Arguments field based on the format, -- input ${test1}${test2}${test3}, with test1, test2, and test3 each being respective variable names. Aftewards, enter the variable values in the Variables section below.

3. In the Create Deployment dialog box, click Deploy.
4. On the Deployments page, find the desired deployment and click Start in the Actions column to change variable values.

Add a secret value

You can only add and delete keys. You cannot change or view the values of keys.

To configure a secret value key:

1. Go to Security → Secret Values.
2. Click Add Secret Value.
3. Fill in Secret Name and Secret Value. Note that they both are case-sensitive. Then click OK.

Within our namespace, we are now able to reference a key named secret_key1. After you configure a Secret Value, you can reference the key in SQL scripts or when creating a new catalog for a job in the console of fully managed Flink.

Reference a secret value key

After you configure a key, you can reference the key in SQL scripts and in catalog creation.

Wherever you use the key, use the ${secret_values.Key_name} format.

Reference the key in the DDL statements in SQL scripts

1. Go to SQL Editor.
2. Reference the key in the SQL scripts. Note that accessKey has a value assigned from our key created previously, secret_key1.

Sample code:

    CREATE TEMPORARY table datahub_source(
      name VARCHAR
    ) WITH (
      'connector' = 'datagen'
    );

    CREATE TEMPORARY table datahub_sink(
      name  VARCHAR 
    ) WITH (
      'connector'='datahub',
      'endpoint'='<yourEndpoint>',
      'project'='<yourProject>',
      'topic'='<yourTopic>',
      'accessId'='<yourAccessId>',
      'accessKey'='${secret_values.secret_key1}'
    );

    INSERT INTO datahub_sink
    SELECT
      LOWER(name)
    from datahub_source;

Reference a secret value key in the Deployment Logging section

1. Navigate to the Logging section of your deployment.
2. Reference a key in the Logging section. The following sample code provides an example on how to reference a key named accessKeyId and a key named accessKeySecret when you configure parameters to export the logs of a deployment to Simple Log Service.

<Appender name="SLS" type="SLS">
  <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS}{GMT+8} %-5p %-60c %x - %m%n" type="PatternLayout" charset="UTF-8"/>  

  <!-- The final effective log path is: ${baseUri}/logs/${namespace}/${deploymentId}/{jobId}/ -->
  <Property name="namespace">{{ namespace }}</Property> <!-- Do not modify this line -->
  <Property name="project">YOUR-SLS-PROJECT</Property>  
  <Property name="logStore">YOUR-SLS-LOGSTORE</Property> 
  <Property name="endpoint">YOUR-SLS-ENDPOINT</Property> 
  <Property name="accessKeyId">${secret_values.accessKeyId}</Property> 
  <Property name="accessKeySecret">${secret_values.accessKeySecret}</Property> 
  <Property name="topic">{{ namespace }}:{{ deploymentId }}:{{ jobId }}</Property>
  <Property name="deploymentName">{{ deploymentName }}</Property>
  <Property name="flushIntervalSeconds">10</Property>
  <Property name="flushIntervalEventCount">100</Property>
</Appender>

Reference a secret value key in Catalog UI

When you create a catalog in Ververica Cloud's console, you can reference a namespace secret value key.

1. In the left-side pane, click Catalogs.
2. After selecting Create Catalog, fill in the required details and then reference a secret value key as the password. The following example shows us creating a new MySQL catalog with the key named secret_key1 from earlier.