JSON

This article introduces you to the usage and type mapping of the JSON format.

Background information

JSON format can read and write JSON data based on the JSON structure. Currently, the JSON structure is automatically deduced from the table structure.

Example of use

An example of constructing a table using Kafka and JSON format is as follows.

    CREATE TABLE user_behavior (
        user_id BIGINT,
        item_id BIGINT,
        category_id BIGINT,
        behavior STRING,
    ts TIMESTAMP(3)
    ) WITH (
       'connector' = 'kafka',
       'topic' = 'user_behavior',
       'properties.bootstrap.servers' = 'localhost :9092',
       'properties.group.id' = 'testGroup',
       'format' = 'json',
       'json.fail-on-missing-field' = 'false',
       'json.ignore-parse-errors' = 'true'
    );

Cofiguration option

Parameter	Required	Default	Type of data	Description
format	yes	(none)	String	The format to use for the declaration. When using the JSON format, the parameter value is json.
json.fail-on-missing-field	no	false	Boolean	The parameter values are as follows: true: Skip current field or row when parsing field is missing. false (default): When parsing fields are missing, an error is thrown and the job fails to start.
json.ignore-parse-errors	no	false	Boolean	The parameter values are as follows: true: When parsing exceptions, skip the current field or row. false (default): When parsing an exception, an error is thrown and the job fails to start.
json.timestamp-format.standard	no	SQL	String	Specifies the input and output timestamp format. The parameter values are as follows: SQL: Parse input timestamps in yyyy-MM-dd HH:mm:ss.s{precision} format, such as 2020-12-30 12:13:14.123, and output timestamps in the same format. ISO-8601: Parse input timestamps in yyyy-MM-ddTHH:mm:ss.s{precision} format, such as 2020-12-30T12:13:14.123, and output timestamps in the same format.
json.map-null-key.mode	no	fail	String	Specifies the method to handle empty key values in the Map. The parameter values are as follows: FAIL: An exception is thrown when the key value in the Map is empty. DROP: Discard the data whose key value in the Map is empty. LITERAL: Use string constants to replace empty key values in the Map. The value of the string constant is defined by canal-json.map-null-key.literal.
json.map-null-key.literal	no	null	String	When the parameter of json.map-null-key.mode is LITERAL, specify a string constant to replace the empty key value in the Map.
json.encode.decimal-as-plain-number	no	false	Boolean	The parameter values are as follows: true: All data of DECIMAL type remain as they are, and are not expressed in scientific notation, for example: 0.000000027 is expressed as 0.000000027. false: All data of DECIMAL type are expressed in scientific notation, for example: 0.000000027 is expressed as 2.7E-8.

Type mapping

Currently, the JSON structure will be automatically deduced from the table structure. In Flink, the JSON format uses the jackson databind API (jackson databind API) to parse and generate JSON. The mapping relationship between Flink and JSON data types is as follows.

Flink SQL type	JSON type
CHAR/VARCHAR/STRING	string
BOOLEAN	boolean
BINARY / VARBINARY	string with encoding: base64
DECIMAL	number
TINYINT	number
SMALLINT	number
INT	number
BIGINT	number
FLOAT	number
DOUBLE	number
DATE	string with format: date
time	string with format: time
TIMESTAMP	string with format: date-time
TIMESTAMP_WITH_LOCAL_TIME_ZONE	string with format: date-time (with UTC time zone)
INTERVAL	number
ARRAY	array
MAP / MULTISET	object
ROW	object

Other Instructions for Use

For writing to object storage S3, currently it does not support writing files in JSON format. For specific reasons, see FLINK-30635 .

note

This page is derived from the official Apache Flink® documentation.

Refer to the Credits page for more information.