Connectors

Available in VPC

You can create, view, edit, and delete integration connectors from the Topic menu. Topic connectors allow you to integrate with external systems and transfer or import data stored in a topic to or from external systems. Since the connector operates via the polling method, where consumers retrieve data from topics at regular intervals, it enables more stable data processing compared to the push method.

Set connector

To set a connector:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Stream.
2. Click the Topic menu.
3. In the topic list, click View service details.
4. On the topic details interface, click the [Connector] tab menu.
5. Click [Set connector].
6. When the Set connector popup appears, enter basic information and set detailed options.
   • Basic information
     • Name: This is entered automatically and cannot be edited by the user.
     • Description: Enter a description of the connector between 0 and 500 bytes.
   • Detailed settings
     • Consumer specifications: Select the specifications for connector processing performance.
       • Small: Supports up to 250 Kbytes throughput per second per partition based on 1 container.
       • Medium: Supports up to 500 Kbytes throughput per second per partition based on 2 containers.
     • Target product: Click to select the service on NAVER Cloud Platform where you want to store your data.
     • Storage path: Click [Set] to select the Object Storage bucket where you want to store the data, then click [Save].
       • Double-click on the bucket name to move to a subdirectory.
       • Use topic names in paths: Click to select whether to create a topic name directory in the Object Storage storage path.
     • Use date format: Click to select whether to use the date format directory in the Object Storage sub-storage path.
       • None: Do not use date format.
       • year={yyyy}/: Include directory in year format.
       • year={yyyy}/month={MM}/: Include directory in year/month format.
       • year={yyyy}/month={MM}/day={dd}/: Include directory in year/month/day format.
       • year={yyyy}/month={MM}/day={dd}/hour={HH}/: Include directory in year/month/day/hour format.
   • Option
     • Flush: Set the data processing policy in the connector.
       • The data is processed according to the condition that is met first among the following conditions. Processing time may vary internally depending on the data size (MB).
       • Interval: The data processing condition based on time (default: every 10 minutes, minimum: 1 minute/maximum: 10 minutes).
       • Count: The data processing condition based on the number of rows (default: every 5,000 rows, minimum: 5,000/ maximum: 50,000).
     • Schema format
       • When you read message Key or Value data of the topic, select the Deserializer class you are using.
       • Classes (AVRO, JSON, and PROTOBUF), excluding STRING, require schema integration.
         • Data that does not conform to the schema may not be processed properly.
         • Result files are created depending on the schema format as follows:
           • STRING class: The .json extension.
           • AVRO,JSON, and PROTOBUF classes: The.parquet extension.
   • Role name: Select the service role that the connector will use to access data by clicking on it in the dropdown menu.
     • [Create new role]: Click Sub Account > Roles to create a new role.
     • For more information about how to create roles, see Sub Account user guide.
7. Click [Save].
   • The connector information set on the interface is displayed.
     • Recent running status: The connector's recent running status.
       • Active: The connector is running normally.
     • Creation date and time: The date and time connector was created (YYYY-MM-DD HH:mm:ss (UTC+09:00 format)).

Check connector information

To see a connector's information:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Stream.
2. Click the Topic menu.
3. In the topic list, click on View service details of the topic you want to check the connector information of.
4. On the topic details interface, click the [Connector] tab menu.
5. When the interface appears, check the connector information.
   • For more information about the connector information, see Set connector.

Edit connector

To edit a connector:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Stream.
2. Click the Topic menu.
3. In the topic list, click on View service details of the topic you want to edit the connector of.
4. On the topic details interface, click the [Connector] tab menu.
5. When the interface appears, edit the necessary information.
   • For more information about the connector information, see Set connector.

Delete connector

To delete a connector:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Stream.
2. Click the Topic menu.
3. In the topic list, click View service details.
4. On the topic details interface, click the Connector tab.
5. Click [Delete].
6. Check the details in the Delete notification popup, and click [Delete].
   • The deleted connector's information disappears from the interface.