Table

Available in VPC

Table is a metadata definition with details and schema of the data. You can create a table through the scanner or by defining your own schema. In the Table menu, you can create and manage tables and view the collected metadata.

Table list interface

A basic description of the Table menu for using Data Catalog is as follows:

Table details interface

A basic description of the Table details interface is as follows:

Create table

You can create tables in any way you want. To create a table:

• Create table with manual schema definition: Create tables by setting up your own database and schema.
• Create table through scanner: Automatically define the schema through the scanner to create a table.

Create table with manual schema definition

You can create tables by setting up your own database and schema.

To create a table with manual schema definition:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the [Create table].
4. Click Create table with manual schema definition, and then click [Next].
5. Enter basic information.
   • Database: Click the dropdown menu to select a database to connect to the table.
     • Click the [Create database] to create a database (see Create database).
   • Table name: Enter a table name.
   • Location: Where the data in the table exists.
   • Description: Enter a table description.
   • Table type
     • Catalog Default: It is the default Hive Table type provided by Data Catalog.
     • Apache Iceberg: It is an open table format for the extensive analysis data set, which supports ACID transactions, schema evolution, and Time Travel queries and helps work safely and simultaneously in Spark, Trino, and Hive.
6. Select a data format.
   • If you select the Apache Iceberg table type, do not select a data type.
   • When you select CSV, you can select or enter the delimiter, data recognition symbol, and character to delete. Also, you can enter the number of header lines to exclude.
   • When you select XML, you can enter Row Tag.
7. Click the [Add] and enter the schema information to add a user-defined schema.
   • For more information about Data type, see Schema data type.
   • Click the check box of the schema, and then click the [Delete] to delete an added schema.
   • If you do not add a user-defined schema, a schema with the field name "default" will be added automatically.
   • Spaces are allowed in field names.
8. If you need to enter the partition key, click the Partition component and add a partition key.
   • After clicking the [Add], enter the partition key name in the input field to add the partition key.
   • Click the check box of the partition to select it, and then click the [Delete] to delete the partition key.
   • Spaces are allowed in partition key names.
   • You do not need to enter the partition for the Apache Iceberg table type.
9. If a tag is necessary, click the Set tag component to add tags.
   • After clicking the [Add], enter the tag information in the input field to add the tag.
     • For more information about Data type, see Tag data types.
   • Click the check box of the tag to select it, and then click the [Delete] to delete a tag.
   • Click the [Load tag template] to display the popup for loading tag templates.
     • Select and click a tag template, and then click the [Add] to add a tag of the relevant tag template.
     • For more information about tag templates, see Tag template.
10. Click [Create].

Schema data type

The data types in the schema that can be defined manually and a description of each type are as follows:

Examples of entering detailed settings for each data type are as follows:

• Example: Detailed settings of array type

  ARRAY <
     STRUCT <
        place: STRING,
        start_year: INT
     >
  >
  

• Example: Detailed settings of map type

  MAP <
     STRING,
     ARRAY<STRING>
  >
  

• Example: Detailed settings of struct type

  STRUCT <
     place: STRING,
     start_year: INT
  >
  

• Example: Detailed settings of uniontype type

  UNIONTYPE <
     INT,
     DOUBLE,
     ARRAY<STRING>,
     STRUCT<a:INT,b:STRING>
  >
  

• Example: Detailed settings of list type

  LIST <
     STRUCT <
        place: STRING,
        start_year: INT
     >
  >
  

Create table through scanner

To create a table by automatically defining the schema through the scanner:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the [Create table].
4. Click Create tables through scanner, and then click [Next].
   • Go to the Scanner creation interface.
5. Tables are created automatically when you create and run the scanner.
   • The table name is automatically set based on the name of the source data.
   • For more information about how to create and run a scanner, see Scanner.

Search for tables and view information

To search for the created tables and view the information:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Enter the search conditions you want and click to search for the table.
4. Click the table to view the information.
   • Database: Name of the database where the table belongs.
   • Table: Table name.
   • Location: The Object Storage location where the data of the table exists.
   • Table type: Type of the table (Catalog Default and Apache Iceberg).
   • Data format: Format of the scanned data (CSV, XML, JSON, Parquet, ORC, AVRO, MySQL, MongoDB, MSSQL, and PostgreSQL).
   • Creation date and time: The date and time when a table was first created.
   • Update date and time: The most recent date and time when you edited a table's information.
   • [Schema]: View the schema registered to the table.
     • For more information about Data type, see Schema data type.
     • You can edit schema by clicking the Edit button. For view tables or Apache Iceberg tables, you cannot edit the schema.
   • [Schema version]: Click to view the schema version list, and then click a version to view the schema of that version.
   • [Partition]: Click to check the partition key and value registered to the table.
     • Partition update feature
       • The partition value update feature is provided only for tables whose data format is CSV, XML, JSON, Parquet, AVRO, or ORC.
       • Update is available only for the Hive partition type, and for the Directory partition type, update is not available. Also, if you run "All synchronization" or "Delete-only synchronization," the partition value may be deleted.
       • Only the partition value is updated, and the partition key is not added. If you want to add a partition key, you must scan the table again.
       • Options: All synchronization (update all added/deleted partition values), add-only synchronization (update added partition values only), and delete-only synchronization (update deleted partition values only).
   • [Tag]: Click to view the tags registered to the table.
     • Click the Settings button to add or delete tags.
   • [Property information]: Click to view the property information about table and source data.
     • For more information about Property keys, see Property information.
   • [Analytics]: Click to view the analytics information in the field/partition unit.
     • If you subscribe to Data Catalog, you can run and view the analytics feature and extract analytics data, such as minimum value, maximum value, average, and so on, in field units.
     • Supported data types: Parquet, AVRO, ORC, CSV, and JSON.
     • The data you can view is from the most recent successful run.
     • If you extract all analytics and then extract analytics for specific columns, those columns are updated, but the other columns remain from the previous run history.
       Caution

       • Unique value estimates the approximate number of data points within an average error range of 5%.
       • For CSV files, you cannot estimate the number of null values or true/false entries.
   • [Optimization]: Perform the Iceberg file optimization feature (only Iceberg table format exposed).
     • Merge files: A feature that combines data files that have been divided into multiple files for more efficient file management and improved performance. Only files smaller than the merge threshold are selected for merging. (The default merge threshold is 100 MB, and its unit is MB.)
       Caution

       • If the merge threshold is too large, files are merged every time the merge operation runs, causing snapshots to be created even when no file changes have occurred.
       • It is recommended to set the merge threshold to an appropriate value in proportion to the data file size.
     • Manage snapshots: Delete snapshots you don't need to use. Snapshots within the maximum retention period are retained, while those beyond the retention period are deleted. (The minimum value for the maximum retention period is 7 days.)
     • Manage orphan files: Organize unused files, such as merged files or incorrectly written files. Orphan files within the maximum retention period are retained, while those beyond the retention period are deleted. (The minimum value for the maximum retention period is 7 days.)

Property information

If you click the [Property information] tab in the table details component, you can view the property information of the table and source data. The information items and description for each item are as follows:

Edit table

To edit the information of the created table or to select the schema version:

Edit basic information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] on the basic information component.
5. Edit the information of the table in the Basic information edit popup.
   • You can edit the location of the source data, table description, and source data format.
6. Click [Save].

Edit schema information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] on the schema tab.
5. When the Edit schema interface appears, you can edit the field name, data type, and description. You can also edit them manually using the Edit JSON button.
   • Click the Version dropdown menu in the Schema component to select the schema version you want to edit.
   • When you edit JSON, the name, type, typeValue, and description items must exist as follows:

   [
     {
       "name": "col_name",
       "type": "decimal",
       "typeValue": "(10,2)",
       "description": "catalog decimal"
      }
    ]
   

6. Click [Save].

Edit property information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] on the property information tab.
5. You can edit inputFormat, outputFormat, and serializationLib to use in the table.
   • Note that if you edit it to a library that is not compatible with the data format, you will not be able to run queries in Data Query, Hive, or Spark.
6. Click [Save].

Delete table

To delete the created table:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the name of the table you want to delete to go to the Table details interface.
4. Click [Delete].
5. When the notification popup appears, read the cautions and click [Delete].