Table

Available in VPC

A table is a metadata definition that contains detailed information and a schema for data. You can create a table through the scanner or by defining your own schema. In the Table menu, you can create and manage tables and view the collected metadata.

Table list interface

The Table menu for using Data Catalog includes the following components:

Table details interface

The Table details interface includes the following components:

Create table

You can create tables in any way you want. To create a table:

• Create tables with manual schema definition: Create tables by setting up your own database and schema.
• Create tables through scanner: Automatically define the schema through the scanner to create a table.

Create tables with manual schema definition

You can create tables by setting up your own database and schema.

To create a table with manual schema definition:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click [Create table].
4. Click Create table with manual schema definition, and then click [Next].
5. Enter basic information.
   • Database: Click the dropdown menu to select a database to connect the table.
     • You can create a database by clicking the [Create database].
     • For more information, see Create database.
   • Table name: Enter a table name.
   • Location: Where the data in the table exists.
   • Description: Enter a table description.
   • Table type
     • Catalog Default: The default Hive Table type provided by Data Catalog.
     • Apache Iceberg: An open table format for extensive analysis datasets which supports ACID transactions, schema evolution, and Time Travel queries, and helps you work safely and simultaneously in Spark, Trino, and Hive.
6. Select a data format.
   • If you select the Apache Iceberg table type, do not select a data format.
   • When you select CSV, you can select or enter the delimiter, data recognition symbol, and character to delete. Also, you can enter the number of header lines to exclude.
   • When you select XML, you can enter Row Tag.
7. Click [Add] and enter the schema information to add a user-defined schema.
   • For more information about Data type, see Schema data type.
   • Click the check box of the schema, and then click [Delete] to delete an added schema.
   • If you do not add a user-defined schema, a schema with the field name "default" will be added automatically.
   • Spaces are allowed in field names.
8. If you need to enter the partition key, click the Partition component and add a partition key.
   • After clicking [Add], enter the partition key name in the input field to add the partition key.
   • Click the check box of the partition to select it, and then click [Delete] to delete the partition key.
   • Spaces are allowed in partition key names.
   • You do not need to enter the partition for the Apache Iceberg table type.
9. If a tag is necessary, click the Set tag component to add tags.
   • After clicking [Add], enter the tag information in the input field to add the tag.
     • For more information on Data type, see Tag data types.
   • Click the check box of the tag to select it, and then click [Delete] to delete a tag.
   • Click [Load tag template] to display the popup for loading tag templates.
     • Click on a tag template to select it, and then click [Add] to add a tag of the relevant tag template.
     • For more information about tag templates, see Tag template.
10. Click [Create].

Schema data type

The data types in the schema that can be defined manually and a description of each type are as follows:

Examples of entering detailed settings for each data type are as follows:

• Example: Detailed settings of array type

  ARRAY <
     STRUCT <
        place: STRING,
        start_year: INT
     >
  >
  

• Example: Detailed settings of map type

  MAP <
     STRING,
     ARRAY<STRING>
  >
  

• Example: Detailed settings of struct type

  STRUCT <
     place: STRING,
     start_year: INT
  >
  

• Example: Detailed settings of uniontype type

  UNIONTYPE <
     INT,
     DOUBLE,
     ARRAY<STRING>,
     STRUCT<a:INT,b:STRING>
  >
  

• Example: Detailed settings of list type

  LIST <
     STRUCT <
        place: STRING,
        start_year: INT
     >
  >
  

Create table through scanner

To create a table by automatically defining the schema through the scanner:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click [Create table].
4. Click to select the check box for Create tables through scanner, and then click [Next].
   • Go to the Scanner creation interface.
5. Tables are created automatically when you create and run the scanner.
   • The table name is automatically set based on the name of the source data.
   • For more information about how to create and run a scanner, see Scanner.

Search for tables and view information

To search for the created tables and view the information:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Enter the search conditions you want and click to search for the table.
4. Click the table to view the information.
   • Database: Name of database where the table belongs.
   • Table: Table name.
   • Location: Object Storage location where the table’s data resides.
   • Table type: Table type (Catalog Default or Apache Iceberg).
   • Data type: Format of the scanned data (CSV, XML, JSON, Parquet, ORC, Avro, MySQL, MongoDB, MSSQL, and PostgreSQL).
   • Creation date and time: The date and time when a table was first created.
   • Update date and time: The most recent date and time when you edited a table's information.
   • [Schema] tab menu: Click to view the schema registered to the table.
     • For more information about Data type, see Schema data type.
     • You can edit schema by clicking the Edit. For view tables or Apache Iceberg tables, you cannot edit the schema.
   • [Schema version] tab menu: Click to view the schema version list, and then click a version to view the schema of that version.
   • [Partitions] tab menu: Click to view the table’s partition keys and values.
     • Partition update feature
       • The partition value update feature is provided for tables whose data format is CSV, XML, JSON, Parquet, AVRO, or ORC.
       • Update is available only for the Hive partition type, and for the Directory partition type, update is not available. Also, if you run "All synchronization" or "Delete-only synchronization," partition values may be deleted.
       • Only the partition value is updated, and the partition key is not added. If you want to add a partition key, you must scan the table again.
       • Options: All synchronization (update all added/deleted partition values), add-only synchronization (update added partition values only), and delete-only synchronization (update deleted partition values only).
   • [Tags] tab menu: Click to view the tags registered to the table.
     • Click the Settings to add or delete tags.
   • [Property information] tab menu: Click to view the property information of table and source data.
     • For more information about Property keys, see Property information.
   • [Analytics] tab menu: Click to view the analytics information in the field/partition unit.
     • If you subscribe to the Data Catalog service, you can run and view the analytics feature and extract analytics data, such as minimum value, maximum value, and average in field units.
     • Supported data types: parquet, avro, orc, csv, json
     • The data you can view is from the most recent successful run.
     • If you extract all analytics and then extract analytics for specific columns, those columns are updated, but the other columns remain from the previous run history.
       Caution

       • Unique value estimates the approximate number of data points within an average error range of 5%.
       • For CSV files, you cannot estimate the number of null values or true/false entries.
   • [Optimization] tab menu: Perform the Iceberg file optimization feature (only Iceberg table format displayed).
     • Merge files: A feature that improves file management efficiency and performance by merging data files that are divided into multiple files. Only files smaller than the merge threshold are selected for merging. (The default merge threshold is 100 MB, and its unit is MB.)
       Caution

       • If the merge threshold is too large, files are merged every time the merge operation runs, causing snapshots to be created even when no file changes have occurred.
       • It is recommended to set the merge threshold to an appropriate value in proportion to the data file size.
     • Manage snapshots: Delete snapshots you don't need to use. Snapshots within the maximum retention period are retained, while those beyond the retention period are deleted. (The minimum value of the maximum retention period is 7 days.)
     • Clean up orphan files: Clean up unused files, such as merged files and incorrectly written files. Orphan files within the maximum retention period are retained, while those beyond the retention period are deleted. (The minimum value of the maximum retention period is 7 days.)

Property information

If you click the [Property information] tab in the table details component, you can view the property information of the table and source data. The information items and description for each item are as follows:

Edit table

To edit the information of the created table or to select the schema version:

Edit basic information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] on the basic information component.
5. Edit the information of the table in the Edit basic information popup.
   • You can edit the location of the source data, table description, and source data format.
6. Click [Save].

Edit schema information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] under the Schema tab.
5. When the Edit schema interface appears, you can edit the field name, data type, and description. You can also edit them manually using the Edit JSON button.
   • Click the Version dropdown menu in the Schema component to select the schema version you want to edit.
   • When you edit JSON, the name, type, typeValue, and description items must exist as follows:

   [
     {
       "name": "col_name",
       "type": "decimal",
       "typeValue": "(10,2)",
       "description": "catalog decimal"
      }
    ]
   

6. Click [Save].

Edit property information

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the table name to go to the Table details interface.
4. Click [Edit] under the [Property information] tab menu.
5. You can edit inputFormat, outputFormat, and serializationLib to use in the table.
   • Note that if you edit it to a library that is not compatible with the data format, you will not be able to run queries in Data Query, Hive, or Spark.
6. Click [Save].

Delete table

To delete a created table:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to Menu > Services > Big Data & Analytics > Data Catalog.
2. Click the Table menu.
3. Click the name of the table you want to delete to go to the Table details interface.
4. Click [Delete].
5. When the notification popup appears, read the cautions and click [Delete].