Using Kibana and OpenSearch

Available in VPC

Kibana and OpenSearch Dashboards are open-source interfaces for visualizing and exploring data stored in Elasticsearch and OpenSearch clusters. You can use data to create various charts and manage clusters. Search Engine Service supports Kibana version 7.7.0 for Elasticsearch clusters and OpenSearch Dashboards version 1.3.3 for OpenSearch clusters. This guide introduces the following features:

Getting started with Kibana: How to access Kibana and an overview of the interface.
Getting started with OpenSearch Dashboards: How to access OpenSearch Dashboards and an overview of the interface.
Dev Tools: Introduction to the Console in the Dev Tools for easily calling APIs.
Data management: How to store and retrieve data using APIs, create index patterns, and search data in the Discover menu.
Manage snapshot: How to create snapshots using APIs, identify causes of snapshot creation failures, restore data from snapshots, and delete snapshots.

Note

This guide describes only selected Kibana and OpenSearch Dashboards features that are commonly used to work with Search Engine Service.
For more information on Kibana, see Kibana Guide.
For more information on OpenSearch Dashboards, see OpenSearch Dashboards.

Getting started with Kibana

To use Kibana, you must first access Kibana. Log in to explore various menus in Kibana.

Access Kibana

When you create an Elasticsearch cluster, Kibana is installed on the manager node and starts automatically.

Note

Before accessing Kibana from an external source, check the following:

You must configure the manager node access. Follow steps 1 through 4 in Connect to manager node via SSH, from creating a target group to configuring the manager node ACG.
Check the IP address of the load balancer connected to the manager node
- In the console, navigate to > Services > Networking > Load Balancer > Load Balancer to check the connection details for the load balancer.
Check your Kibana access ID and password.
- You can view your ID and set the password in Cluster > Manage cluster > Reset access password.

To access Kibana from outside:

In your web browser, connect to load balancer IP:80.
- This accesses port 5601, which is forwarded to port 80.
In the following screen, enter the Kibana access information and click [Log in].
- Username: Enter the Kibana access ID.
- Password: Enter the Kibana access password.
- In the first interface after login, click [Try our sample data] to use the sample data.

Kibana interface

The Kibana interface includes the following:
ses-kibana-vpc_screen

Component	Description
① Home, menu name	: Go to the Home interface. Name of the menu currently in use.
② Kibana menu	Recently viewed: View recently accessed data. Discover: Search stored data (see Search data). Visualize: Visualize data and add it to a dashboard. Dashboard: View visualization tools and search results at a glance. Dev Tools: Manage data using APIs (see Dev Tools). Management: Manage index patterns, saved objects, and advanced Kibana settings.
③ Kibana information	: Click to view the version of your Kibana, Kibana guides and links to external resources. : Click to view Kibana news.

Getting started with OpenSearch Dashboards

To use OpenSearch Dashboards, you must first access OpenSearch Dashboards. Log in to explore various menus in OpenSearch Dashboards.

Access OpenSearch Dashboards

When you create an OpenSearch cluster, OpenSearch Dashboards is installed on the manager node and starts automatically.

Note

Before accessing Kibana from an external source, check the following:

You must configure the manager node access. Follow steps 1 through 4 in Connect to manager node via SSH, from creating a target group to configuring the manager node ACG.
Check the IP address of the load balancer connected to the manager node
- In the console, navigate to > Services > Networking > Load Balancer > Load Balancer to check the connection details for the load balancer.
Check your Kibana access ID and password.
- You can view your ID and set the password in Cluster > Manage cluster > Reset access password.

To access OpenSearch Dashboards from outside:

In your web browser, connect to load balancer IP:80.
- This accesses port 5601, which is forwarded to port 80.
In the following screen, enter the OpenSearch Dashboards access information and click [Log In].
- Username: Enter your OpenSearch Dashboards access ID.
- Password: Enter your OpenSearch Dashboards access password.
- In the first interface after login, you can use sample data by clicking [Add data].

OpenSearch Dashboards interface

The OpenSearch Dashboards interface includes the following:
ses-kibana-vpc_opensearchscreen

Component	Description
① Basic features	Home: Go to Home. Recently viewed: View recently accessed data.
② OpenSearch Dashboards	Overview: Overview of OpenSearch Dashboards. Discover: Search stored data (see Search data). Dashboard: View visualization tools and search results at a glance. Visualize: Visualize data and add it to a dashboard.
③ OpenSearch Plugins	Query Workbench: Run SQL queries. Reporting: Create reports in PDF or PNG format. Alerting: Configure alerts. Anomaly Detection: Detect anomalies. Observability: Visualize data-driven events using Pipeline Processing Language (PPL). Index Management: Automate index management. Security: Manage authentication, access control, and accounts.
④ Management	Dev Tools: Manage data using APIs (see Dev Tools). Stack Management: Manage index patterns, saved objects, and advanced OpenSearch Dashboards settings.

Dev Tools

You can use the console in the Dev Tools menu to easily work with APIs and view request results. The Dev Tools interface includes the following:
ses-kibana-vpc_console

Component	Description
① Console features	History: View and rerun previously executed commands. Settings: Configure the Console environment. Help: View keyboard shortcuts available in the console.
② Input component	Component for entering commands : Click to execute the command. Press [Ctrl]+[Enter] to run the command. : Click to display the following menu: Copy as cURL: Copy your command in cURL format to the clipboard. Open documentation: View documentation for the API related to your command. Auto indent: Automatically add or remove indentation in your command.
③ Result component	Displays the result of the command execution.

Manage data

You can manage data using Kibana and OpenSearch Dashboards. From the console in the Dev Tools menu, you can store and view data through APIs. You can also create index patterns and search data stored in the cluster.

Store data

To store data using APIs:

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, call the API to store data.
- The result is displayed on the right pane.
- Example 1: Store data with the name and dept fields in the test index.
```
PUT test/_doc/1
{
    "name" : "asd",
    "dept" : "Document Management"
}
```
- Example 2: Create an index with 3 primary shards and 1 replica shard, with no data.
```
PUT my_index
{
"settings": {
   "index": {
      "number_of_shards": 3,
      "number_of_replicas": 1
   }
}
}
```

View data

To view data stored in the cluster via the API:

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, call the API to view the data.
- The results are displayed on the right pane.
- Example: View data in text index.
```
GET test/_search
{
    "query" : {
        "match_all" : {}
    }
}
```

Create index pattern

To use indices in Kibana and OpenSearch Dashboards, you must first create an index pattern. Using index patterns, you can view and manage data from multiple indices at once.

Note

To create an index pattern, data must already be stored in the cluster.

To create an index pattern:

In Kibana or OpenSearch Dashboards, click the Management or Stack Management menu.
Click the Index Patterns menu.
Click [Create index pattern].
Enter the index pattern for viewing data in Kibana or OpenSearch Dashboards.
- Indices that match the index pattern are displayed.
Click [Next step].
Review the Configure settings, then click [Create index pattern].
- If date or time fields are available in the index, select the appropriate field in the Time Filter field name or Time field.
- Details of the created index pattern are displayed.

Search data

You can search data stored in the cluster from the Discover menu in Kibana and OpenSearch Dashboards.

Note

To search data, you must first create an index pattern.

To search data in the cluster:

In Kibana or OpenSearch Dashboards, click the Discover menu.
From the dropdown list, select the index pattern to view.
- Data from the indices included in the selected index pattern is displayed.
Enter a search term (for example, name:asd) in the search field and press [Enter] to view the search results.
- Select fields to display only the data you need.

Manage snapshot

You can create and delete snapshots using the console in the Dev Tools menu of Kibana and OpenSearch Dashboards. If snapshot creation fails, you can identify the cause, or restore data using an existing snapshot.

Note

Snapshot features are not available in the Japan region.
To create and manage snapshots in Object Storage, you must configure an API authentication key.

Create Snapshot using Elasticsearch and OpenSearch APIs

If you want to change the folder where snapshots are stored, or if snapshot creation fails in the console due to issues such as a timeout, you can create snapshots directly using the Elasticsearch and OpenSearch APIs in Kibana or OpenSearch Dashboards.

Note that snapshots created through the Elasticsearch or OpenSearch APIs are not shown in the console.

To create a snapshot using the Elasticsearch or OpenSearch APIs:

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, run the following command to create a repository.
```
PUT _snapshot/my-own-snapshot-repository
{    
    "type" : "s3",
    "settings": {        
                "region" : "kr",
                "endpoint" : "https://kr.object.private.ncloudstorage.com", 
                "bucket": "estest",  
                "compress": true,
                "base_path": "my-folder",  
                "path_style_access": true
    }
}
```
- my-own-snapshot-repository: Name the repository.
- base_path: Name the folder.
  - If you enter a folder name that is already used by another cluster, snapshot files may fail or behave unexpectedly.
- region
  - Korea region: kr
  - Singapore region: sgn
- endpoint
  - Korea region: https://kr.object.private.ncloudstorage.com
  - Singapore region: https://sg.object.private.ncloudstorage.com
In the console, run the following command to call the snapshot creation API.
```
POST _snapshot/my-own-snapshot-repository/my-own-snapshot-name
```
- my-own-snapshot-repository: Enter the repository name created in step 2.
- my-own-snapshot-name: Name the snapshot.
- Specify indices when calling the API as shown below to create snapshots only for the specified indices.
```
PUT _snapshot/my-own-snapshot-repository/my-own-snapshot-name
{
"indices": "myindex, myindex2"
}
```
- The snapshot is stored at the specified path in Object Storage.

Identify cause of snapshot creation failure

To identify the cause of snapshot creation failures in Kibana and OpenSearch Dashboards:

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, run the GET _snapshot command to check the repository of a failed snapshot.
- The list of repositories is displayed on the right pane.
Run the following command to view the detailed status of the snapshot.
```
GET _snapshot/"snapshot repository"/"snapshot name"
```

Restore snapshot

You can use snapshots to restore cluster data or migrate data from one cluster to another.

Note

Ensure that a snapshot exists before attempting a restore. (see Create snapshot or Create snapshot using Elasticsearch and OpenSearch APIs).

To restore a snapshot:

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, run the following command to create a repository to restore.
```
PUT _snapshot/restore-repository
{  
"type" : "s3",  
"settings": {    
   "region" : "kr",    
   "endpoint" : "https://kr.object.private.ncloudstorage.com",
   "bucket": "estest",
   "compress": true,
   "base_path": "estest-1722213",
   "path_style_access": true
}
}
```
- region
  - Korea region: kr
  - Singapore region: sgn
- endpoint
  - Korea region: https://kr.object.private.ncloudstorage.com
  - Singapore region: https://sg.object.private.ncloudstorage.com
- bucket: Name of the bucket where the snapshot to restore is stored.
- base_path: Folder name where the snapshot is located (cluster-name-cluster-number).
- Run the following command to verify that the repository was created successfully.
```
GET _snapshot
```
- Run the following command to view the list of snapshots stored in the repository.
```
GET _cat/snapshots/restore-repository
```
Run the following command to view the list of indices available for restoration in the snapshot.
```
GET _snapshot/restore-repository/"snapshot name"
```
Run the following command to view the list of indices currently present in the cluster.
- Indices cannot be restored if an index with the same name already exists in the current cluster.
```
GET _cat/indices
```
Run the command to restore indices.
- Restore a specific index
  - Example: Restore metricbeat-7.7.0-2021.06.23.
```
POST _snapshot/restore-repository/test4/_restore
{
"indices": "metricbeat-7.7.0-2021.06.23",
"ignore_unavailable": true
}
```
- Restore multiple indices using a wildcard (*)
  - Example: Restore all indices starting with metricbeat.
```
POST _snapshot/restore-repository/test4/_restore
{
"indices": "metricbeat*",
"ignore_unavailable": true
}
```
- Restore all indices except for specific ones.
  - Example: Restore all indices except kibana.
```
POST _snapshot/restore-repository/test4/_restore
{
"indices": "-.kibana*",
"ignore_unavailable": true
}
```
- Setting this to "indices": "*" will trigger a restore for all indices, but duplicate indices will cause the operation to fail.
Run the following command to check if the indices were restored successfully.
- Immediately after restoration, the index status may temporarily be yellow, and the cluster status may show a warning.
```
GET _cat/indices
```

Delete snapshot

In Kibana or OpenSearch Dashboards, click the Dev Tools menu.
In the console, run the following command to delete a snapshot.
```
DELETE _snapshot/restore-repository/test4
```
- You can run the following command to verify that the snapshot has been deleted.
```
GET _cat/snapshots/restore-repository
```

Note

When deleting a large snapshot, a timeout response may occur. This happens when the operation exceeds 30 seconds, but the snapshot deletion continues and completes normally in the background.