Replace Search Engine Cluster using Logstash

Available in VPC

This section describes the process of rotating Search Engine Service Clusters from CentOS-based systems to Rocky Linux (or other operating systems) using Logstash.
The example shown in this section addresses how to operate LogStash in the server and replicate and replace Search Engine Service Cluster's data.

Preparations

Preparations before proceeding with this guide are as follows:

• Check existing Search Engine Service Cluster information (Cent Linux installed version)
• Check newly created Search Engine Service Cluster information (Rocky Linux installed version)

Cluster transition order

Guide the order to transfer to the newly created cluster.

1. Complete creating VPC Server, installing and applying LogStash, and setting ACG, following the next guide.
2. Search index information and validate whether the newly created cluster's data is replicated properly through the server with LogStash installed.
3. Change the endpoint of the application server that searches the saved data of the cluster to the newly created cluster.
4. Change the endpoint of the application server that saves data to the cluster to the newly created cluster.
5. Once you change the endpoint of the application server, compare the index and the number of documents between the existing cluster and the new cluster to check/validate that the transfer is completed.
6. Remove the VPC Server where LogStash is installed.

Creating VPC Server

Step 1. Create server

Create a server on which LogStash is to be installed.

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Compute > Server in order.
2. Click the [Create server] button to create a server. (Server creation guide)
3. When creating servers, use Subnets of the same type (Private or Public) as the Cluster's Manager Node, or identical to those of the Manager Node.
4. Connect to the created server through SSH to prepare for Logstash installation. (Server access guide)

Installing LogStash

This section introduces an example of installing LogStash on a server.
The installation includes processes of ElasticSearch or OpenSearch.
In order to test normally, install the proper version that works with the version you are using.

Step 1. Install java

1. Install java by entering the following command:

sudo yum install java-devel -y

Step 2. Install LogStash

To install LogStash, follow these steps:
You need to install the proper version of LogStash compatible with the version of Elasticsearch or OpenSearch.

1. Enter the following command to download LogStash to the /root path:

# For Elasticsearch version (install OSS version)
wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.7.0.rpm

# For OpenSearch version
wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-7.16.3-linux-x64.tar.gz

2. Here is the command to install the downloaded file:

# For Elasticsearch version
rpm -ivh logstash-oss-7.7.0.rpm

# For OpenSearch version
tar -zxvf logstash-oss-with-opensearch-output-plugin-7.16.3-linux-x64.tar.gz

3. Define the tasks that LogStash is to perform by referring to the following command and configuration file:

• For ElasticSearch version

mv /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/logstash.conf
vi /etc/logstash/conf.d/logstash.conf

• ElasticSearch version LogStash.conf

input {
  elasticsearch {
    hosts => ["http://A-cluster-address:9200"]
    index => "your-index-*"
  }
}

output {
  elasticsearch {
    hosts => ["http://B-cluster-address:9200"]
    index => "%{[index]}"
  }
}

• For OpenSearch version

# When you install LogStash to /root/, the {installation path} is /root/logstash-7.16.3.
mv {installation path}/config/logstash-sample.conf {installation path}/config/logstash.conf
vi {installation path}/config/logstash.conf

• OpenSearch version LogStash.conf

input {
  opensearch {
    hosts => ["http://A-cluster-address:9200"]
    index => "your-index-*"
    user => ${userID}
    password => ${password}
    ssl_certificate_verification => false
  }
}

output {
  opensearch {
    hosts => ["http://B-cluster-address:9200"]
    index => "%{[index]}"
    user => ${userID}
    password => ${password}
    ssl_certificate_verification => false
  }
}

• Description of each item of LogStash

["http://A-cluster-address:9200"] - enter the information of the subscribed cluster. 
["http://B-cluster-address:9200"] - enter the information of the newly created cluster.
"your-index-*" - name of the index to be replicated from the subscribed cluster. 
"%{[index]}" - define the name of the replicated index from the subscribed cluster as the variable.
${userID} - user ID registered to the cluster. 
${password} - user password registered to the cluster. 
ssl_certificate_verification => false - setting for passing a private certificate when accessing OpenSearch.

4. Run LogStash

The example command to run LogStash is as follows, and it may apply differently depending on circumstances:

# For Elasticsearch version
systemctl start logstash

# For OpenSearch version
# Use nohup for background execution.
# You need to specify the path for logstash.conf by using the -f option.
nohup {installation path}/bin/logstash -f ~{installation path}/config/logstash.conf &

Setting network

Step 1. Set ACG

To set ACG so that both 2 Search Engine Service Clusters can access the 9200 port from the server where LogStash is installed, follow these steps:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Search Engine Service in order.
2. Select the cluster and click the new window open button next to the Manager node ACG.
3. Select the ACG that has the same Manager node ACG name and click the [Set ACG] button on top.
4. Enter the following information to the inbound tap and click the [Add] button.
   • Protocol: TCP
   • Access source: IP of the server where LogStash is to run
   • Allowed port: 9200
5. Click the [Apply] button on the bottom.

In the same method, register ACG with both the subscribed cluster and the cluster to change.