Integrate Cloud Data Streaming Service using Logstash

Prev Next

Available in VPC

You can use Logstash to send data from Cloud Data Streaming Service to Search Engine Service and verify the results.

Preliminary tasks

Before following this guide, complete these prerequisites:

  • Create a VPC and server.
  • Create a Cloud Data Streaming Service cluster.
  • Create a Search Engine Service cluster.

The example shows how to run Logstash on a Server and send Kafka data to Search Engine Service.

Configure network

This section introduces a network configuration example.

STEP 1. Configure ACG

To configure the ACG to allow access to port 9092 of the Cloud Data Streaming Service broker node:

  1. In the VPC environment on the NAVER Cloud Platform console, navigate to i_menu > Services > Compute > Server > ACG.
  2. Select "cdss-b-xxxxx" from the ACG list and click [Set ACG].
  3. Enter ACG rules, and then click [Add].
    cdss-5-4_ko
    • Protocol: TCP
    • Access source: IP of the server where Logstash is to run
    • Allowed port: 9092
  4. Click [Apply].

To configure ACG to allow access to port 9200 of the Search Engine Service manager node:

  1. In the VPC environment on the NAVER Cloud Platform console, navigate to i_menu > Services > Compute > Server > ACG.
  2. Select "searchengine-m-xxxxx" from the ACG list and click [Set ACG].
  3. Enter ACG rules, and then click [Add].
    cdss-5-6_ko
    • Protocol: TCP
    • Access source: IP of the server where Logstash is to run
    • Allowed port: 9200

Install Logstash

This section introduces an example of installing Logstash on a server. The installation process includes steps for both Elasticsearch and OpenSearch. Install the version that matches the service you are using to ensure proper testing.

STEP 1. Install Java

  1. Enter the following command to install Java:
yum install java-devel -y

STEP 2. Install Logstash

To install Logstash:

  1. Enter the following command to download Logstash to the /root directory.
# For Elasticsearch version (install OSS version)
wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.7.0.rpm

# For OpenSearch version
wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-7.16.3-linux-x64.tar.gz
  1. Enter the following command to install the downloaded file.
# For Elasticsearch version
rpm -ivh logstash-oss-7.7.0.rpm

# For OpenSearch version
tar -zxvf logstash-oss-with-opensearch-output-plugin-7.16.3-linux-x64.tar.gz
  1. Before starting Logstash, edit the logstash.conf file.
  • For Elasticsearch version
mv /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/logstash.conf
vi /etc/logstash/conf.d/logstash.conf
  • Elasticsearch version logstash.conf
input {
 kafka {
  bootstrap_servers => "${bootstrap_servers}"
  topics => "cdss_topic"
 }
}

output {
  elasticsearch {
    hosts => ["http://${ses manager node1 ip}:9200", "http://${ses manager node2 ip}:9200"]
    index => "cdss-%{+YYYY.MM.dd}"
  }
}
  • For OpenSearch version
# When installing to /root/, the {installation path} is /root/logstash-7.16.3. 
mv {installation path}/config/logstash-sample.conf {installation path}/config/logstash.conf
vi {installation path}/config/logstash.conf
  • OpenSearch version logstash.conf
input {
 kafka {
  bootstrap_servers => "${bootstrap_servers}"
  topics => "cdss_topic"
 }
}

output {
  opensearch {
    hosts => ["https://${ses manager node1 ip}:9200", "https://${ses manager node2 ip}:9200"]
    index => "cdss-%{+YYYY.MM.dd}"
    user => ${userID}
    password => ${password}
    ssl_certificate_verification => false
  }
}
  • Logstash Conf Comment
${bootstrap_servers} - Enter the IP address and Kafka port of the Cloud Data Streaming Service broker node.  Example: 172.16.19.6:9092,172.16.19.7:9092,172.16.19.8:9092
${ses manager node1 ip} - Enter the IP address of the Search Engine Service manager node.
${ses manager node2 ip} - Enter the IP address of the second Search Engine Service manager node, or leave it blank if the manager nodes are not redundant.
${userID} - For OpenSearch, enter the user ID you set when creating the cluster.
${password} - For OpenSearch, enter the password you set when creating the cluster.

Run Logstash

Example commands for running Logstash are as follows.

# For Elasticsearch version
systemctl start logstash

# For OpenSearch version
# Use nohup for background execution.
# You need to specify the path for logstash.conf by using the -f option.
nohup {installation path}/bin/logstash -f ~{installation path}/config/logstash.conf &

Build CDSS integration environment

Example commands for running Logstash are as follows.

  • Install Java
yum install java-devel -y
  • Install Kafka Binary
wget https://archive.apache.org/dist/kafka/2.4.0/kafka_2.12-2.4.0.tgz
tar -zxvf kafka_2.12-2.4.0.tgz
  • Produce
./kafka_2.12-2.4.0/bin/kafka-console-producer.sh --broker-list ${broker list} --topic cdss_topic
>this is my first message
>this is my second message

# For ${broker list}, enter the IP address and Kafka port of the broker node. Example: 172.16.19.6:9092,172.16.19.7:9092,172.16.19.8:9092

View Cloud Data Streaming Service data

The following example demonstrates how to query Cloud Data Streaming Service data in Search Engine Service.

GET cdss-2022.08.08/_search

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cdss-2022.08.08",
        "_type" : "_doc",
        "_id" : "VtmKe4IBicE7MyrTaKJ5",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2022-08-08T03:40:44.335Z",
          "message" : "this is my first message"
        }
      },
      {
        "_index" : "cdss-2022.08.08",
        "_type" : "_doc",
        "_id" : "V9mKe4IBicE7MyrTg6IW",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2022-08-08T03:40:51.248Z",
          "message" : "this is my second message"
        }
      }
    ]
  }
}

Use Kafka with SSL

When using SSL with Cloud Data Streaming Service, you can configure SSL by adding certificates.

  • INPUT
    input {
  kafka {
    bootstrap_servers => “{BrokerNode-HostName}:9093” 
    topics => "test“
    ssl_truststore_location => "/etc/logstash/conf.d/kafka.client.truststore.jks"
    ssl_truststore_password => “${password}"
    security_protocol => "SSL"
  }  
}
    • ${BrokerNode-HostName} - This is available in Cluster list -> Broker node info -> Details -> TLS. Example: "networktest12-d-251:9093,networktest12-d-252:9093,networktest12-d-253:9093"
    • ${password} - Separate password for the certificate.