Troubleshoot log issues

Available in Classic and VPC

You might run into the following problems when using Cloud Log Analytics: Find out causes and possible solutions.

Delete logs

I want to delete logs.

Cause

Logs stored in Cloud Log Analytics cannot be deleted arbitrarily.

Solution

You cannot arbitrarily delete logs. After the retention period (up to 30 days), logs are automatically deleted starting with the oldest. For more information, see Cloud Log Analytics prerequisites.

Disable log collection

I am using Cloud DB for MySQL but do not want to use the log collection feature.

Cause

Cloud DB services collect and provide Error logs and Slow Query logs by integrating with Cloud Log Analytics. Cloud DB requires Cloud Log Analytics, so you cannot unsubscribe from Cloud Log Analytics or delete logs.

Solution

You cannot disable the integration with Cloud Log Analytics while using Cloud DB.

View filebeat logs

After installing the collection agent, I want to check on the server if logs are being sent properly by viewing filebeat logs.

Cause

filebeat logs are configured not to generate log files.

Solution

To retain log files:

1. Add the following content at the end of the /etc/filebeat/filebeat.yml file:

logging.level: debug
logging.to_stderr: false
logging.to_syslog: false
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

2. Restart the agent.

sudo systemctl restart filebeat.service

3. Check the error message in the log file created in the /var/log/filebeat path.

Log transfer error

I installed the agent and saw the "Finish Installation" message, but the logs aren't being sent.

Cause

Log display issues fall into the following categories. Check the items for the applicable case first.

The causes of logs not being sent at all are as follows:

Solution

Check the following items in order and take the appropriate action:

1. Check whether the agent process is running.

ps -ef | grep filebeat

• If the process is not running, restart it using the sudo systemctl restart filebeat.service command.

2. Check whether the agent configuration file has been created and contains the collection server address.

ls -altr /etc/filebeat/filebeat.yml
cat /etc/filebeat/filebeat.yml

• If the file does not exist or is empty, rerun the installation command from the console on the server.
• The output.logstash.hosts entry must contain the collection server address and port.

3. Check whether the server can communicate with the collection server. The collection server port is 5043/TCP.

telnet {collection server port} 5043

• On Windows servers, use the Test-NetConnection {collection-server-address} -Port 5043 command.
• The collection server address is the output.logstash.hosts value from step 2.

4. If communication fails, allow outbound traffic to the relevant address and port in the ACG or firewall. If you use iptables on the server, register the allowed target as the destination.
5. If the server was created from an image with the agent already installed, the agent from the previous server may remain and prevent logs from being sent. Remove the agent, then reinstall it using the installation command issued from the server's Management page.
6. If logs are still not sent after checking all the items above, view filebeat logs to collect diagnostic logs, then contact Customer Support with the error message from the /var/log/filebeat path.

Unable to collect specific logs

Logs configured with templates such as SYSLOG and Security are displayed, but Custom Log is not displayed.

Cause

If only some logs are not collected, the issue is not with the agent itself but with the collection settings for the relevant Log Type or the target files. The causes are as follows:

Solution

1. Check whether the collection settings have been applied

This is the most common cause. After adding a log for collection or changing its path in the console, rerun the installation command shown on the Management page to apply the changes to the agent configuration file.

1. Check whether the configuration file contains the path of the log that is not being collected.

cat /etc/filebeat/filebeat.yml

2. If the path is missing from the paths item, copy the installation command for the server from the console and run it again.

2. Check the collection path and target file

1. Check whether the paths item in the configuration file contains only a directory path. Logs are collected on a per-file basis, so the path must include a file name or file name pattern.

• Examples that are not collected: /home/nginx/log
• Examples that are collected: /home/nginx/log/access*.log

2. Check whether the string in the paths item matches the actual file name. Use the following command to check whether the file exists in the specified path.

ls -altr {log path}

3. If the collection target is a symbolic link whose target file is replaced periodically, use a pattern for the actual log file name instead of the fixed link file name.
4. After changing the path, rerun the installation command as described in step 1.

3. Check the target file format

The agent collects text files only. Binary files (such as wtmp, btmp, and lastlog) and compressed files are not collected properly even if they are specified as collection targets. Specify log files written in text format.

4. Check the account used to install the agent

If the agent was installed using an account that does not have read access to some log files, those logs may not be collected In this case, changing the log file permissions does not resolve the issue, so reinstall the agent with root privileges.

5. Additional checks

If logs are still not collected after checking all the items above, view filebeat logs to collect diagnostic logs, then contact Customer Support with the error message from the /var/log/filebeat path.

Unable to view collected logs

Some collected logs do not appear on the Search page.

Cause

The following are all normal behaviors and do not indicate a log collection failure:

Solution

1. Only the 5 log types with the highest log volume appear in the Log Type list in Search. A log type not appearing in the list does not mean the corresponding logs are not being collected. To check whether logs are being collected, navigate to Menu > All Services > Management & Governance > Cloud Log Analytics > Management and check the collection settings. For details on search conditions and how results are displayed, see Search.
2. Logs may take some time to appear in search results after being sent. If logs do not appear immediately after they are generated, check again after a short while.
3. When you collect rolling logs using wildcards, the agent tracks the last read position in each file and sends only content added afterward. Logs are not duplicated or omitted when a file is created or replaced.