elgentos commerce & configurator agency

7 min reading

This is a quick tutorial on how to capture Magento's log files using the ELK stack. ELK stands for Elasticsearch, Logstash and Kibana. I won't go into too much detail, but broadly speaking, Elasticsearch is used to store and quickly retrieve log entries, Logstash is responsible for pulling the data into Elasticsearch, and Kibana is used to create overviews and visualizations of the large amount of log entries.

In addition to ELK, we will use the command-line tool log-courier to send the logs directly from our production server to the ELK stack via an encrypted connection.

Please note; we are using log-courier 1.8.3 because that is the current version installed on Byte's Hypernode hosting solution (which we use for all of our customers). The latest version is now 2.0.5 and offers slightly more features, especially in the lc-admin tool.

Much of my research on how to set this up came from this Gist from Github user purinda, DigitalOcean's blog on ELK and log-courier's documentation.

Setting up the ELK stack

DigitalOcean used to offer a one-click app for the ELK stack. Unfortunately, it is no longer available. I leave it up to you to set up your own ELK stack. See DO's blog linked above or choose a Docker image like this one. You can also try emailing DigitalOcean to see if they have an image for you. Just skip the Filebeat part of the DO blog; we'll use log-courier to get our logs into ELK.

Kibana and Elasticsearch are pretty easy, just follow the blog. Remember where you placed your certificate file and your secret key file for Logstash.

Configuring Logstash

A Logstash configuration has at least three parts; input, output and filter.

Entry

The input configuration file defines how the logs enter Logstash. In our case, we will use log-courier, so we need the logstash-input-courier plugin for Logstash;

SSH to your ELK stack
Go to /opt/logstash/
Run sudo bin/plugin install logstash-input-courier

Now we are going to set up the input;

Go to /etc/logstash/conf.d/
Create a new file called 01-courier-input.conf
Use this content;

input {
    courier {
        port            => 5043
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key         => "/etc/pki/tls/private/logstash-forwarder.key"
    }
}

Possibly replace the ssl certificateand ssl keywith your paths.
Open port 5043 in your firewall (ufw insert 1 allow from $PRODUCTION-SERVER-IP to any port 5043)

You can read more about setting up Logstash in log-courier here.

Output

Logstash needs to know where to send the received data. Since we are using ELK, Elasticsearch is the output. In this configuration, we assume it is running on the same machine (localhost) and on port 9200.

Go to /etc/logstash/conf.d/
Create a new file called 10-elasticsearch-output.conf
Use this content;

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    document_type => "%{[@metadata][type]}"
  }
}

Filter

The third element is a filter; the incoming data must be understood by Logstash, so we must specify the format in which it can expect data. This is done using a regex-like syntax called grok. I have written two groks (for Magento 1 and Magento 2). There is a great tool to test your groks; Grok Constructor Matcher.

Go to /etc/logstash/conf.d/
Create a new file called 20-magento-filter.conf
Use this content;

filter {
  if [type] == "magento2" {
    grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:log_level}: %{GREEDYDATA:message}"}
      add_field => [ "received_at", "%{@timestamp}" ]
    }
  }
  if [type] == "magento1" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:date} %{DATA:log_level} \([0-9]+\): %{GREEDYDATA:message}"}
      add_field => [ "received_at", "%{@timestamp}" ]
    }
  }
}

Note; this is only for system.log and does not take into account multi-line log entries. You can find a multi-line log entry grok for Magento here.

Test your Logstash configuration with service logstash configtest. If you see Configuration OK, run service logstash restart. Check that it is listening on port 5034 with sudo lsof -i:5043. It should say something like;

COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    2737 logstash   16u  IPv6  54468      0t0  TCP *:5043 (LISTEN)

Setting up your production server

Now we need to set up your production server to send data to Logstash.

Create a folder where you store log-courier related settings, such as ~/log-courier/
Create a config.json file in that directory with these contents;

{
	"general": {
		"admin enabled": true
	},
	"network": {
		"servers": [ "ELK-IP-ADDRESS:5043" ],
		"ssl ca": "/absolute/path/to/your/log-courier/logstash.cer"
	},
	"files": [
		{
			"paths": [ "/path/to/magento2/var/log/*.log" ],
			"fields": { "type": "magento2" }
		}
	]
}

Change the path to your Magento var/log folder
Change the type to magento1 if you are using Magento 1 (this matches the types set in the filter file you created earlier)
Change ELK-IP-ADDRESS to the hostname or IP address of your ELK stack
If desired, disable management by changing true to false
Copy the Logstash certificate from your ELK stack to ~/log-courier/logstash.cer (in my example, the path of the certificate on the ELK stack was /etc/pki/tls/certs/logstash-forwarder.crt)
Launch log-courier manually from the CLI by running log-courier -config ~/log-courier/config.json &

Note: The management tool helps you understand what's going on when something doesn't work. See this Github page to see what it can do and how it works.

Configuring Kibana

I will talk briefly about configuring Kibana because there is a lot of information available on how to configure it and it largely depends on your usage scenario.

Open Kibana by going to your ELK hostname with the browser
Go to Settings and under 'Index name or pattern' type 'logstash-'.
Click on Create
Go to the Discover tab and you should see potential logs coming in

Troubleshooting

If you don't see any logs coming in, make sure there are logs by checking the configured directory on your production server. If so, SSH to your ELK stack and run sudo lsof -i:5043. If a connection has been established from your production server to your ELK stack, you should see something like this;

COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    2737 logstash   16u  IPv6  54468      0t0  TCP *:5043 (LISTEN)
java    2737 logstash   41u  IPv6  55003      0t0  TCP ELK-IP-ADDRESS:5043->ELK-IP-ADDRESS:40176 (ESTABLISHED)
java    2737 logstash   45u  IPv6  54474      0t0  TCP ELK-IP-ADDRESS:5043->ELK-IP-ADDRESS:56348 (ESTABLISHED)

You can run the same command for ports 9200 and 5601 to check Elasticsearch and Kibana, respectively.

If there is no connection, check that Logstash is really running with ps aux | grep -i logstash. If it is not running, check Logstash's error and log files in /var/log/logstash/logstash.err and /var/log/logstash/logstash.log.

If there are logs and there is a connection and no logs come into Kibana, you can check these things;

Check your grok patterns. If nothing matches, nothing appears
Run status in lc-admin on your production server to see if it processes lines
Check your index pattern in Kibana

Install cron on Hypernode

Because Hypernode is a managed hosting solution, we cannot change the default configuration. We have to start log-courier manually. This also means that it will not be started automatically on any system reboot. Therefore, we need to set up a cron to ensure that log-courier is executed.

Add this to your crontab; */30 * * * * flock -n ~/.log-courier.lock -c 'log-courier -config /absolute/path/to/your/log-courier/config.json &'
Make sure your directory for log-courier is not ~/.log-courier (note the dot) as this is a file that log-courier creates when running
Make sure the certificate file in config.json is absolutely referenced

Good luck!