Hello, I'm Alan

Logstash Fun

Thursday, August 29 2013

I’ve been hearing about Logstash for quite a while, but just never got around to it. I finally have it installed and running in a way I’m generally happy with, even though at the moment its just nginx & rails getting sent to it.

Logstash + Rails

I know people have mentioned lograge, but I didn’t like how it swallowed the normal rails log, so I extract the parts I wanted, and now log to a second logfile for logstash specifcially. I’ve also added a method to our application controller that returns a hash of events for logstash, so its as easy as

logstash_payload[:search_terms] = %w[array of search terms]

and it will get sent up to elasticsearch

Elasticsearch

Elasticsearch looks like its actually rather cool and I keep hearing good things about it. I installed via the offical cookbook, and was going fine for about a day until we ran out of file handles due to a bug in the cookbook version. Once the limit was increased we had a red health status and some indices that just couldn’t be brought back, so I deleted them and got back to green.

Next problem was chewing through a good chunk of space very quickly and filling the partition ES was on. This time around just had some unallocated shards which I cleaned up by rerouting.

Almost OK until I see the cluster status back to yellow. This time it was because we are only running a single ES node, and a default somewhere is setting the number of replicas to 1 which is impossible with a single node.

I created a template for all logstash indices to have 0 repliacs by sending a PUT to _template/logstash with

{
    "template": "logstash-*",
    "settings": {
        "index.number_of_replicas": 0
    }
}

I’m using the Sense extension to manage my ES node. For security I can’t connect directly to ES from my local computer, so I tunnel 9200 on my es node back to my computer using SSH (ssh -L 9200:localhost)

Kibana

I love Kibana, but its still under very rapid development, and I’ve found its not so unusual to break it, so at the moment I’m specifying a specific known good commit when I deploy it.