Write processes data to kibana

Short example on how to write the top 10 processes that consume memory

  • Install logstash
  • Create the following configuration
input { 
 exec {
 command => 'ps aux --sort -rss | head -10 | tail -9'
 interval => 60
 type => 'processes_by_mem'
 } 
}

filter {
 split { }
 csv {
 columns => ['USER','PID','%CPU','%MEMORY','VSZ','RSS','TTY','STAT','START','TIME','COMMAND']
 separator => ' '
 }
 mutate {
 convert => { 
 "%MEMORY" => "integer"
 "%CPU" => "integer"
 }
 }
}

output {
 redis {
 host => 'elk-redis1.nyj.taptica.info'
 port => 6379
 key => 'logstash:redis'
 data_type => 'list'
 }
}

Explanation:

  • I use exec input that run the ps command which sort process by memory consumption
  • I use split filter to split the lines from the command (each line is different event)
  • I use the csv filer with a space separator and give for each filed from the command output a name using columns  property
  • I use mutate filter to convert %CPU and %MEMORY to integer so I can sort in kibana by this field
  • I use redis output as a buffer to elasticsearch because I have another logstash process that reads from this redis and forward all data to elasticsearch