How to use logstash
Logstash
Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite "stash." Logstash (part of the Elastic Stack) integrates data from any source, in any format with this flexible, open source collection, parsing, and enrichment pipeline.
Pipeline Architecture
Input to logstash
Download
Download logstash using following command line in the current working directory
1wget https://artifacts.elastic.co/downloads/logstash/logstash-7.10.1-linux-aarch64.tar.gz
Extract
1tar -xvzf logstash-7.10.1-linux-aarch64.tar.gz
This will extract the directory structure inside compressed file into logstash-7.10.1 in the current working directory.
logstash executable is present under logstash-7.10.1/bin. You can directly use this executable or add it to the PATH variable to use it from anywhere.
Usage
General usage of logstash is using config file which specifies the following skeletal structure
1logstash -f path-to-config-file.conf
2
3eg.:
4
5logstash -f sample.conf
sample.conf
1input{
2 # Input plugin details
3}
4
5filter {
6 # filter to input data
7}
8
9output {
10 # out to stdout file, db or elastic search
11}
WAP to input from standard input and write to a file
stdin-to-file.conf
1input {
2 stdin{}
3}
4
5output {
6 file {
7 path => "std_in_out-logstash-log.log"
8 }
9}
Steps
- Start logstash using
logstash -f stdin-to-file.conf - Once it settles down, terminal will wait for standard input >_
- To observer the output written to a file we will use tail command on the file to observer changes
tail -n 1 -f std_in_out-logstash-log.log - Enter some text on logstash terminal and observer the output on the other terminal as shown below.
Output
Similarly, applying filter is easy
Field name is message here let's apply mutate filter on input data and convert it to upper case.
1filter {
2 mutate {
3 uppercase => ["message"]
4 }
5}