logmatic/beats-forwarder

beats-forwarder

A simple forwarder to send beats everywhere through udp, tcp, syslog and third-party

Quick and dirty: send beats to Logmatic.io

sh
LOGMATIC_KEY=<YOUR_LOGMATIC_KEY>
docker run -dt \
    -p 5044:5044 \
    -e BFWD_OUTPUT_TYPE=logmatic \
    -e BFWD_LOGMATIC_API_KEY=$LOGMATIC_KEY \
    --name beats-forwarder \
    logmatic/beats-forwarder

How-to use

Installation

Notice that we've released a docker version: get the docker https://hub.docker.com/r/logmatic/beats-forwarder/ image Download the last version of the beats-forwarder (https://github.com/logmatic/beats-forwarder/releases)

sh
# linux 64
mkdir -p beats-forwarder/etc && cd beats-forwarder
curl -OL https://github.com/logmatic/beats-forwarder/releases/download/v0.1-rc1/beats-forwarder
cd etc && curl -OL https://raw.githubusercontent.com/logmatic/beats-forwarder/dev/etc/config.yml
cd ..
chmod +x beats-forwarder

# windows
# download the lastest version and go to the installation directory
mkdir etc/
# add the https://raw.githubusercontent.com/logmatic/beats-forwarder/dev/etc/config.yml file into the etc/dir
./beats-forwarder -c <your_config>
 

Configuration

Beats-forwarder use a Yaml file as configuration. By default, beats-forwarder listens on all interfaces of the server and use 5044 as port. Beats are forwarder to the local syslog. Beats-forwarder allows you to ship them to:

• A syslog server (local or remote)
• A tcp/udp endpoint (secure or not)
• To Logmatic.io

The default configuration can be found here: config.yaml Create a new file, beats-fwdr.yaml and set these attributes at least (it's a recommendation not a mandatory):

yaml
####
#### beats-forwarder default configuration
####

input:
  # The port to listen on
  port: 5044

output:

  # The wanted output (syslog|udp_tcp|logmatic), by default syslog
  type: syslog

  # Syslog specific settings
  syslog:
    # Tag or application reported for each log
    tag: beats-fowarder-demo

  # Logmatic specific settings
  logmatic:

    # The Logmatic API Key for authentification
    key: "<YOUR_API_KEY>"

If you want to send beats directly to Logmatic.io, just set output.type to logmatic and add copy/paste your Logmatic APY Key to output.type.logmatic.key.

Run

Here we are!

./beats-forwarder -c beats-fwdr.yaml

Now, just configure your already existing beats to send them to the forwarder.

Beats configuration

All you need to do is to add and configure the logstash output for each beat. Edit metricbeat-config.yml and add at the end the following code:

yaml
output:

  logstash:
    # Set the beats-forwarder address
    hosts: [ "localhost:5044"]

  # The rest of the output configuration goes here ...

Restart the beat, and check the incomming beats. If you have followed this tutorial, beats are sent to the local syslog.

sh
# this can be different depending on your OS
journalctl -f

And the magic goes on:

...
Oct 06 15:33:56 jarvis beats-by-gpolaert[6417]: {"@metadata":{"beat":"topbeat","type":"filesystem"},"@timestamp":"2016-10-06T13:33:55.108Z","beat":{"hostname":"jarvis","name":"jarvis"},"count":1,"fs":{"avail":0,"device_name":"cgroup","files":0,"free":0,"free_files":0,"mount_point":"/sys/fs/cgroup/freezer","total":0,"used":0,"used_p":0},"type":"filesystem"}
Oct 06 15:33:56 jarvis beats-by-gpolaert[6417]: {"@metadata":{"beat":"topbeat","type":"filesystem"},"@timestamp":"2016-10-06T13:33:55.108Z","beat":{"hostname":"jarvis","name":"jarvis"},"count":1,"fs":{"avail":0,"device_name":"cgroup","files":0,"free":0,"free_files":0,"mount_point":"/sys/fs/cgroup/pids","total":0,"used":0,"used_p":0},"type":"filesystem"}
Oct 06 15:33:56 jarvis beats-by-gpolaert[6417]: {"@metadata":{"beat":"topbeat","type":"filesystem"},"@timestamp":"2016-10-06T13:33:55.108Z","beat":{"hostname":"jarvis","name":"jarvis"},"count":1,"fs":{"avail":0,"device_name":"mqueue","files":0,"free":0,"free_files":0,"mount_point":"/dev/mqueue","total":0,"used":0,"used_p":0},"type":"filesystem"}
Oct 06 15:33:56 jarvis beats-by-gpolaert[6417]: {"@metadata":{"beat":"topbeat","type":"filesystem"},"@timestamp":"2016-10-06T13:33:55.108Z","beat":{"hostname":"jarvis","name":"jarvis"},"count":1,"fs":{"avail":0,"device_name":"configfs","files":0,"free":0,"free_files":0,"mount_point":"/sys/kernel/config","total":0,"used":0,"used_p":0},"type":"filesystem"}