Every Network Engineer needs a syslog server

Here's a quick and fairly reliable syslog collector that we have used.

OS: Ubuntu 16.04.2
CPU: Dual Core
RAM: 4GB or 8GB
Software Package: syslog-ng

Goal: Collect log from multiple sources and place them into logical files and folder. With that said, let get started...

First, disable rsyslog so that it can not interfere with syslog-ng service and reboot the server. Issue the following command to disable the service from start up. 

#sudo systemctl disable rsyslog

Upon reboot, install syslog-ng and enable the service.

#sudo apt-get install syslog-ng
#sudo systemctl enable syslog-ng

Create a new configuration file for syslog-ng service. 

#sudo nano /etc/syslog-ng/conf.d/syslog.conf

-------------------------------------------------------------

options {

chain_hostnames(no);

create_dirs(yes);

dir_perm(0755);

dns_cache(yes);

keep_hostname(yes);

log_fifo_size(2048);

log_msg_size(8192);

perm(0644);

time_reopen(10);

use_dns(yes);

use_fqdn(yes);

};

### Defining source

source syslog514

        {

        udp(port(514));

        };

### Defining destination

destination logstore

        {

        file("/var/log/Log-collector/$HOST/$YEAR-$MONTH-$DAY/$HOUR/$HOST.log");

        };

### Putting it all together.

log

        {

        source(syslog514);

        destination(logstore);

        };

-------------------------------------------------------------

Now lets go over few options that we have used.

create_dirs(yes) - create directory if necessary.
use_dns(yes) - uses reverse dns to resolve $HOST variable.
dns_cache(yes) - cache dns responses
use_fqdn(yes) - use fqdn for $HOST variable.

Restart syslog-ng service to read newly created conf file.

#sudo systemctl restart syslog-ng

At the point, the server is ready to take logs on udp port 514 and place them in to following format.

/var/log/Log-collector/$HOST/$YEAR-$MONTH-$DAY/$HOUR/$HOST.log

$HOST - Server will lookup the DNS name for the source ip of syslog.
$YEAR - Variable for current year
$MONTH - Variable for current month
$DAY - Variable for current day
$HOUR - Variable for the hour of the day

/var/log/Log-Collector/Kdog-FW/2017-4-15/15/Kdog-FW.log

Leverage log rotate to compress log files that are older than a day. (12.5 x compression)

#sudo nano /etc/logrotate.d/syslog-ng

### add the following at the end.

/var/log/Log-collector/*/*/*/*.log
{
        rotate 3
        daily
        missingok
        notifempty
        compress
        nocreate
        sharedscripts
        postrotate
                invoke-rc.d syslog-ng reload > /dev/null
        endscript
}

Finally, let's setup step up corn job to purge logs that are older than 4 days. 

# sudo corntab -e
find /var/log/Log-collector/* -mtime +4 -exec rm -rf {} \;

There it is!! A simple syslog collector...

Enjoy logging and troubleshooting...

WOOF!

P.S. Will create instructions on joining ubuntu server to active directory environment and leveraging AD security groups to grant access to the logs.