From Network Management Wiki

About

This guide will walk you through a standard installation of both Ubuntu 9.10 Server edition as well as LogZilla. The guide was developed for LogZilla v3.0 beta.

The majority of the commands on this page require running with root privileges. This can be accomplished by:

sudo su -

Name Change

Php-syslog-ng has been officially renamed to LogZilla

LogZilla on non-Ubuntu OS's

LogZilla on Redhat/CentOS/Fedora

Installing Logzilla on Gentoo Linux

Installing Logzilla 3.0 on OpenSolaris

LogZilla on SUSE/SLES

Ready-to-Run Virtual Machine

A user-contributed VM is available here:

Virtualizing Logzilla

Obtaining LogZilla

Logzilla can be downloaded to your /tmp directory

 
cd /tmp
wget  http://php-syslog-ng.googlecode.com/files/logzilla_x.x.x.tgz

Change to your desired installation location and extract.

cd /var/www/
tar zxvf /tmp/logzilla_x.x.x.tgz

Requirements

Beyond the base install, there are a few other dependencies that need to be resolved for LogZilla to run.

perl
- Date::Calc
- Text::LevenshteinXS
syslog-ng
gcc
Apache
PHP
MySQL v5.1 or greater

The majority of these can be installed by selecting LAMP Server when either installing Ubuntu initially, or by running:

 
sudo tasksel

From the popup you will be able to select LAMP server, as well as configure the MySQL password. Or by running:

sudo aptitude install apache2 php5 php5-gd php5-cli php5-mysql mysql-server-5.1 libmysqlclient15-dev \
build-essential libmysqlclient-dev

NOTE: For Ubuntu 10.x and up use libmysqlclient16-dev

syslog-ng

Simple:

sudo apt-get install syslog-ng

gcc

gcc can be installed by:

 
sudo apt-get install gcc

PERL

Perl is used for the installation of LogZilla as well as the database inserts during normal operation - these files are located in the scripts/ directory. The following perl modules are used with LogZilla and will need to be installed prior to running the install script:

Date::Calc
Text::LevenshteinXS
String::CRC32

You can easily install them from the command line by typing:

sudo cpan Date::Calc Text::LevenshteinXS String::CRC32

As an alternative, Ubuntu has Date Calc and String CRC32 as a package in it's repository. It can be installed with the:

 
sudo apt-get install libdate-calc-perl libstring-crc32-perl

If these dependencies are not resolved, you will see errors like:

 
Can't locate Date/Calc.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.0 \
/usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share\
/perl/5.10 /usr/local/lib/site_perl .) at ./install.pl line 23.

Installing LogZilla

Once we have all the pre-requisite packages installed, we can move on to the actual installation of LogZilla.

 
cd /var/www/logzilla/scripts
sudo ./install.pl

Thats it. :) Below is a walk through of all questions that are asked during the install phase:

 



====================

        LogZilla Installation

====================



Enter the MySQL root username [root]: 
Enter the password for root [mysql]: 
Database to install to [syslog]: 
Database table to install to [logs]: 
Enter the name of the MySQL server [127.0.0.1]: 
Enter the port of the MySQL server [3306]: 
Enter the name to create as the owner of the logs database [syslogadmin]: 
Enter the password for the syslogadmin user [syslogadmin]: 
Enter the name to create as the WEBSITE owner [admin]: 
Enter the password for admin [admin]: 
Enter your email address [cdukes@cdukes.com]: 
Enter a name for your website [The home of LogZilla]: 
Enter the base url for your site (include trailing slash) [/logs/]: /
Where should log files be stored? [/var/log/logzilla]: 
How long should I keep old logs? (in days) [30]: 



========================================

        Path Updates

========================================



Getting ready to replace paths in all files with "/var/www/logzilla"
Ok to continue? [y]: 
Updating file paths
Modifying ../scripts/db_insert.pl
Modifying ../scripts/contrib/system_configs/logzilla.crontab
Modifying ../scripts/contrib/system_configs/syslog-ng.conf
Modifying ../scripts/contrib/system_configs/logzilla.apache
Modifying ../sphinx/indexer.sh
Modifying ../sphinx/sphinx.conf
Updating log paths
Modifying ../scripts/contrib/system_configs/logzilla.crontab
Modifying ../scripts/contrib/system_configs/logzilla.logrotate



====================

        Database Installation

====================


All data will be installed into the syslog database
Ok to continue? [y]: 


====================

        Config.php generation

====================



Generating /var/www/logzilla/html/config/config.php
Ok to continue? [y]: 


====================

        System files

====================



Adding LogZilla logrotate.d file to /etc/logrotate.d
Ok to continue? [y]: 
Where is your syslog-ng.conf file located? [/etc/syslog-ng/syslog-ng.conf]: 
Adding syslog-ng configuration to /etc/syslog-ng/syslog-ng.conf
Ok to continue? [y]: 
Found 1 sources
Which source definition would you like to use? [s_all]: 

        LogZilla installation complete...
Note: you may need to enable the MySQL Event Scheduler in your /etc/my.cnf file.
Please visit http://forum.logzilla.info/index.php/topic,71.0.html for more information.

Also, please visit http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers to learn how to increase your UDP buffer size (otherwise you may drop messages).

Please run /etc/init.d/syslog-ng restart

Installing Sphinx

Sphinx is used to provide the full text search capabilities for LogZilla, it is an order of magnitude faster than using MySQL alone (much, much faster).

The Sphinx distribution (source) is included in the LogZilla tarball, here are the steps needed to compile it:

1. cd to sphinx and extract:

cd logzilla/sphinx/src
tar xzvf sphinx-0.9.9.tar.gz

2. cd to the source directory and compile using a prefix of your logzilla path:

cd sphinx-0.9.9
./configure --prefix `pwd`/../..

If you see this error:

ERROR: cannot find MySQL include files.

Then you didn't install all the prerequisites listed above.

3. Make and install

make && make install

4. Create your first index and start the search daemon:

cd /var/www/logzilla/sphinx
./indexer.sh full
bin/searchd

NOTE: If you see an error when creating the full index that looks similar to this:

ERROR: index 'idx_logs': sql_query_pre[0]: Column 'max_id' cannot be null (DSN=mysql://syslogadmin:***@127.0.0.1:3306/syslog).

Then it means you have no logs in the database yet, wait until you get a few entries, then try again.

5. Once you have this working, add your indexer to cron.

A crontab file is included in scripts/contrib/system_configs/logzilla.crontab

Here's a sample:

#####################################################
# Run Sphinx "full" scan 30 minutes after midnight
# in order to create a new index for today.
# 
#####################################################
30 0 1 * * /path_to_logzilla/sphinx/indexer.sh full >> /path_to_logs/sphinx_indexer.log 2>&1

#####################################################
# Run Sphinx "delta" scans every 5 minutes throughout 
# the day.
# Delta indexing should be very fast but you may need
# to adjust these times on very large systems.
#####################################################
*/5 * * * * /path_to_logzilla/sphinx/indexer.sh delta >> /path_to_logs/sphinx_indexer.log 2>&1

#####################################################
# Run Sphinx "merge" scans every day at midnight
# Merging is much faster than a full scan.
# You may need to adjust these times on very large systems.
#####################################################
0 0 * * * /path_to_logzilla/sphinx/indexer.sh merge >> /path_to_logs/sphinx_indexer.log 2>&1

That should be all you need to do to get sphinx running.

You will also want to make sure that you add /var/www/logzilla/sphinx/searchd to your startup files in case you reboot.

The full command for searchd is:

/var/www/logzilla/sphinx/bin/searchd -c /var/www/logzilla/sphinx/sphinx.conf

Replacing, of course, the paths to reflect your install.

For Ubuntu users, you can make searchd run during startup by adding it to your /etc/rc.local, like this:

vi /etc/rc.local

And add (replace the path with your installed path):

/var/www/logzilla/sphinx/bin/searchd -c /var/www/logzilla/sphinx/sphinx.conf

So it should look something like this:

If you need more help, check the Ubuntu Guide

Notes/Troubleshooting

Make sure that the indexer.sh can perform a delta merge by running:

mysql -usyslogadmin -psyslogadmin syslog

If you see this error:

ERROR 1045 (28000): Access denied for user 'syslogadmin'@'localhost' (using password: YES)

Then it means you installed LogZilla using 127.0.0.1 and not localhost, so you will need to add permission for localhost to mysql:

Note: be sure to replace the syslogadmin username and password with the one you created at install.

mysql> GRANT ALL PRIVILEGES ON syslog.* TO 'syslogadmin'@'localhost' IDENTIFIED BY 'syslogadmin';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

Configure MySQL

LogZilla makes use of the MySQL event scheduler to handle log rotation, a good tutorial on the event scheduler can be found here:

MySql 5.1 Scheduler

LogZilla uses partitioning (at midnight by default) to segment the logs. This utilizes the event scheduler in MySQL 5.1, which may be disabled by default in your OS.

To check the current status of event scheduler:

logzilla# mysql -u<username> -p<password>
mysql> SELECT @@event_scheduler;
+-------------------+
| @@event_scheduler |
+-------------------+
| OFF               |
+-------------------+
1 row in set (0.00 sec)

To enable it, log into mysql and then:

mysql> SET GLOBAL event_scheduler = 1; 
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @@event_scheduler;
+-------------------+
| @@event_scheduler |
+-------------------+
| ON                |
+-------------------+
1 row in set (0.00 sec)

mysql> quit;

NOTE: You will also need to enable the event scheduler in your my.cnf file so that it persists across restarts.

MySQL (my.cnf) Settings

Here are the my.cnf settings I used when testing system scalability.

The tests were inserting around 15-20k MPS and had, at the time, around 40m messages.

Queries took < 1 second.

# BEGIN Clay's settings
event-scheduler=1
skip-name-resolve
query_cache_size = 16M
table_cache = 512
tmp_table_size = 128M
max_heap_table_size = 128M
myisam_sort_buffer_size = 512M
sort_buffer_size = 8M
join_buffer_size = 256K
key_buffer = 512M
bulk_insert_buffer_size = 512M
# END Clay's settings

You also may want to disable binary logging as noted HERE

Events not being inserted

If the MySQL event scheduler fails to run (by default, at midnight), you will have to manually add the missing partition(s).

To see if you have a missing partition:

mysql> show create table logs \G

The output will look something like this:

/*!50100 PARTITION BY RANGE ( TO_DAYS( lo ))
(PARTITION p20100603 VALUES LESS THAN (734291) ENGINE = MyISAM,
 PARTITION p20100604 VALUES LESS THAN (734292) ENGINE = MyISAM,
 PARTITION p20100605 VALUES LESS THAN (734293) ENGINE = MyISAM) */

The last partition you see should be for tomorrow, if not, add one like this:

Note: Replace p20100605 and '2010-06-05' with tomorrow's date.

alter table logs PARTITION BY RANGE( TO_DAYS( lo ) ) (PARTITION p20100605 VALUES LESS THAN (to_days('2010-06-05')));

Configure Syslog-ng

Now we need to move on to our syslog-ng.conf file and make a small edit in it. By default udp() is commented out. We need to change that:

sudo cp /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.orig
## This way, if we screw it up, we have a copy to restore
sudo vi /etc/syslog-ng/syslog-ng.conf
## In mine, it was around line 93. We just need to remove the # from in front of udp();

Apache

Naturally, if we're going to use a web interface, we'll need a web server :-)

First, edit the /etc/apache2/apache2.conf file and add a ServerName directive

 
sudo vi /etc/apache2/apache2.conf

When you're done, it should look like this (replace logzilla with your server's name):

 
ServerRoot "/etc/apache2" <<- Existing line
ServerName logzilla

Replace 'logzilla' with your actual server name

Note: You can skip the following section if you are installing to the root web server.

Not all systems will need to use the following config. This is only provided as an example.

Next, create a file in /etc/apache2/sites-available called "logzilla" and add the following to it:

 
# LogZilla
   Alias /logs "/var/www/logzilla/html/"
   <Directory "/var/www/logzilla/html/">
       Options Indexes MultiViews FollowSymLinks
       AllowOverride All
   Order allow,deny
   Allow from all
   </Directory>

Note: AllowOverride should be set to "All" so we can modify php variables using a .htaccess file (more on that later)

Now save this file in the appropriate location. For Ubuntu users, it would be saved as:

 
/etc/apache2/sites-available/logzilla

Then, for Ubuntu, you would type:

 
a2ensite logzilla

If you are on a distro other than Ubuntu, you'll have to look up the documentation to see how to implement this (you might be able to simply add it to the default config file)

Now restart Apache and make sure you don't see any errors.

 
 sudo /etc/init.d/apache2 restart
* Restarting web server apache2                                                                               [ OK ]

This install script should have taken care of all of the details for us.

If we navigate to http://127.0.0.1/logs/login.php we should be greeted with a login page
Login using username/passwd set during install.pl

IONCube Licensing

LogZilla 3.0 uses a licensing encoder provided by IONCube

Installation is simple, but requires access to the php.ini To install, you must use one of the loaders provided in the html/ioncube directory. For example:

cd logzilla/html/ioncube
ls -al
-rw-r--r-- 1 root root  1.3K 2009-07-14 17:37 ioncube-encoded-file.php
-rw-r--r-- 1 root root   26K 2009-07-14 17:37 ioncube-loader-helper.php
-rw-r--r-- 1 root root  294K 2009-07-14 17:37 ioncube_loader_lin_4.1.so
-rw-r--r-- 1 root root  287K 2009-07-14 17:37 ioncube_loader_lin_4.2.so
-rw-r--r-- 1 root root  296K 2009-07-14 17:37 ioncube_loader_lin_4.2_ts.so
-rw-r--r-- 1 root root  287K 2009-07-14 17:37 ioncube_loader_lin_4.3.so
-rw-r--r-- 1 root root  297K 2009-07-14 17:37 ioncube_loader_lin_4.3_ts.so
-rw-r--r-- 1 root root  292K 2009-07-14 17:37 ioncube_loader_lin_4.4.so
-rw-r--r-- 1 root root  302K 2009-07-14 17:37 ioncube_loader_lin_4.4_ts.so
-rw-r--r-- 1 root root  277K 2009-07-14 17:37 ioncube_loader_lin_5.0.so
-rw-r--r-- 1 root root  286K 2009-07-14 17:37 ioncube_loader_lin_5.0_ts.so
-rw-r--r-- 1 root root  988K 2009-07-14 17:37 ioncube_loader_lin_5.1.so
-rw-r--r-- 1 root root  1.1M 2009-07-14 17:37 ioncube_loader_lin_5.1_ts.so
-rw-r--r-- 1 root root  960K 2009-07-14 17:37 ioncube_loader_lin_5.2.so
-rw-r--r-- 1 root root 1011K 2009-07-14 17:37 ioncube_loader_lin_5.2_ts.so

For instructions on loading, browse to your server url /ioncube/

For example:

http://localhost/ioncube/

Or, for 64 bit systems,

http://localhost/ioncube_64/

According to the output, I need to add zend_extension = /usr/local/ioncube/ioncube_loader_lin_5.3.so to my php.ini.

First, copy the ioncube_64/ (or just ioncube/ if you are 32bit) to the /usr/local directory

cp -rp html/ioncube_64 /usr/local/ioncube

Now, edit your php.ini and add the zend line:

vi /etc/php5/apache2/php.ini
zend_extension = /usr/local/ioncube/ioncube_loader_lin_5.2.so

Next, restart Apache:

/etc/init.d/apache2 restart
* Restarting web server apache2
waiting                                           [ OK ]

That's it!

Once you have the licensing working, you should remove the html/ioncube directories - or at least move them out of the html/ directory for security purposes.

Testing with Logreplay

I've included a new in script as of version 2.9.9 that will allow you to "replay" a log file taken from another server. The script is located in:

scripts/contrib/logreplay/logreplay.pl

To use the sample logs included simply un-gzip it:

 
gzip -d syslog.sample.gz

Before we can run this, we need to compile spoof

The "spoof" program that I've included will rewrite the outgoing syslog packet and insert the hostnames from the syslog.sample file so that when syslog-ng receives the messages they appear to come from that host instead of your local machine.

 
gcc spoof_syslog.c -o spoof_syslog

This will create the ./spoof_syslog executable used by logreplay.pl And run the logreplay script:

 
sudo chmod +x logreplay.pl
./logreplay.pl -h

Which will give you help on the scripts syntax:

 
This program is used to replay a standard *Cisco* syslog dumpfile into the local syslog receiver (syslog-ng)
    usage: logreplay.pl [-hvfs] 
    -h        : this (help) message
    -v        : verbose output
    -f        : Filename to import (required)
    -s        : path to the spoof program (required)
    -r        : Generate random IP's based on incoming hosts (last octect will be randomized)
    example: logreplay.pl -v -f ./syslog.sample -s ./spoof_syslog

So to run it, you would do:

 
./logreplay.pl -v -f ./syslog.sample -s ./spoof_syslog

Upgrade Procedures

Upgrade Procedures for Logzilla have been moved to their own page, to ease page size. :) Click below.

Upgrade Procedures for Logzilla 3.0

Appendix/FAQ

Password Reset

To reset your password, you can change it by logging into a mysql shell:

mysql -uroot -p syslog
Enter password: 
mysql> update users set pwhash=md5('MYNEWPASSWORD') where username='admin';

> quoting socket(7) on Linux
>
>       rmem_default
>              contains the default setting in bytes of the socket receive buffer.
>
>       rmem_max
>              contains the maximum socket receive buffer size in bytes which a user may set by using the SO_RCVBUF socket option.
>
>
> syslog-ng allows you to change the socket buffer given rmem_max allows
> that using the so_rcvbuf() option for the given source.
>
> Changing rmem_default at the system scope might be a bit of an overkill,
> as _all_ udp sockets would get this large socket buffers.
>
> And socket buffers are in non-pageable memory, don't forget that.

So, this is a much simpler thing than modifying the rmem_default. To make the necessary changes for syslog-ng to not drop udp messages, take these steps:

1. Change the rmem_max value to 1m in your system as described above:

sysctl -w net.core.rmem_max=1048576

Remember - this only changes it temporarily. See above for making it permanent.

2. Change your syslog-ng.conf from

udp();

to:

udp( so_rcvbuf(1048576));

So, here's my final syslog-ng.conf:

source s_all {
        # message generated by Syslog-NG
        internal();
        # standard Linux log source (this is the default place for the syslog()
        # function to send logs to)
        unix-stream("/dev/log");
        # messages from the kernel
        file("/proc/kmsg" log_prefix("kernel: "));
        # use the following line if you want to receive remote UDP logging messages
        # (this is equivalent to the "-r" syslogd flag)
        # cdukes: Added so_rcvbuf because I was dropping udp messages when they exceeded around 50-100mps.
        udp( so_rcvbuf(1048576));
        #udp();
};

VMWare Installations

If you plan to install LogZilla to a VMWare ESX Server and you will be receiving a large amount of logs, then you'll want to set the resource allocation on the disk to high.

To do this, right-click on your VM and select 'edit'.

Next, click the 'Resources' tab and click 'disk' and change the drop-down menu from 'normal' to 'high' like the image below:

Security

Specific code has been implemented to avoid any XSS/SQL injections to LogZilla. However, as you may know, no software is truly secure.

If you are concerned about security with LogZilla, it is highly recommended that you also employ the use of the Apache ModSecurity tool.

For Ubuntu users, here's an easy to follow install guide

Here are the basic steps from that guide:

1. Install

sudo apt-get -y install libapache-mod-security

2. Configure

Using any editor, make a file “/etc/apache2/conf.d/modsecurity2.conf” and put the following contents in the file.

<ifmodule mod_security2.c>
Include conf.d/modsecurity/*.conf
</ifmodule>

3. Set up logging

By default, mod_security logs to /etc/apache2/logs, the following commands will put the log in /var/log/apache2/mod_security and create a symbolic link back to /etc/apache2/logs

sudo mkdir /var/log/apache2/mod_security
sudo ln -s /var/log/apache2/mod_security/ /etc/apache2/logs

4. Download and install the rules

sudo mkdir /etc/apache2/conf.d/modsecurity
cd /etc/apache2/conf.d/modsecurity
sudo wget http://www.modsecurity.org/download/modsecurity-core-rules_2.5-1.6.1.tar.gz
sudo tar xzvf modsecurity-core-rules_2.5-1.6.1.tar.gz
sudo rm CHANGELOG LICENSE README modsecurity-core-rules_2.5-1.6.1.tar.gz

5. Enable

sudo a2enmod mod-security

6. Restart Apache

sudo /etc/init.d/apache2 restart

Note the comments in that guide (specifically comment's #17 and 18) - I had an error when I restarted Apache like this:

... waiting apache2: Syntax error on line 232 of /etc/apache2/apache2.conf: Syntax error on line 132 of 
/etc/apache2/conf.d/modsecurity/base_rules/modsecurity_40_generic_attacks.data: /etc/apache2/conf.d/modsecurity/base_rules
/modsecurity_40_generic_attacks.data:170: 
<input> was not closed.\n/etc/apache2/conf.d/modsecurity/base_rules/modsecurity_40_generic_attacks.data:132: <![cdata[> was not closed.

As noted in those two comments, you will need to modify your apache2.conf and change:

Include /etc/apache2/conf.d/

To:

Include /etc/apache2/conf.d/*.conf
Include /etc/apache2/conf.d/security
Include /etc/apache2/conf.d/localized-error-pages
Include /etc/apache2/conf.d/charset

Another nice little trick is to make Apache report itself as IIS by adding:

SecServerSignature "Microsoft-IIS/5.0"

To your modsecurity_crs_10_config.conf

While you're at it, you may also want to include mod_evasive.

Troubleshooting

The best resource for help on troubleshooting your installation is available in the LogZilla Forums

The list below offers some of the collected fixes that have been reported.

Mysql Error: Event Scheduler Damaged

Sample Error:

DBD::mysql::st execute failed: Cannot proceed because system tables
used by Event Scheduler were found damaged at server start at ./
install.pl line 361, <STDIN> line 17.

This error occurs when an upgrade has been performed to MySQL 5.1 You need to also upgrade your databases, like this:

mysql_upgrade -u root-p --verbose

Then restart Mysql.

Mysql Error: The used command is not allowed with this MySQL version

If you get this error when debugging the db_insert.pl script, then it means you have the load data local infile command disabled. Check the MySQL manual for more information.

Blank page after install

If you get a blank page after installation, check your apache error log. If it says:

PHP Fatal error:  license.txt is missing from the logzilla directory. Please visit http://www.logzilla.info or contact cdukes@logzilla for a valid license. in Unknown on line 0

Then you will need to go get a license (free or commercial) from http://www.logzilla.info and place it into the logzilla directory.

Install Guide for LogZilla v3.0