From NMSWiki

About

This page will help guide you through a standard installation process.

For this demo, I will be using the latest version (v2.9.8) located at The Google code site

Requirements

Please note that you should have a decent understanding of Linux, Apache, Mysql and PHP before attempting to use this software. I've made every attempt to make it as easy as possible, but due to the nature of this stuff, it does require a decent skill base to implement and operate properly.

The following tools/software must be installed in order to use Php-Syslog-NG:

Syslog-ng

This software system is based on data collected from a program called Syslog-NG which is "is an open source implementation of the Syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport."

In the case of php-syslog-ng, we're going to use it to collect syslog messages from our network and store the information into a MySQL database for reporting capabilities.

Here's a simple diagram of how these pieces fit together:

Installing syslog-ng

This section covers syslog-ng using Ubuntu, if you use a different distro you're on your own...

From a console, type:

sudo aptitude install syslog-ng

You will likely see an error in ubuntu like this:

Remove the following packages:
ubuntu-minimal

It's a bug in Ubuntu and it's safe to continue.

Once the syslog-ng installation completes, modify /etc/syslog-ng/syslog-ng.conf and add the following lines to the bottom:

###########################################################################################
# Added by cdukes for php-syslog-ng // ref: http://gentoo-wiki.com/HOWTO_setup_PHP-Syslog-NG
###########################################################################################
options { 
long_hostnames(off);
# doesn't actually help on Solaris, log(3) truncates at 1024 chars
log_msg_size(8192);
# buffer just a little for performance
sync(1);
# memory is cheap, buffer messages unable to write (like to loghost)
log_fifo_size(16384);
# Hosts we don't want syslog from
#bad_hostname("^(ctld.|cmd|tmd|last)$");
# The time to wait before a dead connection is reestablished (seconds)
time_reopen(10);
#Use DNS so that our good names are used, not hostnames
use_dns(yes);
dns_cache(yes);
#Use the whole DNS name
use_fqdn(yes);
keep_hostname(yes);
chain_hostnames(no);
#Read permission for everyone
perm(0644);
# The default action of syslog-ng 1.6.0 is to log a STATS line
# to the file every 10 minutes.  That's pretty ugly after a while.
# Change it to every 12 hours so you get a nice daily update of
# how many messages syslog-ng missed (0).
stats(43200); 
};
source s_everything { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); udp(); };
destination d_mysql {
program("/usr/bin/mysql -usyslogadmin -psyslogadmin syslog"
template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg)
VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n")
template-escape(yes));
};
log {
source(s_everything);
destination(d_mysql);
};
# Added above by cdukes for php-syslog-ng enhancements

Now restart syslog-ng and make sure you don't see any errors.

 sudo /etc/init.d/syslog-ng restart
* Stopping system logging syslog-ng                                                                                                     [ OK ] 
* Starting system logging syslog-ng

Apache

Naturally, if we're going to use a web interface, we'll need a web server :-)

In this installation, we'll be using an Apache name-based virtual host.

You don't have to use a vhost, but it's what I use.

Depending on your flavor of linux, the vhost configuration should look something like this:

<VirtualHost *>
   ServerName logs.domain.com
   ServerAlias logs.domain.com *.logs.domain.com
   ServerAdmin cdukes
   DocumentRoot /www/php-syslog-ng/html
   <Directory />
       Options FollowSymLinks
       AllowOverride All
   </Directory>
   <Directory /www/php-syslog-ng/html>
       Options -Indexes FollowSymLinks MultiViews
       AllowOverride All
       Order allow,deny
       allow from all
   </Directory>
   ErrorLog /var/log/php-syslog-ng/apache_error.log
   LogLevel warn
   CustomLog /var/log/php-syslog-ng/apache_access.log combined
   ServerSignature On
</VirtualHost>

There some important things to note here:

1. AllowOverride should be set to "All" so we can modify php variables using a .htaccess file (more on that later)
2. Replace ServerName and ServerAlias with your server's name
3. Replace ServerAdmin with your admin name
4. log files are set to be stored in /var/log/php-syslog-ng so make sure that directory exists and is writeable by your webserver user!

mkdir -p /var/log/php-syslog-ng

Now save this file in the appropriate location. For Ubuntu users, it would be saved as:

/etc/apache2/sites-available/php-syslog-ng

Then, for Ubuntu, you would type:

a2ensite php-syslog-ng

If you are on a distro other than Ubuntu, you'll have to look up the documentation to see how to implement a virtual host.

Once you have apache set up, test it to make sure it's working:

mkdir -p /www/php-syslog-ng/html
echo "Hello World" > /www/php-syslog-ng/html/index.php
# Note that the echo statement above will overwrite any data that you have in index.php if it already exists!

Installing Apache from scratch

This section covers Apache using Ubuntu, if you use a different distro you're on your own...

From a console, type:

sudo aptitude install apache2

(yeah, it's that easy...see why I use Ubuntu?)

After the install completes, edit the apache2.conf file and add a ServerName directive

sudo vi /etc/apache2/apache2.conf

When you're done, it should look like this (replace cdukes-lnx with your server's name):

ServerRoot "/etc/apache2" <<- Existing line
ServerName cdukes-lnx

Now restart Apache and make sure you don't see any errors.

 sudo /etc/init.d/apache2 restart
* Restarting web server apache2                                                                               [ OK ]

MySQL

Php-syslog-ng is designed to work with Mysql v5.x

It may work fine with older versions, but I'm not sure so stick with 5.x when possible.

PHP

Php-syslog-ng is designed to work with PHPv5.x

If you are using PHP v4, and plan to use the CEMDB (Cisco Error Message Database), then you will need to convert the CEMDB.class file to an older format as noted in the issues list on the code site:

http://code.google.com/p/php-syslog-ng/issues/detail?id=13&can=1&q=cemdb

If you don't, you will get errors like this:

Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or
T_FUNCTION or T_VAR or '}' in
/var/www/php-syslog-ng/includes/CEMDB.class.php on line 11

Installing PHP from scratch

This section covers PHP using Ubuntu, if you use a different distro you're on your own...

You will need to install the following, not just php:

• php5
• php5-cli
• php5-gd
• php5-mysql

From a console, type:

sudo aptitude install php5 php5-cli php5-gd php5-mysql

Note that we are installing both PHP5 and the CLI version - this is important as some of the scripts use the command line processor.

After the install completes, edit the php.ini file and change a few settings that we'll need during the web portion:

sudo vi /etc/php5/apache2/php.ini

Change:

memory_limit = 16M 

to:

memory_limit = 128M 

Change:

max_execution_time = 30

to:

max_execution_time = 300

Don't forget to restart apache to start using PHP

 sudo /etc/init.d/apache2 restart
* Restarting web server apache2                                                                               [ OK ]

PERL

Perl is used mainly for some backend data and cleanup functions - most of these files are located in the scripts/ directory.

It's important that you have a working copy of perl installed

Misc Requirements

While not absolutely necessary to have a working system, the following tools/programs should be part of your operating system:

Logrotate

A sample logrotate.d script is included in the scripts/ directory which should (depending in your distro) be placed in /etc/logrotate.d

The file contents should be similar to:

#
# Php-Syslog-ng logrotate snippet
# contributed by Clayton Dukes
#
/var/log/php-syslog-ng/*.log {
  missingok
  compress
  rotate 5
  daily
  postrotate
  /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
  endscript
}

Note, again, that you will want to be sure you've created the directories used to store log data:

(skip this step if you already did it before)

mkdir -p /var/log/php-syslog-ng

Cron

Cron is used for regular data maintenance, the following entries should be made in root's crontab by typing

crontab -e

as the root user on your system

# PHP-Syslog-NG
@daily php /www/php-syslog-ng/scripts/logrotate.php >> /var/log/php-syslog-ng/logrotate.log 
@daily find /www/php-syslog-ng/html/jpcache/ -atime 1 -exec rm -f '{}' ';' 
*/5 * * * * php /www/php-syslog-ng/scripts/reloadcache.php >> /var/log/php-syslog-ng/reloadcache.log
*/5 * * * * php /www/php-syslog-ng/scripts/SqueezeDB-v2.0.php >> /var/log/php-syslog-ng/squeezedb.log

If you are not using Ubuntu, you may need to play with the /5 * * * * settings to get it to work, basically, we want those jobs to run every 5 minutes.

IMPORTANT NOTE: Remove the line using #SqueezeDB if you are not planning to use that feature!

Graphing

To use the graphing feature, you must install the mstcorefonts package. In Ubuntu, type:

sudo aptitude install msttcorefonts

This will install the mstcorefonts into:

/usr/share/fonts/truetype/msttcorefonts

Advanced Features

This section covers advanced options for implementation

Authentication Methods

Some alternate authentication methods are available which include:

LDAP

To enable LDAP, simply edit the config.php file located in config/ after the install is complete and set the following variables:

define('LDAP_ENABLE', "NO");
define('LDAP_SRV', "ldap.company.com");
define('LDAP_BASE_DN', "ou=active, ou=employees, ou=people, o=company.com");
define('LDAP_CN', "uid");

MS AD

To enable MS Active Directory, simply edit the config.php file located in config/ after the install is complete and set the following variables:

define('LDAP_MSAD', "NO");
define('LDAP_DOMAIN', "mydomain.com");

Web Basic Auth (htaccess)

To enable htaccess support using mod_auth_krb5, simply edit the config.php file located in config/ after the install is complete and set the following variables:

define('WEBBASIC_ENABLE', FALSE);

Note that you will still need to add the appropriate user to the MySQL database before this will work.

For assistance with setting up apache .htaccess, please visit http://home.golden.net/htaccess.html

For more information on this feature, please visit http://code.google.com/p/php-syslog-ng/issues/detail?id=62

SqueezeDB

The SqueezeDB concept is a fairly new feature of PHP-syslog-ng and should REALLY know what you are doing prior to using it!

I had a lot of users who were getting tons of log data and came up with this method to help filter and store data in a more efficient matter.

File: SqueezeDB-v2.0.php

This file (meant to be run from cron) will go through your logs table and perform a "de-duplication" of rows.

It uses a an algorithm similar to Levenshtein's to compare the number

of "edits" it would take to make each of the compared strings become the same.

The result is displayed as a percentage (a variable you can alter at the beginning of the file)

I have seen an AMAZING improvement in my tests so far:

Averages of 90% efficiency after the script runs

Before implementing this feature, please be aware of the potential damage you can do (i.e. deleted rows).

I encourage you to read more about it here

Note that in order to run the SqueezeDB script, you will need to modify your tables like so:

ALTER TABLE logs ADD counter INT NOT NULL DEFAULT 1;
ALTER TABLE logs ADD fo datetime default NULL;
ALTER TABLE logs ADD lo datetime default NULL;

DBGen

The dbgen.pl file located in the scripts/ directory is used to generate random fake events for testing.

This file can be used to test whether or not php-syslog-ng is working without having syslog-ng installed. It's written in PERL and you may need to install a couple of modules in order for it to work properly.

Generate a few fake events to verify that the php-syslog-ng part works:

sudo perl /www/php-syslog-ng/scripts/dbgen.pl 

I got an error on my system

Can't locate Net/MySQL.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib
/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /www/php-syslog-ng/scripts/dbgen.pl line 22.
BEGIN failed--compilation aborted at /www/php-syslog-ng/scripts/dbgen.pl line 22.

Which means I don't have the proper perl libraries installed, so I used cpan:

cpan -i Digest::SHA1 (required to install Net::MySQL)
cpan -i Net::MySQL

Now try dbgen.pl again:

/www/php-syslog-ng/scripts/dbgen.pl

And viola'

Debug off, showing only inserted data...
Host: server-5
Facility: authpriv
Priority: debug
Level: alert
Tag: Tag
YMDHMS: 2008-05-10 21:42:38
Program: DBGen
Message: DBGen: %FABRIC-5-FABRIC_MODULE_ACTIVE: Module 3 is online 
Affected row: 1

Now check your url to see the entries:

http://<url>

Installation Process

This section will guide you through a step-by-step installation process, the end of the section includes screencasts so that you can get an idea of how it should work if things go wrong :-)

NOTE: Before starting this process, be sure you've completed the items listed in the #Requirements section.

Console Portion

Create web root and change to that dir (we'll want to do most of this as root, so su first):

sudo su -
mkdir -p /www && cd /www

Download:

wget http://php-syslog-ng.gdd.net/current.tgz

Extract:

 tar xzvf php-syslog-ng-2.9.8.tgz

Create apache virtual host entry and make log directory (see #Apache)

cd php-syslog-ng/scripts

Now edit apache.vhost to your match what I outlined in the #Apache section

Next, copy it over and enable it:

cp apache.vhost /etc/apache2/sites-available/php-syslog-ng
a2ensite php-syslog-ng

Make sure you have your log directory created:

mkdir -p /var/log/php-syslog-ng

Reload apache to start using the site:

/etc/init.d/apache2 reload

This concludes the console portion of the install, now point your browser to the URL you set to continue to the...

Web Portion

Make sure everything on the pre-installation check screen is green, if not, fix it before continuing!

Screen 1: Click Next at the top right to begin install

Screen 2: Accept the license agreement and click next

Screen 3: Enter the mysql ROOT user's password Leave everything else as default unless you really need to change something. (You may want to uncheck the "install sample data" box) Click Next to continue:

Click ok to accept the notice and continue...

Screen 4: Enter a site name, eg: "My Syslog Server" and click Next

Screen 5: Leave the default fields as is unless necessary. Enter email address into email field Enter a password for the admin or leave the random one there (but write it down so you can get into the site later!) When you're done, click next

Screen 6: If you opted to install the CEMDB, then you will be presented with the following screen:

click Install CEMDB to continue...

special note for Internet Explorer users: 2 people have reported that this button (Install CEMDB) does not work for them. You will need to use Firefox in order to make it work, or manually import the sql data from the command line. Sorry, maybe someday MS will decide to make a standards-based browser, or someone will fix this incompatibility :-)

You should now see a page that looks like the following: Click the "Start Import" link to begin inserting CEMDB data into the database.

Once it completes, click on "Continue" near the bottom.

Once you click continue, you should be at the main login screen:

Main Site: Login using admin and the password you selected on screen 5

If you installed the sample data there will be a couple of entries, if not, you'll get an error message like this:

Not to worry, that error just means that log messages have yet to arrive in the database.

This concludes the Web based portion of the install, hooray!

Console Portion (part 2)

Again? Yes...again :-)

A couple of things left to do: If you did not install everything to the default path of /www/php-syslog-ng then you will need to modify the paths set in some of the scripts in the scripts/ directory.

cd /<install_path>/scripts

Next, edit any file that has /www/php-syslog-ng and replace with your path. You can use perl to do this, or you can do each one manually.

Edit each using vi:

 for file in `sudo grep -l "www/php-syslog-ng" *`; do sudo vi $file; done

Perl:

perl -i -pe 's/\/www\/php-syslog-ng/<new_path>/g' *

Replace <new_path> with your installed path, like so:

perl -i -pe 's/\/www\/php-syslog-ng/\/var\/www\/psng\//g' *

Note that you must escape the forward slashes on the directories using a backslash -- If you don't understand how to use perl, you may want to try the manual method instead :-)

Screencasts

This screencast walks through a normal installation of the Web-based portion of the install:

Media:Php-syslog-ng_webinstall.ogg Filetype: OGG

I would advise that you right-click the link and save the video locally rather than trying to stream it which may slow it down a bit.

Caveats

If you get an error running logrotate.php that looks like this:

Query failed: DROP,ALTER command denied to user 'syslogadmin'@'localhost'

Then try running this in mysql:

USE mysql;
update user set Drop_priv='Y',Alter_priv='Y' where User='syslogadmin';
FLUSH PRIVILEGES;

Upgrade Procedures

Although I include a patch file for upgrades in the upgrades/ directory, here's the method I use to upgrade my servers:
1. Rename the current directory to .old

mv php-syslog-ng php-syslog-ng.old

2. Untar the new one

tar xzvf php-syslog-ng.<ver>.tgz

3. Get a backup of your whole syslog DB in case something goes wrong:

mysqldump syslog > syslog.sql 

5. Get a text listing of all your current log tables

echo "show tables" | mysql syslog | grep "^logs" > list

6. Dump a sql file for each log table in the "list" file created in step 5

for table in `cat list`; do mysqldump syslog $table > $table.sql; done

7. Do a fresh install and tell it to drop the old database
8. Verify that new install works properly
9. Import old data

for t in `ls logs*.sql`; do mysql syslog < $t ; done

Verify everything works as expected. If something blows up, you can always just import your original syslog.sql file :-)