Installing Logzilla on Gentoo Linux

From Network Management Wiki

Jump to: navigation, search

Contents

Moved

LogZilla v4.0 has been released. This version is no longer supported.

The documentation for LogZilla v4.0 is located at http://docs.logzilla.pro









Preface

This guide will walk you through the installation of Logzilla 3.0 on a pre-existing Gentoo Linux system. Once again, this log will -ONLY- walk you through the installation of Logzilla 3.0 and its prerequisites. If you would like more information on Gentoo Linux, please visit their official website.

This guide also makes the assumption that you have a functioning knowledge of portage. Due to the level of customization that Gentoo offers, your experience may vary, and there is no way to document all the possible issues that may arise due to the flags you have added to your installation. If you need help with a problem that arises during your installation, please consult either the Logzilla forums or consult the Gentoo Linux community forums.

If you choose to utilize the Logzilla forums, you can optionally send a private message to user Pryoidain.

PLEASE read this entire document before starting so that you are aware of the process. Logzilla requires certain versions of software, or certain flags to be set. Not reading this document through could result in time spent recompiling a software package just to add a single flag.

About the current version of PHP-Syslog-ng

At the time of writing, the 2.9.8-r1 version of Logzilla (formerly PHP-Syslog-ng) is available as a masked package within the portage tree. No documentation will be provided for unmasking and installing this version, as the 2.* branch is no longer supported by Logzilla's author. Additionally, it is important to note that due to IonCube being proprietary software, it is doubtful that the new version of Logzilla will be injected into Portage, although this may change in the future.

Architecture Information

x86/amd64 Arch

This installation guide will document the installation of Logzilla on an x86 based architecture. It should be noted that there is absolutely no difference in steps between an x86 installation and an amd64 installation. If you are installing on an amd64 platform, simply replace x86 with amd64, and you should be good to go.

As an example, if you see the following line:

echo "dev-db/mysql ~x86" >> /etc/portage/package.keywords

You would instead type:

echo "dev-db/mysql ~amd64" >> /etc/portage/package.keywords

SPARC/SPARC64 Arch

Unfortunately, the IonCube loader that is required by Logzilla is currently unavailable for the Linux kernel on SPARC/SPARC64. IonCube DOES, however, provide a loader for the Solaris Operating Environment. We can assume that this loader will also work on the OpenSolaris operating environment, although this has not been tested.

Other Archs

Despite the fact that Gentoo Linux supports several additional architectures, the IonCube loader does not. IonCube does offer a service to compile their loader on alternative architectures, however, they require a secured account and a small porting fee. If you are interested in this service, you are encouraged to contact IonCube by filing a support ticket.

User Privileges and Session management

Almost everything in this guide will require root privileges to install. Personally, I found it easier to just log out and log back in as root, although you can accomplish the same thing by running.

su - root

Setting up Screen (Optional)

Now, it is important to consider the size of the packages that will need to be emerged to properly install Logzilla. The longest and most involved of which is PHP. It is also important to note that certain packages will need to be upgraded. If you would like the ability to keep the process running in the background without needing to keep a terminal open, you can utilize the screen command. First, make sure screen is installed by emerging it with the following command:

emerge -va screen

Once you have emerged the program, you can start a new screen session with the following command:

screen -S syslog

You can substitute "syslog" with whatever you want the screen session to be named. When you are ready to detach from the screen ((i.e. during a compile)) just hit Control-A + D. Once again, that is hitting CTRL and A, and then hitting D. When it is time to re-attach to the session, just type

screen -x syslog

Again, where "syslog" is the name you chose for your screen session. This is just a basic method of utilizing screen, it is really a very useful program. If you would like to learn more about screen, there is an excellent article on the Gentoo wiki, which you can access by clicking here

Prerequisites

PLEASE make sure to read through this entire section. If you goof a prerequisite, it is unlikely you will find out about it until the final stage of installing Logzilla, and then you will have to back track, resulting in wasted CPU time from compiling the same package multiple times.

Portage will automatically pull in any packages needed, and since some of these packages on others listed here, I have elected to build -UP- to the final level. Please make sure to follow the guide as stated.

GCC

GCC comes standard with Gentoo. At the time of writing, the current version of GCC on x86 platforms is 4.3. There is no additional work needed here.

Perl

Lucky for us, this comes pre-installed as well, as the portage system runs on top of it. However, Logzilla requires some additional packages to be installed for Perl to function.

  • Date::Calc
  • Text::LevenshteinXS
  • String::CRC32

These packages can be installed in one of two ways. The official Logzilla installation guide recommends cpan, however, these packages are set up in portage, and utilizing cpan means the packages will NOT be recorded in the world package set. This will most certainly cause file conflicts if another package ever attempts to emerge these modules, therefore, we will use portage.

emerge -va dev-perl/Date-Calc dev-perl/Text-LevenshteinXS dev-perl/string-crc32

It is likely that there will be two additional packages pulled in, namely Carp-Clan and Bit-Vector. This is normal, and would have happened even if the cpan method had been used.

MySQL

Logzilla requires MySQL version 5.1+. At the time of writing, 5.0 is the most recent version in the stable portage tree. You can check to see what version is available by running the following command.

emerge --search mysql

Look through what appears. If you see the following:

root@sakaki ~ # emerge --search mysql
Searching...
[ Results for search key : mysql ]
[ Applications found : 32 ]

*  dev-db/mysql
----->Latest version available: 5.0.90-r2
      Latest version installed: 5.0.90-r2
      Size of files: 22,806 kB
      Homepage:      http://www.mysql.com/
      Description:   A fast, multi-threaded, multi-user SQL database server.
      License:       GPL-2

Then you will need to add a keyword to your package.keywords file to unmask and upgrade to MySQL 5.1. Follow the directions under the "Unmasking MySQL 5.1" section below. However, if the search term returns and you see:

root@sakaki ~ # emerge --search mysql
Searching...
[ Results for search key : mysql ]
[ Applications found : 32 ]

*  dev-db/mysql
----->Latest version available: 5.1.46
      Latest version installed: 5.1.46
      Size of files: 28,695 kB
      Homepage:      http://www.mysql.com/
      Description:   A fast, multi-threaded, multi-user SQL database server.
      License:       GPL-2

Then you are all set, and you can skip ahead to the "Installing MySql" section below.

Unmasking MySQL 5.1

If you are reading this, then it means you need to unlock, or Unmask, the 5.1 version of MySQL. At this point, you are mixing the stable and testing branches of the portage database. While this is common practice, it is advisable that you read through the Mixing Software Branches section of the Gentoo handbook, to familiarize yourself with the process.

To unmask MySQL 5.1, you need to add the both the daemon package AND the virtual package to your Keywords directive. This is accomplished with the commands below.

echo "dev-db/mysql ~x86" >> /etc/portage/package.keywords
echo "virtual/mysql ~x86" >> /etc/portage/package.keywords
echo "dev-db/mysql extraengine" >> /etc/portage/package.use

If you have never modified Package files before, it may be necessary to create the portage directory, which can be accomplished with the command below.

mkdir /etc/portage

Now, if you need to UPGRADE to MySQL 5.1 from a MySQL 5.0 installation, follow the directions under the "Upgrading to MySQL 5.1" section, and skip over the "Installing MySQL 5.1" section. The easiest way you can check if you are upgrading is to run:

emerge --search mysql

Look through the output. If you see:

root@sakaki ~ # emerge --search mysql
Searching...
[ Results for search key : mysql ]
[ Applications found : 32 ]

*  dev-db/mysql
----->Latest version available: 5.1.46
----->Latest version installed: 5.0.90-r2
      Size of files: 22,806 kB
      Homepage:      http://www.mysql.com/
      Description:   A fast, multi-threaded, multi-user SQL database server.
      License:       GPL-2

Then you need to upgrade.

Installing MySQL 5.1

If you are reading this section, then it means you need to install MySQL 5.1 for the first time. This is different from upgrading, in that you need to properly configure the MySQL daemon after installing.

First, emerge MySQL.

emerge -va mysql

You're probably going to want to go and get a cup of coffee, as MySQL can take up to an hour to compile on some newer systems, and up to two hours on older systems. Perhaps go and watch a stand-up performance, might I suggest George Carlin? Once the emerge operation is completed, you will notice some information about configuring MySQL for its first run. To do this, run the following command.

emerge --config =dev-db/mysql-5.1.46

This will set up the initial database and configure your root password. ALWAYS SET A SECURE ROOT PASSWORD, even if this system will not be exposed to the outside world. Now that you have properly configured the daemon, let's go ahead and run it.

/etc/init.d/mysql start

If everything goes well, you'll see a [OK] pop up on the right hand side of your terminal. This means your MySQL configuration is properly set up, so now we just need to configure it so that it runs at boot, to do this, add it to the default runlevel.

rc-update add mysql default

Congratulations! You have properly installed the MySQL Daemon. Now proceed to the Syslog-NG installation guide below.

Note: I highly recommend you take a look at the Gentoo MySQL guide for more information on running MySQL on Gentoo. The document is slightly dated, but still a good read.

Upgrading to MySQL 5.1

Upgrading your existing MySQL setup to 5.1 isn't too difficult, but it does require a few extra steps. First, make sure you have unmasked the package as instructed above. Next, you need to emerge the new version of MySQL.

emerge -vua mysql

Now, you are going to be emerging 4 packages, 2 of them are dependencies required due to new features in MySQL. This WILL take a while, so go and get yourself some coffee and snacks, and maybe watch some stand-up. Might I suggest the comedic stylings of George Carlin? This process has been known to take up to an hour on new machines, and two or more hours on older machines.

Now that the installation is complete, it is time to update your system to utilize the new libraries. Many programs depend on the MySQL library, and specifically, the Shared Object libraries, which have had their version numbers changed. This is where the Reverse Dependency rebuild comes in handy. First, make sure you have the Gentoolkit installed, by running the following command.

emerge -va gentoolkit

Once the installation is complete, you need to tell the system that the shared object libraries have been changed. Since revdep-rebuild simply looks for package names, we need to specify specific dependencies to rebuild for. Execute the following two commands in order. Please note that the commands should be ran as separate processes, one after the other.

revdep-rebuild --library libmysqlclient.so.15
revdep-rebuild --library libmysqlclient_r.so.15

If you're running on a mature system that is heavily reliant on MySQL, this may take a while. After everything is complete, you could choose to run one last instance of revdep-rebuild without any parameters, just to verify everything is up to date. This step is optional though, and is only for the truly paranoid. ;)

Now, let's make sure your configuration files are up to date by running dispatch-conf.

dispatch-conf

Note that any questions about dispatch-conf are far beyond the scope of this article, or the Logzilla community. If you run into questions, please consult the Gentoo community. The next step is to update the databases. MySQL 5.1 utilizes newer features than 5.0, such as the event scheduler, and partitioning support. To upgrade the internal databases, run the following command.

mysql_upgrade -p

Once the upgrade is complete, restart the MySQL database, and any reliant programs. If your system was heavily reliant on MySQL, it may actually be easier to just reboot. Whichever way you choose, make sure that MySQL is restarted and running properly.

/etc/init.d/mysql restart

Finally, just to make sure MySQL is added to the default runlevel, execute an rc-update.

rc-update add mysql default

Configuring MySQL

The new version of Logzilla requires the event scheduler built in to the new version of MySQL to be enabled. To enable it, execute the following commands.

mysql -p
mysql> SET GLOBAL event_scheduler = 1; 
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @@event_scheduler;
+-------------------+
| @@event_scheduler |
+-------------------+
| ON                |
+-------------------+
1 row in set (0.00 sec)

mysql> quit;

After this, we need to make sure the event scheduler remains on between reboots. To do this, open your my.cnf file, and add the following line of code under the [mysqld] section.

event_scheduler=ON

While in the my.cnf file, it is advisable that you disable binary logging. This logging function is only useful in the most extreme of debugging situations, and in addition to putting tremendous strain on the MySQL server, it can lead to unexpected execution, which you can read about here. To do this, arrow down to around line 57.

Replace

log-bin

with

#log-bin

or remove it entirely.

Syslog-NG

Gentoo Linux provides for many system loggers, the most common of which are syslog, Metalog, and syslog-ng. For purposes of this guide, we will assume you are using syslog-ng. This guide may be upgraded in the future to allow for the use of multiple types of logging daemons.

To begin, we are going to specify that Syslog-NG be built with a local package specific use flag. Since the 3.0 release, syslog-ng supports the capacity to write directly to MySQL and other SQL databases, but it has to be built with this feature. Although Logzilla still makes use of file based pipes, emerging syslog-ng with the sql flag only adds a few extra megabytes, and opens up a new, fairly valuable feature. You can also view it as a form of future-proofing, because in the event that the author of Logzilla decides to make use of the new SQL logging features, you will not need to recompile syslog-ng.

Let's specify that syslog-ng be built with the sql flag by running the following command.

echo "app-admin/syslog-ng sql" >> /etc/portage/package.use"

Now that we have properly set up the USE flags, we should emerge syslog-ng using the following command.

NOTE: Syslog-ng requires the use of a Mail-Transfer-Agent (MTA) to function. If you have not already emerged a MTA, portage will automatically pull in the highly simplistic sSMTP. If you would like to use a more advance MTA, you should emerge it NOW, before you emerge syslog-ng. For more information, you can consult the Gentoo wiki page on Postfix or Exim. Please note that it is also possible to just install syslog-ng, and allow it to pull in the ssmtp dependency, and then remove ssmtp and replace it with your MTA of choice later. Any problems encountered during the installation of a MTA other than sSMTP is beyond the scope of this document and the Logzilla community, and should be discussed on the Gentoo community forums.

emerge -va syslog-ng

Go and grab yourself a quick snack, syslog-ng will only take about 5-10 minutes to install on newer systems, and about 20 minutes on older systems, and that includes the time needed to install the dependencies. Now that it is installed, run it once so that it will create all the files it needs, then go ahead and stop it.

/etc/init.d/syslog-ng start
/etc/init.d/syslog-ng stop

Now that you have completed installing syslog-ng, let's go ahead and add it to the default runlevel.

rc-update add syslog-ng default

Now you can continue with installing Apache.

Apache

Gentoo supports multiple web servers. However, for purposes of this guide, we will assume that you are running Apache, as it is one of the most common web servers currently in use.

Just like syslog-ng, we need to enable a few additional flags for use with Apache. Namely, we need to tell it to support PHP and mysql. If you're worried about Apache pulling in the PHP package before we get to it, don't be. For some reason, even compiling Apache with the PHP use flag will not pull in the PHP build as a dependency. This is good for us, because PHP requires quite a bit of tweaking before we can install it properly, which is why it has its own rather large section below.

Let's tell portage that Apache will need to support MySQL as well as PHP. Optionally, You may want to enable virtual hosts on Apache. If you choose to enable virtual hosts, I highly recommend you add it to your make.conf file as well as your Local Package USE flags.

To enable Apache to utilize MySQL and PHP without virtual hosting, run the following command.

echo "www-servers/apache mysql php" >> /etc/portage/package.use

To enable Apache to utilize MySQL and PHP with virtual hosting, run the following TWO commands. ((The second echo to your make.conf is technically optional, but HIGHLY recommended.))

echo "www-servers/apache mysql php vhosts" >> /etc/portage/package.use
echo "USE=${USE} vhosts" >> /etc/make.conf

Now, we need to emerge the Apache web server. Run the following:

emerge -va apache

This process takes roughly 10-15 minutes to compile JUST the core Apache library. Depending on how new your system is ((in other words, how many dependencies need to be installed)) this could take upwards of 30 minutes on newer systems, possibly an hour or more on older systems.

Once again, we need to start and stop Apache to have it create the nescessary files for us to edit later on in this guide. Execute the following two commands.

/etc/init.d/apache2 start
/etc/init.d/apache2 stop

Now, let's make sure to add Apache to the default run level, so that it executes on boot.

rc-update add apache2 default

PHP

READ THIS SECTION

At the time of writing, there is a known bug that involves PHP Version 5.2+ and GCC Version 4.2+ due to an optimization error. This bug results in segmentation faults when running PHP on Apache web servers. You can read more about this bug in the official Gentoo Bugzilla, Number 234177 and 307779. While it is true that this bug may not effect you, it is just as likely that it will. Because of this factor, this guide will take specific steps to AVOID this bug at all costs. Please note that ignoring the steps required for avoiding this bug means you waive the right to support. We cannot help you with problems with Logzilla unless we can be assured that you are running a stable version of PHP.

For those of you thinking about utilizing PHP 5.1 to bypass this bug all together, PHP 5.1 is no longer listed in the portage tree, and as such, installation instructions for it will not be provided.

How to bypass the GCC 4.2+ Bug

There is a method to bypass the bug completely, there are in fact, two methods. Both of these methods will result in non-optimal code, but one of them more-so than the other. You can either choose to force PHP to build with GCC debug symbols ((resulting in incredibly bloated code)) or you can choose to compile PHP without -any- optimization flags. While this means the code wont be as fast as it could be, it will result in an executable size similar to that of the PHP binary release. We will go with the non-optimization method.

To begin, you need to set Package Specific GCC compilation flags. This is typically only done by Gentoo developers, or highly experienced users. Note that this procedure is not difficult, just uncommon, and it will bear no effect on your global configuration. To bypass the bug, run the following commands.

mkdir -p /etc/portage/env/dev-lang
nano -w /etc/portage/env/dev-lang/php

Once inside the file, type in the following two lines:

CFLAGS="${CFLAGS/-O?/} -O0"
CXXFLAGS="${CFLAGS}"

Hit CTRL-O to save the file, and CTRL-X to exit. Now that you have effectively bypassed the bug, lets move on to installing PHP.

Install PHP

Before we install, we need to make sure that all the features required by Logzilla are set to be installed. Run the following command:

echo "dev-lang/php json" >> /etc/portage/package.use

Installation is one of the easiest things to do. You should know the drill by now. :P

emerge -va php

Installing just the PHP package alone can take upwards of 35 minutes on newer hardware, not including time required to compile additional dependencies. On older hardware, the package can take upwards of two hours! If you haven't napped, now would be the time to do so. Once everything has completed, you should restart Apache to initialize the freshly installed PHP infrastructure.

/etc/init.d/apache2 restart

With that, you've satisfied all the prerequisites! Now we can move on to actually downloading and installing logzilla!

Logzilla

Obtaining Logzilla

Please see refer to the main installation guide for software and licensing (free or commercial):

Install_Guide_for_LogZilla_v3.0#Obtaining_LogZilla

Installing Logzilla

Change to your desired installation location and extract. I highly recommend installation under /var/www, but you can install anywhere.

cd /var/www
tar zxvf /tmp/logzilla_x.x.x.tgz

Now, to actually run the installation, we will execute the perl script provided with Logzilla.

cd /var/www/logzilla/scripts
./install.pl

Installing Sphinx

Sphinx via Portage

Note: Sphinx is present in the Portage tree. However, installing Sphinx globally will require tweaking several files that come included with Logzilla. While I will eventually add this information, currently the only documented method is to use the pre-packaged version included with Logzilla. There are some individuals who are quite possibly already running a Sphinx engine on their systems. You are encouraged to document your experiences, either through a post on the forums, or by adding to this wiki.

Sphinx via Logzilla

We will use the distribution of sphinx that comes packaged with Logzilla.

Please follow the main installation guide - Sphinx section: Install_Guide_for_LogZilla_v3.0#Installing_Sphinx

Syslog-NG Configuration

By default, the Gentoo syslog-ng configuration does not support remote logging. So we will enable it by adding the following to your syslog-ng configuration file. You should add this within the src directive, which is defined by the following line.

source src {

Immediately after, add the following lines. Make sure to substitute x.x.x.x with the IP address of your machine.

        udp(
                ip(x.x.x.x)
                port(514)
        );
        tcp(
                ip(x.x.x.x)
                port(514)
        );
Personal tools