Bazsi's blog

syslog-ng rewrite use case: dpkg logs

2009-07-08T09:43:00.002+02:00

One of my collegues (Péter Höltzl, he does all our trainings) has created a nice detailed example on how to use the parser/rewrite framework to pull in yet another application into syslog: dpkg, the Debian package manager.

If you are interested in what rewrite/parser can do for you, but didn't have the time to find out, the blog post is worth a read.

syslog-ng pipelines

2009-06-19T06:54:00.003+02:00

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
type(pcre) value('rdate'));
};

The way it works is as follows:

the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro)
the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens"
Later on in the configuration you can use "rdate.second_tens" just like any other macro/value.

This proves that the current rewrite/parser/filter subsystems are really powerful, however even though this proved to be possible, there are some lessons learned from this example:

the macro and name-value space should really converge to each, this would mean that the match() filter could directly match against the macro value $R_DATE without the need for the separate rewrite statement
when you are after a given goal, you don't really want to differentiate rewrite/parser/filter rules at all. The current syntax of using separate blocks for separate type of log processing elements is a pain.

So I'm thinking about inventing yet another block, which simply wouldn't care what kind of processing element is added to it, something along the lines:

pipeline rdateseconds {
set("$R_DATE", value("rdate"));
match('^... .. [0-9]+:[0-9]+:(?[0-9])[0-9]$'
type(pcre) value('rdate'));

};

And then:

log {
source(src);
pipeline(rdateseconds);
destination(dst);
};

Maybe I should even allow the creation of rewrite/parser/filter elements right there in the log statement:

log {
source(src);
filter(facility(mail));
destination(dst);
};

What do you think?

Nordic Meet on Nagios 2009

2009-06-03T10:34:00.003+02:00

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios.

If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website, so you can catch me at 15:50 Central European Time.

syslog-ng 4.0 roadmap plus release policy changes

2009-05-30T19:20:00.004+02:00

I've updated the syslog-ng OSE roadmap on the syslog-ng webpage to include information about the upcoming syslog-ng version:

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap/

Also, I'd like to bring the changed release/support policy to your attention, that you can read at the same location above. I'd like to introduce stable track and feature track releases, the first being supported for a long time, whereas feature track releases are only supported until the next feature/stable release is published. When a sufficient number of features were published via feature track releases, the last one becomes stable and the cycle continues. Note that feature releases are NOT development snapshots, they are releases just like the major versions previously, the only difference is that instead of a large feature list like with syslog-ng 3.0, only a smaller set of changes are included.

This makes it possible to publish features more often, always concentrating on a few of them at a time, instead of doing development for a long time and come out with a feature packed release. I hope to increase the pace of syslog-ng development with this change and also to cause less problem for users who prefer stability over features. Please read the details on the roadmap page.

I've also opened the syslog-ng 3.1 repository and pushed it to our git server. Right now there are no differences (except for the version number) between 3.0 and 3.1, I'm planning to integrate Marton's message tagging and patterndb changes as soon as possible (his git tree is here). Hopefully the 3.1 cycle will be quite short as most of the things on the roadmap are already implemented, although scattered around in various public and private trees.

With the opening of the 3.1 branch, I'm also obsoleting 2.0 (in the new support model two stable track versions are supported at any given time and we have 2.0, 2.1 and 3.0 right now), but that'll go in a separate post/announcement.

syslog-ng OSE 3.0.2 released

2009-05-08T22:23:00.003+02:00

After a long time and a lot of accumulated bugfixes, I've pressed the "release" button and syslog-ng OSE 3.0.2 was published on our website. The first official version to feature binary packages for Linux and BSD platforms. Since there was a long time between 3.0.1 and 3.0.2 the changelog is quite large, however most of it are bugfixes, only some minor enhancements here and there.

Hopefully I didn't miss any important bugs and problems. It must be much better stability/functionality wise than 3.0.1 was.

The diffstat since 3.0.1:
150 files changed, 4332 insertions(+), 3000 deletions(-)

You can also check the patches in our git repository.

If you are using the 3.0 branch you are really recommended to check out this release. If you are using anything earlier than 3.0 you are also recommended upgrade, syslog-ng 3.0 is revolutionary to previous versions in many ways, especially if you want to do more to your logs than merely store them in a plain text file.

OSDC 2009 slides

2009-05-08T15:30:00.003+02:00

I've uploaded my OSDC 2009 presentation slides to
http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp Which has an example for processing iptables logs with db-parser() and putting the results in a customized SQL table.

Nordic Nagios Meet 2009

2009-05-03T20:36:00.003+02:00

I'm going to give a talk on syslog-ng on the upcoming Nordic Nagios Meet 2009. I expect the event to be great fun, just like last year. If you are in the Nordic region and use Nagios, rrdtools or syslog-ng, I recommend to pay a visit as you can meet the primary authors and some active contributors to these projects.

If you are there and have anything to ask/talk about syslog-ng, just feel free to approach me, I'm probably going to wear a badge, so you can recognize me :)

OSDC 2009 and syslog-ng automatic testing

2009-05-03T20:22:00.006+02:00

I've spent the last week in the nice city of Nuremberg where Open Source Data Center Conference took place, organized by Netways AG. I really liked the talks about Puppet, DRBD and the description of the booking.com infrastructure which runs MySQL.

Although I really enjoyed the conference I also had some free time to improve the automatic test program for syslog-ng, which now also covers TLS encrypted source and SQL destinations. I've also implemented a small script to collect coverage data of the testcases, thus right now I know that about 63% of syslog-ng is covered by automatic tests. (initially it was 55% but there were some low hanging fruits). I expect to raise this number easily to around 80%, then it'll probably become much more difficult to increase it further as the rest is error processing paths, and unless I come up with something to inject errors from the testcases those are difficult to test.

Of course having a test suite is not a replacement for real-life, field testing, but nevertheless it makes it much easier to do releases as it ensures that no important functionality is broken completely.

Based on this test infrastructure I'm going to release 3.0.2, after which I'll probably change the way I manage releases for syslog-ng, but I'll talk about that in a forthcoming post.

My son is 7 weeks old

2009-05-03T08:55:00.004+02:00

From Dani-aprilis-25

The reason I was absent from this blog in the couple of last weeks is my now 7 weeks old son, Dániel. You can find a picture of him right here in the post, but some additional ones in my Picasa albums.

Features that fell off the radar

2009-03-23T22:15:00.004+01:00

I was long pondering with the problem that it is quite tricky to enter regexps into syslog-ng configuration file, since if you enclose the string in double quotes (e.g. in ""), the backslash character needs to be escaped.

Since backslash is used in regexps quite often, it can become cumbersome to enter regexps like:

match("[a-z\\-]+");

Note that the backslash is doubled because otherwise the syslog-ng string parser would pass the sequence to the regexps compiler as: "[a-z-]+" which is certainly different in meaning what the above expression says.

I always remembered that syslog-ng also supports single quotes (aka apostrophes), but I remembered they behaved just as if you used normal quotation marks. Therefore I was thinking about a 3rd string format, one that would not require escaping.

However I was reading the related code the other day, and found that apostrophes work exactly the way I planned this 3rd string syntax to behave: not to get in the way when entering regexps. In fact it behaves just like apostrophes in the UNIX shells. It does not care about escaping, it only cares about the terminating apostrophe.

I was dealing with regexp related questions on the mailing list a lot, and the root cause of the problems was most times this escaping stuff, and I never knew the proper answer and behaviour is already in syslog-ng, I've just forgotten about it completely.

And now as I check the documentation for syslog-ng, it does not mention this syntax either, even though it had been present even in the 1.6.x times.

So if you had trouble writing lots of regexps in syslog-ng configuration, and I told you to properly escape your regexps, please forgive me. syslog-ng is better than I've thought :)

Newborn baby

2009-03-16T17:59:00.002+01:00

After about two weeks being late, my son was born yesterday evening at 22:45CET. He weights 3270g and 56cm. Both the mother and the child are fine and I'm a proud new father.

I guess this starts a section in my life, hopefully for the better.

syslog-ng OSE binary packages

2009-03-14T11:06:00.003+01:00

I' happy to announce that BalaBit has decided to make the binary packages for syslog-ng OSE available for free.

As you may know, BalaBit has various syslog-ng support packages and as a part of this service it prepared binary installation packages for different platforms. The access to these packages either required a support contract but could also be purchased separately for a yearly fee.

With syslog-ng 3.0, the binary packages for syslog-ng OSE will become freely accessible.

Since syslog-ng is an open source project, BalaBit planned to finish this task in the Open Source spirit: open and visible to all community members. This also means that the set of packages published with this e-mail is NOT yet release grade, rather it is more of a development snapshot of the current state of affairs. So please don't ruin your production systems with this package, it is more advisable to try them in a test environment (chroot or a dedicated test machine).

With all these said, here is the link:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/upgrades/

Please pick the release named "3.0HEAD". This contains a source snapshot (effectively git from two days ago), and a set of packages for SUSE 10, RHEL4/5, FreeBSD 6.x, Debian etch, and Linux generic.

The binary packages contain all runtime dependencies needed to run syslog-ng, thus no further packages are required, it is an all-in-one package. The rpm/deb packages are prepared the same, they install syslog-ng in /opt/syslog-ng in order to avoid clashes with a system supplied syslog-ng daemon.

There are two install kits for each platform:

one that includes database drivers (dubbed as "server")
one that does not include database drivers (dubbed as "client")

Currently there are no other differences between the packages, but later on there might be.

With the current infrastructure in place, I'm confident that with each syslog-ng OSE release, I can publish the source AND binary packages at the same time.

I'd really appreciate success/failure reports and also any kind of comment you may have.

I'd like to release 3.0.2 together with its binary packages, let's hope that I get enough feedback on these packages so that I can do that.

Enjoy!

First IETF syslog-protocol related question

2009-03-11T22:55:00.003+01:00

I'm happy as I've received the first question about the new IETF specified syslog-protocol support. There's a need for that after all :)

Next event on the horizon

2009-03-11T08:31:00.004+01:00

I didn't realize it is already that time of the year, but I was reminded that I'm going to give a talk on syslog-ng 3.0 on Open Source Data Center conference in Nürnberg, Germany at the end of April. I'm going to talk about the nifty new features of syslog-ng 3.0.

It would be very nice to meet syslog-ng users there. :)

An introduction to db-parser()

2009-03-03T10:58:00.005+01:00

As promised on the mailing list here comes a short description of the new db-parser functionality of syslog-ng. For an introduction to parsers in general see my previous blog post here.

The aim for db-parser is two-fold:

extract interesting information from a log message
attach tags to a log message for later classification.

For instance here's a log sample (lines broken for readability):


Feb 24 11:55:22 bzorp sshd[4376]: Accepted password for bazsi \
      from 10.50.0.247 port 42156 ssh2

This message states that a user named "bazsi" has logged into the host named "bzorp" using SSH2 from the quoted IP and port. When you read this message as a human, the event that happened is perfectly clear. However if it is not a human, but a piece of software that has to make out the meaning of the message, you need to identify the event (e.g. that a user login has happened) and the additional information associated with the event (e.g. that he used 10.50.0.247 as the client).

If I wanted to express this as name-value pairs, it would be something like this:


event="user login", protocol="ssh2", \
      client="10.50.0.247:42156", method="password"

Surely this latter form is easier to analyze than the first. So the first step of all kinds of log analysis is to extract information from messages. At a first glance, the easiest way to extract this information is the use of
regular expressions. For example:


^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sshd\[[0-9]+\]: Accepted \
  (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \
  for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

Once you match with the regular expression above (courtesy of the logcheck project), the parentheseses mark the variable part of the information that you can reference as $1, $2 and so on.

The problem with regular expressions are several fold:

they are difficult to write (just look at the example above)
they are even more difficult to understand, once written (again, please look at the example)
they are slow and they scale poorly with the number of regexps that we need to match against the incoming message stream.

Projects like logcheck use regular expressions, but with the number of patterns increasing, the time needed to analyze logs skyrockets, which makes the whole thing unfeasible. Also, logcheck does not aim at extracting information from messages, it merely classifies them.

Clearly a different approach is needed. And that's what db-parser in syslog-ng is.

The db-parser() functionality of syslog-ng has the following objectives:

use a database to match various messages (and not filters embedded in the configuration file)
classify events into logcheck-like classes (cracking, violation, ignore, unknown)
extract variable information from messages, and place those into name-value pairs
be fast, scale to a high number of events/sec and high number of patterns
integrate well to the rest of syslog-ng

db-parser() is a generic parser, fits nicely to the parser framework inside syslog-ng. You can use it just like csv-parser():


...
parser p_db { db-parser(); };
...
log { source(src); parser(p_db); destination(d_parsed); };
...

The database used by db-parser is an XML file that is read during syslog-ng startup. Here is an example entry from the db-parser() database:


<patterndb>
<ruleset name='sshd'>
 <pattern>sshd</pattern>
 <rules>
   <rule provider='balabit' id='1' class='system'>
     <patterns>
       <pattern>Accepted rsa for@QSTRING:username: @from\
@QSTRING:client_addr: @port @NUMBER:port:@ ssh2</pattern>
     </patterns>
   </rule>
   ...
 </rules>
</ruleset>
</patterndb>

As you can see the database is structured, and the first selection criteria to apply is the name of the application (e.g. the value for $PROGRAM). Then each rule matches against the message payload (e.g. the value for $MESSAGE) with the syslog header stripped off. The rule specifies the classification (e.g. 'system' in the example above) and lists one or more patterns. If any of the patterns match, the rule is considered a match.

The variable part of the pattern is specified using special sequences, starting and ending with a '@' character. Within the enclosing '@' characters a colon separated list of parameters are listed:

the parser to apply (QSTRING and NUMBER in the example above)
the name of the value to be extracted from this position
additional arguments to be passed to the parser

The available parsers are currently not really documented, but here is a
list of them (you can find these in the radix.c source file):

IPv4: to parse an IPv4 address
NUMBER: to parse a number
STRING: to parse a word
ESTRING: to parse a sequence of characters ending with a specific character
QSTRING: to parse a string enclosed within quotes

Of course further parsers can be added to the code easily. You don't have to specify monsterous regexps to match an IPv4 address anymore. Not to mention IPv6 :)

If a message matches a rule, the db-parser() will make the following list of values defined for the given message:

.classifier.class: logcheck-like classification
.classifier.rule_id: the ID of the database entry that matched
pattern specific values: variable part that get extracted from the message by patterns

Each of the values defined previously can be referenced inside syslog-ng using a macro, e.g. you can do things like:


# You can use them in a filter:
filter f_class {
match("system" value(".classifier.class"));
};

# but you can also use them in the names of files:
destination d_parsed {
file("/var/log/messages/${.classifier.class}.log");
};

That's a rough skeleton of what db-parser() is. If you are interested, you can find the db-parser() implementation in syslog-ng OSE 3.0:

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/

You can also find some example pattern databases here:

http://www.balabit.com/downloads/files/patterndb/

We are also thinking about further ideas to enhance db-parser() and make it the foundation of an Open Source log analysis framework. Stay tuned!

GStaticMutex and AIX

2009-01-18T12:02:00.003+01:00

If you use GLib on non-Linux platforms such as AIX and think that G_STATIC_MUTEX_INIT does nothing but zero-initialize the mutex, think twice. Although quite clearly stated in the documentation, I thought I was smarter and used a GStaticMutex embedded in a structure that was zero initialized.

If you look at the definition of G_STATIC_MUTEX_INIT on most platforms (Linux, Solaris, BSDs), it contains nothing but zeroes. This lead me to the impression that zero filling a GStaticMutex instance is enough to initialize it.

In reality it isn't. On AIX this renders the mutex to be entirely useless without warnings or aborts. The results are of course bugs that are difficult to track down and fix.

This took me an entire day to figure out, as the SQL driver in syslog-ng had this problem. This was fixed since, but if you are running syslog-ng on AIX with the SQL driver, be sure to have this patch applied.

syslog-ng OSE 3.0 finally released

2009-01-15T13:20:00.003+01:00

Finally I could take the time to actually announce the freshly released syslog-ng OSE 3.0 branch. It was uploaded to our website during the winter holidays, but I had to integrate syslog-ng OSE to our new release infrastructure, which among others has a much nicer web interface.

Here is a summary on what is new in syslog-ng 3.0:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch01s04.html

Enjoy!

include file support implemented

2008-12-11T23:41:00.004+01:00

I've implemented basic include file/directory functionality in syslog-ng, using the format numbered second in my previous post.

I've now pushed an expermental implementation of include files in the syslog-ng OSE 3.0 repository, in a separate branch called 'include'.

E.g. in order to test the code, please clone the syslog-ng 3.0 repository:

$ git clone git://git.balabit.hu/bazsi/syslog-ng-3.0.git

Then check out the 'include' branch:

$ git checkout --track -b include origin/include

Then compile as usual. I didn't want to integrate it right into syslog-ng OSE 3.0 tree as I'd like to release that first as 3.0.1.

include syntax

2008-12-10T16:16:00.003+01:00

I'm about to implement configuration file includes, and although the implementation is quite straightforward, the syntax to be used is something to give a thought or two.

Currently the syslog-ng configuration file consists of statements, each with the following basic format:

stmt [] { ... };

The "id" gives a unique identifier of the statement, and the braces enclose the contents. Currently only the ID part is optional, the braces are always there.

To make the include statement consistent with that, it'd have to look something like:

include { "filename" };

Obviously I don't like this too much, as it is way different from all other applications permitting the use of include statements. What about this:

include "filename";

E.g. use the ID part the name of the file to be included. I like this better. A third option might be the use of 'pragma' directives, currently only used to specify the file format compatibility in the case of syslog-ng 3.0:

@version: 3.0

This'd mean that include statements would look like this:

@include: filename

The problem with this last option is that pragmas are currently only processed at the beginning of the configuration file. So that code should also be generalized.

I think I'd go with the second option, that's not completely inconsistent, but still the most intuitive to use.

What do you think?

syslog-ng 3.0 and SNMP traps

2008-11-23T13:55:00.007+01:00

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description.

It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload.

Of course it'd be much easier to process syslog messages, if this were not the case.

A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0.

First, you need to filter snmptrapd messages:

filter f_snmptrapd { program("snmptrapd"); };

Then we'd need to grab the first field of the message payload, where snmptrapd is configured to put it:

rewrite r_snmptrapd {
subst("^([^ ]+) (.*)$ ", "${2}");
set("${1}" value("HOST"));
};

Both rewrite expression kinds are demonstrated here:

subst() has two arguments: the first is a regexp to search for, the second is a template to be substituted if there's a match
set() has a single argument: a template to be used as the new value

Rewrite rules operate by the contents of the $MESSAGE value by default, which holds the message payload. Of course this can be changed by specifying the value() option. The notion 'value' in syslog-ng 3.0 refers to a name-value pair, in syslog-ng 3.0 every message is composed of a set of name-value pairs. The names of standard values match the name of the corresponding macro, but without the '$' sign.

Please NOTE that the new value is a template which makes it possible to use macros such as $HOST or $MESSAGE defined by syslog-ng.

Now let's wire the complete configuration together:

filter f_snmptrapd { program("snmptrapd"); };

rewrite r_snmptrapd {
subst("^([^ ]+) (.*)$ ", "${2}");
set("${1}" value("HOST"));
};

log {
source(s_all);
filter(f_snmptrapd);
rewrite(r_snmptrapd);
destination(d_all);
flags(final);
};

log {
source(s_all);
destination(d_all);
flags(final);
};

Of course this is only an example of the power of what syslog-ng is now capable of doing. Please let me know if you can think of other uses.

The current 3.0 branch of syslog-ng has not been released yet, it is available in the git repository at git.balabit.hu, and also as nightly snapshots.

I'd be grateful for any kind of feedback you might have, please post it either as comments on this blog, or to the mailing list.

syslog-ng statistics

2008-11-08T09:05:00.003+01:00

For a long time I meant to give the "log statistics" feature of syslog-ng an overhaul, and finally with the advent of syslog-ng 3.0, this was done.

I'm not sure all of you know, but even earlier syslog-ng versions (2.1 and 2.0) did collect some per-source and per-destination statistics. These were reported periodically in the system log. The problem with this approach that it didn't really scale: with a large configuration the statistics message could become kilobytes long, and parsing this information from a file possibly several gigabytes in size is daunting.

syslog-ng 3.0 has two important changes in this area: it adds several new kinds of counters (like per-host counters), and a UNIX domain socket where you can query the current status of these counters.

As counters certainly have an overhead, you can now control how much statistics you want to gather. The new stats_level() option has three levels for now:

stats_level(0) is basically the same as earlier syslog-ng versions, per-source and per-destination statistics are kept here. This is the default.
stats_level(1) adds new counters without a big overhead, that is it adds counters for TCP connections, but does not keep per-host counters
stats_level(2) adds counters that can have a measurable performance impact, it adds for example per-host (as in $HOST) counters and also keeps track of the time the last message was received from a given host. These counters usually require an hash table lookup in the fastpath.

Once you have the counters, you can still use the venerable "log statistics" message, by setting the stats_freq() option which defaults to 10 minutes, just like in earlier versions.

However if you don't want to dig the logs produced by syslog-ng, you can also use the new UNIX domain socket at /var/run/syslog-ng/syslog-ng.ctl (the path might depend on the compilation options).

If you connect to this socket using netcat (some netcat versions do support UNIX domain sockets), and you send a "STATS" command to it, you get the list of counters.

There are no proper, command line clients for the UNIX domain channel yet, but if you have some scripting ability, you can start gather statistics easily, without the hassles of parsing log files, right after installing a syslog-ng 3.0 snapshot. :)

syslog-ng message parsing

2008-10-30T11:45:00.006+01:00

Earlier this month, I announced the new syslog-ng 3.0 git tree, adding a lot of new features to syslog-ng Open Source Edition. I thought it'd be useful to describe the new features with some more details, so this time I'd write about message parsing.

First of all, the message structure was a bit generalized in syslog-ng. Earlier it was encapsulating a syslog message and had little space to anything beyond that. That is, every log message that syslog-ng handled had date, host, program and message fields, but syslog-ng didn't care about message contents.

This has changed, a LogMessage became a set of name-value pairs, with some "built-in" pairs that correspond to the parts of a syslog message.

The aim with this change is: new name-value pairs can be associated with messages through the use of a parsing. It is now possible to parse non-syslog logs and use the columns the same way you could do it with syslog fields. Use them in the name of files, SQL tables or columns in an SQL table.

Here is an example:


parser p_parse_apache_logs { ... };

destination d_peruser { 
  file("/var/log/apache/${APACHE.USER_NAME}.log"); 
};

log { 
  source(s_local); 
  parser(p_parse_apache_logs); 
  destination(d_peruser_files); 
};

This means that you can "extract" information from the payload and use this information for naming destination files or SQL tables, basically anywhere where you can use a template.

There are currently two parsers implemented in syslog-ng:

a generic CSV (comma separated-values) parser, which can be parameterized to basically accept any kind of formally formatted input (so tab/space separated is also ok)
a database based parser, which uses a log pattern database to recognize messages belonging to specific applications and extract information on that.

Since the database based parser is quite complex so it deserves its own post, I'd skip that for now. The CSV parser has the following options:

template: defines the input to be used for parsing, can use macros
columns: list of strings, the names to be associated with the columns parsed
delimiters: the set of characters that delimit columns
quotes or quote_pairs: the quote characters to support, quote_pairs makes it possible to use different start and end quote (like enclosing fields in braces)
null: the null value which if found should substituted with an empty string
flags: see the documentation

The csv parser is capable of parsing real CSV data, e.g. it knows about quoting rules. So if you have an application that logs into files using space or comma separated data, you can almost be sure that you can process it with CSV parser.

Here is an example that parses Apache logs, so that each field in the message becomes a name-value pair:


parser p_apache {
csv-parser(columns("APACHE.CLIENT_IP",
                "APACHE.IDENT_NAME",
                "APACHE.USER_NAME",
                "APACHE.TIMESTAMP",
                "APACHE.REQUEST_URL",
                "APACHE.REQUEST_STATUS",
                "APACHE.CONTENT_LENGTH",
                "APACHE.REFERER",
                "APACHE.USER_AGENT",
                "APACHE.PROCESS_TIME",
                "APACHE.SERVER_NAME")
               # flags:
               #   escape-none,escape-backslash,escape-double-char,
               #   strip-whitespace
               flags(escape-double-char,strip-whitespace)
               delimiters(" ")
               quote-pairs('""[]')
    );
};

parser p_apache_timestamp {
    csv-parser(columns("APACHE.TIMESTAMP.DAY",
                       "APACHE.TIMESTAMP.MONTH",
                       "APACHE.TIMESTAMP.YEAR",
                       "APACHE.TIMESTAMP.HOUR",
                       "APACHE.TIMESTAMP.MIN",
                       "APACHE.TIMESTAMP.MIN",
                       "APACHE.TIMESTAMP.ZONE")
               delimiters("/: ")
               flags(escape-none)
               template("${APACHE.TIMESTAMP}"));
};

The first parser splits the major fields, and the second splits the timestamp to manageable pieces. You can then bind this parser to a log path of your choosing:


log {
  source(s_apache);
  parser(p_apache); parser(p_apache_timestamp);
  destination(d_apache);
};

As you can see the second parser uses a value created by the previous parser, using its template() option. Once this parsing is done, you can use any of the values created this way
in your d_apache destination, be it the name of the file, or a column in an SQL table.

6th Netfilter workshop

2008-10-08T21:36:00.004+02:00

I've spent my last week in Paris, where this year's Netfilter Workshop was held. I wanted to take this opportunity to thank Eric of INL for the organization. It was a wonderful and useful event, and I enjoyed it a lot. It is always nice to meet these wonderful guys.

Here are some blog posts about the same event:

INL: http://nfws.inl.fr/en/
DaveM: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2008/10/05#nfws2008
Patrick McHardy: http://people.netfilter.org/kaber/weblog/

Finally we could get Transparent Proxying merged, now queued for 2.6.28.

syslog-ng OSE 3.0 git tree published

2008-10-01T16:03:00.002+02:00

I could finally get my syslog-ng 3.0 OSE tree published at git.balabit.hu. No nightly snapshots yet and I still have to prepare a formal announcement to post on the mailing list, but for those I teased with functions from the 3.0 branch, here it comes.

From the top of my head, OSE 3.0 supports:

TLS encrypted channels,
syslog message rewrite,
parse parts of the syslog message and use the parsed parts in macros
PCRE and glob filters (in addition to POSIX regexps),
support for the new IETF syslog protocols,
program sources,
new statistics framework that can be queried using UNIX domain sockets
etc.

I just wanted to get the word out. Success/failure reports would be appreciated.

Migrate over to PCRE?

2008-06-30T16:30:00.003+02:00

As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE.

For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent.

So the situation is as follows:

various filters use POSIX regexps
rewrite uses PCRE

This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run.

If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a forthcoming release I'd change the default regexp style to PCRE, and yet another syslog-ng release later, I'd phase out POSIX completely.

If the decision is to keep them both in the long run, it would mean that I'd need to implement POSIX style regexps for rewrite rules as well. This would probably the least intrusive for users, but also a lot more work. Also, this would allow adding other filtering options like globbing or prefix search.

What do you think? Is the addition of modular search algorithms worth it?

Please send your opinions to the mailing list: syslog-ng@lists.balabit.hu