pfSense

Probably some daemon should be done that reads this and does not complain much.
Does not seem hard to merge the tcpdump print code into pflogd and remove this issue and the performance problems you get from it.

Mostly 2-3 hours of work for someone wanting to put in place this.
It would help also with pfSense parsing since it can be customized to better fit into pfSense.

I don't know that I would call returning to the original functionality a "feature", this is a "bug" that broke the syslog feature. I don't know that I would call this the default logging format, when it comes to syslog there is an implicit one event = one message, and this change in v2 NOW violates that. I would call it tcpdump's NEW default output format screwing up syslog messages because the two are now incompatible. tcpdump NOW writes a single multi-line message and that's fine, but, syslog is incapable of handling it as a single mulit-line message and breaks all the line breaks into separate syslog messages. Maybe its more appropriate to escape the characters that syslog can't handle so that the tcpdump message at least has a placeholder, but the current implementation is broken. With this NEW behavior it is pointless to do syslog logging at all... the number of syslog messages are now more than double and its not possible to reassemble the out-of-sequence fragments. This is actually a breaking change from v1. More specifically the DShield firewall log parser was broken by this change in v2 and I don't see a way to fix it without first fixing the BUG in pfSense syslog output, which I've already done locally just to get the broken bits functioning again. Anyone that does syslog parsing and processing will have similar concerns with this BUG.

A warning to those trying the proposed change, it doesn't quite work as written. It works if you run it from the command line but not when run from PHP. Sed needs the \n to be \n, but it's getting escaped somewhere along the way to \\n and breaks.

Wow, that's troubling. I escaped it because it looked like PHP was swallowing the backslash. But I'm looking now and see it made it through to the process:

PID  TT  STAT      TIME COMMAND
24058  v0- I      0:08.52  /usr/bin/sed -e N;s/\\n //;P;D;

and yet its been working fine here since I wrote this article. Wish I knew why we're getting different results. I need to look a little closer at it.

I'm having this exact problem with Dshield log parsing and I'm also having Jim P's issue with getting the sed line to work.

Havis tried the original, I then tried changing quotes from " to ' and thus doing

mwexec_bg('/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | /usr/bin/sed -e \'N;s/\n //;P;D;\' | logger -t pf -p local0.info')

 /usr/bin/sed -e 'N;s/\\n //;P;D;'

This is also a major irritation as it utterly borks log parsing/analysis without manually putting each line back together in the parse code. And that is not exactly simple.

Correction. The ' version does seem to work even though ps ax shows the \\n. It seems that the other version did too but I was too fast to assume that it wasn't logging anything.

If you save the following code as a php script (e.g. chgfilter.php) and scp it to the pfsense home directory (/root/) you can create a new and a backup filter.inc in that directory.

$filter=file_get_contents ('/etc/inc/filter.inc');
$filternew =
    str_replace(
        "-ttt -i pflog0 | logger -t pf -p local0.info",
        "-ttt -i pflog0 | /usr/bin/sed -e 'N;s/\\\\n //;P;D;' | logger -t pf -p local0.info",
        $filter);
if (strcmp($filter, $filternew) !=0) {
    file_put_contents('filter.inc.new',$filternew);
    file_put_contents('filter.inc.org',$filter);
}

php chgfilter.php

mv filter.inc.new /etc/inc/filter.inc

Note the script only makes a change if the replacement is successful so this will not hurt anything if run multiple times. Also if it goes wrong you can copy the original back by

mv filter.inc.org /etc/inc/filter.inc

sed should get the -l switch to make output line buffered. Otherwise the filter log will only get updates when several log lines are pending, instead of in real time.