Project

General

Profile

Bug #8229

syslog-ng stops parsing logs after logrotate run

Added by Orion Poplawski over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
syslog-ng
Target version:
-
Start date:
12/20/2017
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.2
Affected Architecture:

Description

I'm having trouble on one of my three pfsense boxes with syslog-ng stopping processing logfiles after the logrotate run. I have it configured to monitor three suricate eve.json file and forward the output to a remote syslog server. But after the logrotate run no more messages are sent. fsfat shows that it still has the eve.json files open, but truss just has it spinning:

_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x80301f080,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0)

If I restart syslog-ng it works again.

Nothing in /var/syslog-ng/default.log, last message is just a statistics message:

Dec 15 23:45:14 inferno syslog-ng[89458]: Log statistics; processed='destination(d_tcp)=632984', processed='center(received)=633084', processed='source(eve_pub)=725', processed='center(queued)=633084', queued='global(scratch_buffers_count)=0', processed='src.none()=0', stamp='src.none()=0', processed='global(payload_reallocs)=164', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=100', dropped='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=0', processed='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=632984', queued='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=0', written='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=632977', processed='src.internal(_DEFAULT#0)=100', stamp='src.internal(_DEFAULT#0)=1513406412', processed='source(eve_lan)=241383', queued='global(scratch_buffers_bytes)=0', processed='source(eve_dmz)=390876', processed='destination(_DEFAULT)=100', processed='global(internal_queue_length)=0', processed='global(msg_clones)=0'
Dec 15 23:55:14 inferno syslog-ng[89458]: Log statistics; processed='destination(d_tcp)=636239', processed='center(received)=636340', processed='source(eve_pub)=725', processed='center(queued)=636340', queued='global(scratch_buffers_count)=0', processed='src.none()=0', stamp='src.none()=0', processed='global(payload_reallocs)=166', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=101', dropped='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=0', processed='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=636239', queued='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=0', written='dst.network(d_tcp#0,tcp,loghost.cora.nwra.com:514)=636232', processed='src.internal(_DEFAULT#0)=101', stamp='src.internal(_DEFAULT#0)=1513406714', processed='source(eve_lan)=241383', queued='global(scratch_buffers_bytes)=0', processed='source(eve_dmz)=394131', processed='destination(_DEFAULT)=101', processed='global(internal_queue_length)=0', processed='global(msg_clones)=0'

syslog-ng.conf:

# This file is automatically generated by pfSense
# Do not edit manually !
@version:3.11
destination d_tcp { network("hostname" port(514)); };
destination _DEFAULT { file("/var/syslog-ng/default.log"); };
log { source(eve_lan); source(eve_dmz); source(eve_pub); destination(d_tcp); };
log { source(_DEFAULT); destination(_DEFAULT); };
options { log-msg-size(16384); };
source eve_pub { file("/var/log/suricata/suricata_igb355293/eve.json" default-facility(local1) flags(no-parse) program-override("suricata")); };
source eve_lan { file("/var/log/suricata/suricata_igb018282/eve.json" default-facility(local1) flags(no-parse) program-override("suricata")); };
source eve_dmz { file("/var/log/suricata/suricata_igb264462/eve.json" default-facility(local1) flags(no-parse) program-override("suricata")); };
source _DEFAULT { internal(); syslog(transport(udp) port(5140) ip(127.0.0.1)); };

syslog-ng sysutils 1.14
syslog-ng-3.11.1_1
logrotate-3.9.2
pfsense 2.4.2-RELEASE

History

#1 Updated by Orion Poplawski over 1 year ago

I see that syslog-ng 3.13.2 has been released. Perhaps an update is in order.

#2 Updated by Orion Poplawski over 1 year ago

After switching to use tls for forwarded log traffic this seems even worse. It requires several attempts to restart the server via the web UI to get the server running again.

#3 Updated by Orion Poplawski over 1 year ago

Well, tried syslog-ng-3.13.2_1 from http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/ but that went crazy after a couple hours and consumed all cpu. Now giving http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/All/syslog-ng312-3.12.1_1.txz a try.

Also available in: Atom PDF