Project

General

Profile

Bug #4393

syslogd stops and fails to restart during boot in some cases

Added by Adam Esslinger over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Logging
Target version:
Start date:
02/08/2015
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.2
Affected Architecture:
amd64

Description

Sometimes after rebooting syslogd shows that it has crashed. Rebooting usually fixes the issue. Here is what I see in the system logs.

Feb 8 20:46:25 syslogd: exiting on signal 15
Feb 8 20:46:25 kernel: done.

I have pfsense to only log locally.

Capture5.JPG (94.7 KB) Capture5.JPG Adam Esslinger, 02/08/2015 08:00 PM
Capture2.PNG (5.2 KB) Capture2.PNG Albert Yang, 04/17/2016 11:07 PM

Associated revisions

Revision 209ba3aa (diff)
Added by Chris Buechler over 4 years ago

Wait a bit after sending a TERM to syslogd as in some instances it can take too long to stop, and it fails to restart because it's still running at that point. Add a KILL in case it's still running after that. Ticket #4393

Revision cfda8861 (diff)
Added by Chris Buechler over 4 years ago

Wait a bit after sending a TERM to syslogd as in some instances it can take too long to stop, and it fails to restart because it's still running at that point. Add a KILL in case it's still running after that. Ticket #4393

History

#1 Updated by Jim Pingle over 4 years ago

This typically happens when you have a corrupted log file. The first attempted write to said log file will crash syslogd. Reset the log files from Status > System Logs, Settings tab.

That said, we probably need to come up with a way to detect broken/corrupted log files, assuming your problem is from that same cause.

#2 Updated by Adam Esslinger over 4 years ago

It looks like this fixed my issue. I had the log size set to 8192 thinking that it was in KB and realized that the field is in bytes so I was setting it to 8K when the default shows 500KB. Im guessing that it was corrupting because the log file was to small and unable to rollover right after boot is complete.

#3 Updated by Jim Pingle over 4 years ago

  • Status changed from New to Feedback

I set the log file size low and ran some tests and still couldn't make it crash.

If someone else finds syslogd crashing similar, please tar up the contents of /var/log/ before resetting the logs. If resetting the logs fixes the problem again, make arrangements to get the archive/backup of the logs to us, either attached on the ticket or privately.

#4 Updated by Adam Esslinger over 4 years ago

I just had this happen again after rebooting my box from Bug #4392. How can I email you the logs?

#5 Updated by Jim Pingle over 4 years ago

If you have made an archive of the logs you can post them on a site like Google Drive or Dropbox and PM me the URL on the forum or send it to me directly, though it may be too large to e-mail. Address is jimp (at) pfsense [dot] org.

#6 Updated by Adam Esslinger over 4 years ago

I sent you an email yesterday with the logs (9MB). Did you get the email?

#7 Updated by Jim Pingle over 4 years ago

I received it, but it may be Tuesday before I have a chance to look at the logs.

#8 Updated by Chris Buechler over 4 years ago

  • Category set to Logging
  • Status changed from Feedback to Confirmed
  • Priority changed from Normal to High
  • Target version set to 2.2.1

I added a tgz of /var/log from a system exhibiting the problem to projects/ticket-files/

#9 Updated by Jim Pingle over 4 years ago

I still can't seem to reproduce it here even with a "bad" set of logs and similar settings. There must be some other contributing factor at work.

I'm curious if lighttpd will produce any errors if/when syslogd breaks when saving the syslog settings. To enable some extra logging in lighttpd, apply this patch:

http://files.atx.pfsense.org/jimp/patches/lighttpd-breakage-log.diff (don't apply that now, see Chris' later message)

And then from the console, use option 11 to restart lighttpd. Try clicking Save on Status > System Logs, Settings tab and then if syslogd is not running, check /var/log/lighttpd-breakage.log to see if anything shows up there.

#10 Updated by Chris Buechler over 4 years ago

  • Status changed from Confirmed to Feedback

every circumstance I could replicate is fixed by what I just pushed.

#11 Updated by Adam Esslinger over 4 years ago

I applied the patch and it created the lighttpd-breakage.log but its 0bytes. After applying the patch and restarting ligttpd here is what was logged in system log.

Feb 18 08:34:35 syslogd: kernel boot file is /boot/kernel/kernel
Feb 18 08:34:35 syslogd: exiting on signal 15
Feb 18 08:34:35 check_reload_status: Syncing firewall
Feb 18 08:33:42 php: rc.restart_webgui: Creating rrd update script
Feb 18 08:33:29 sshd12449: Accepted keyboard-interactive/pam for admin from xx:xx:xx:xx port 55839 ssh2
Feb 18 08:32:18 check_reload_status: Syncing firewall
Feb 18 08:31:20 check_reload_status: Syncing firewall
Feb 18 08:29:48 check_reload_status: Syncing firewall
Feb 18 08:28:34 check_reload_status: Reloading filter
Feb 18 08:28:34 php-fpm18732: /pkg_mgr_install.php: Successfully installed package: System Patches.
Feb 18 08:28:34 check_reload_status: Syncing firewall
Feb 18 08:28:34 check_reload_status: Syncing firewall
Feb 18 08:28:32 check_reload_status: Syncing firewall
Feb 18 08:28:31 php-fpm18732: /pkg_mgr_install.php: Beginning package installation for System Patches .

I havent seen this bug crop up on my other 2 systems so I'm beginning to wonder if its just something very specific to this machine. My other 2 systems were on 2.1.5 had inplace upgrades to 2.2-RELEASE. The system that is having this issue was a new install on 2.2RC (not sure what daily snapshot) and upgraded through the daily snapshots to 2.2-RELEASE. I thinking I should try rebuilding this machine from the 2.2-RELEASE and see if that solves the issue. All 3 systems have the same base hardware, however this box also has a Broadcom accelerator card (cryptodev) and and Atheros AR9380 wireless card (AP mode).

#12 Updated by Jim Pingle over 4 years ago

No need for that patch I posted now, Chris found the actual fix after that. The fix is in the commits shown in the "Associated revisions" section here on the ticket.

#13 Updated by Chris Buechler over 4 years ago

  • Subject changed from syslogd crashing after reboot to syslogd stops and fails to restart during boot in some cases

anyone who's having issues with syslogd not running, install the System Patches package, then browse to System>Patches, and add a new patch. Give it a description as desired, put in commit ID cfda8861b93ef0a2daf2cc9ca2424200e45dd84f, and leave all else at defaults. Save. Then hit Fetch, and Apply. This fix will then be in place.

#14 Updated by Adam Esslinger over 4 years ago

I have applied the patch. Ill let you know if I see any more syslog crashes. Thanks!

#15 Updated by Chris Buechler over 4 years ago

  • Status changed from Feedback to Resolved

confirmed fixed multiple times.

#16 Updated by Albert Yang over 3 years ago

Im not sure if it was fixed for 2.2.4? But seems to effect it also? Was trying to send syslogs to logstash seems to be not working :(

pfSense 2.2.4 x64bit

#17 Updated by Chris Buechler over 3 years ago

Albert Yang wrote:

Im not sure if it was fixed for 2.2.4? But seems to effect it also?

It does not, this has been fixed since 2.2.1. Your log clearly shows it's starting again. Post to the forum with more info on what you're trying to do.

#18 Updated by Albert Yang over 3 years ago

Hi Chris Thank you for the reply when you say it clearly shows its starting again but why the syslogd: exiting on signal 15 im running ELK which pfSense sends all the logs and for some reason wont work i posted on the forums but no luck so far :(

https://forum.pfsense.org/index.php?topic=110234.msg613717#msg613717

Thank you

Also available in: Atom PDF