Project

General

Profile

Todo #8350

Remove clog in favor of standard syslogd or syslogd alternative with rotation via newsyslog or logrotate

Added by Jim Pingle over 1 year ago. Updated 20 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
Category:
Logging
Target version:
Start date:
08/30/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd

We should remove our dependence on clog and use plain text log files which can be rotated and archived and still maintain a small disk footprint, while not being strictly/exactly limited like clog.

Important points:

  • User log reading/searching must look back in archives for recent messages since rotating will remove them from the current active log file.
  • Log size/rotation schedule/number of past archived log files should be configurable, perhaps more. Log size is already there from clog settings, but for example newsyslog has several options for configuring when rotation happens and what it does at that time (e.g. compress with bzip)
  • Alternatives to syslog (e.g. syslog-ng, rsyslog) could be considered provided there is no loss of functionality
  • Functions dealing with clog files will need replaced/rewritten/removed -- probably not a significant effort since the option already exists today to use plain text logging, may only be a matter of removing dead code

Subtasks

Todo #9711: Add GUI options to control log rotationFeedbackJim Pingle

Todo #9712: Add code for packages to set their own log rotation parametersFeedbackJim Pingle

Todo #9713: Review log rotation behaviorResolvedJim Pingle

Todo #9714: Add page to view "other" logsFeedbackJim Pingle

Bug #9715: Call to undefined function sort_related_log_filesFeedbackJim Pingle

Bug #9730: newsyslog cron job not present after every upgradeFeedbackJim Pingle

Todo #9734: Re-evaluate log size, line defaults, and limitsFeedbackJim Pingle

Associated revisions

Revision f9e8c833 (diff)
Added by Jim Pingle about 2 months ago

Change logging to plain text, deprecate clog. Issue #8350

Revision 3a26e715 (diff)
Added by Jim Pingle about 1 month ago

Move log-related functions to their own file. Issue #8350

Also add a simple shell program that will dump all log entries for a given
log + all rotated/compressed logs in order.

Revision 24b1410a (diff)
Added by Jim Pingle about 1 month ago

Don't add .log to filename twice. Issue #8350

Revision aa2cac18 (diff)
Added by Jim Pingle about 1 month ago

Incorporate filter_log.inc into syslog.inc. Issue #8350

Now all log-related functions are together.

Revision c63ae216 (diff)
Added by Jim Pingle about 1 month ago

Fix rotation count check. Issue #9711

While here, fixup default static newsyslog entries. Issue #8350

History

#1 Updated by Darren Spruell over 1 year ago

+1 on this - clog is kind of neat for the use case it addresses, but is fairly inconvenient in terms of modern log analysis/log delivery. For example, clog is not well known and well-documented so there are relatively few references for it (pfSense being one of the best sources today). There is also a complication when considering remote log delivery; syslog works, but it suffers from being syslog. Modern log collection agents like Filebeat and Fluent Bit are used in increasingly more environments today and would benefit from having plaintext, rotated system logs to read from.

Offtopic - It would be good to see this change followed by creation/maintenance of Fluent Bit and Filebeat packages for pfSense to facilitate evolution of log delivery.

#2 Updated by Jim Pingle about 1 year ago

  • Target version changed from 2.4.4 to 48

#3 Updated by Jim Pingle 7 months ago

  • Target version changed from 48 to 2.5.0

#4 Updated by Jim Pingle about 2 months ago

As a part of this, make sure to check other logs that were not displayed in the GUI before, such as the nginx logs, to help with issues such as #7198

#5 Updated by Jim Pingle about 2 months ago

  • Status changed from New to In Progress

#6 Updated by Jim Pingle about 2 months ago

  • % Done changed from 0 to 70

Most of this is done but there are a few remaining issues:

  • Needs wider testing (obviously)
  • Log rotation may need further adjustments since daemons may need kicked in various ways. Only basics are present now.
  • Wants GUI settings to adjust rotation size vs time requirements (can be either, or both), disable compression or change compression type, PIDs to kick or actions to take, etc.
  • Wants code to allow packages to adjust rotation needs similar to above.

#7 Updated by Jim Pingle about 2 months ago

I split some of those into their own separate issues:

  • #9711: Add GUI options to control log rotation
  • #9712: Add code for packages to set their own log rotation parameters
  • #9713: Review log rotation behavior

#8 Updated by Jim Pingle about 1 month ago

  • Status changed from In Progress to Feedback

This should be ready for general feedback once the latest changes are in snapshots.

#9 Updated by Matt Gilchrist 21 days ago

This stops MailReports package from working
(not a big deal - just letting you know)

This is a periodic report from your firewall, pfSense.example.com.
Current report: Daily Report
Log output: System (system.log)
Cannot locate clog which is required for reading log files.
Log output: DHCP (dhcpd.log)
Cannot locate clog which is required for reading log files.

#10 Updated by Jim Pingle 20 days ago

I moved that to its own issue, since it isn't relevant to the base system: #9787

Also available in: Atom PDF