Project

General

Profile

Actions

Todo #8350

closed

Remove clog in favor of standard syslogd or syslogd alternative with rotation via newsyslog or logrotate

Added by Jim Pingle over 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Logging
Target version:
Start date:
08/30/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Plus Target Version:
Release Notes:

Description

Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd

We should remove our dependence on clog and use plain text log files which can be rotated and archived and still maintain a small disk footprint, while not being strictly/exactly limited like clog.

Important points:

  • User log reading/searching must look back in archives for recent messages since rotating will remove them from the current active log file.
  • Log size/rotation schedule/number of past archived log files should be configurable, perhaps more. Log size is already there from clog settings, but for example newsyslog has several options for configuring when rotation happens and what it does at that time (e.g. compress with bzip)
  • Alternatives to syslog (e.g. syslog-ng, rsyslog) could be considered provided there is no loss of functionality
  • Functions dealing with clog files will need replaced/rewritten/removed -- probably not a significant effort since the option already exists today to use plain text logging, may only be a matter of removing dead code

Subtasks 7 (0 open7 closed)

Todo #9711: Add GUI options to control log rotationResolvedJim Pingle08/30/2019

Actions
Todo #9712: Add code for packages to set their own log rotation parametersResolvedJim Pingle08/30/2019

Actions
Todo #9713: Review log rotation behaviorResolvedJim Pingle08/30/2019

Actions
Todo #9714: Add page to view "other" logsResolvedJim Pingle08/30/2019

Actions
Bug #9715: Call to undefined function sort_related_log_filesResolvedJim Pingle08/31/2019

Actions
Bug #9730: newsyslog cron job not present after every upgradeResolvedJim Pingle09/06/2019

Actions
Todo #9734: Re-evaluate log size, line defaults, and limitsResolvedJim Pingle09/08/2019

Actions
Actions #1

Updated by Darren Spruell over 3 years ago

+1 on this - clog is kind of neat for the use case it addresses, but is fairly inconvenient in terms of modern log analysis/log delivery. For example, clog is not well known and well-documented so there are relatively few references for it (pfSense being one of the best sources today). There is also a complication when considering remote log delivery; syslog works, but it suffers from being syslog. Modern log collection agents like Filebeat and Fluent Bit are used in increasingly more environments today and would benefit from having plaintext, rotated system logs to read from.

Offtopic - It would be good to see this change followed by creation/maintenance of Fluent Bit and Filebeat packages for pfSense to facilitate evolution of log delivery.

Actions #2

Updated by Jim Pingle about 3 years ago

  • Target version changed from 2.4.4 to 48
Actions #3

Updated by Jim Pingle over 2 years ago

  • Target version changed from 48 to 2.5.0
Actions #4

Updated by Jim Pingle about 2 years ago

As a part of this, make sure to check other logs that were not displayed in the GUI before, such as the nginx logs, to help with issues such as #7198

Actions #5

Updated by Jim Pingle about 2 years ago

  • Status changed from New to In Progress
Actions #6

Updated by Jim Pingle about 2 years ago

  • % Done changed from 0 to 70

Most of this is done but there are a few remaining issues:

  • Needs wider testing (obviously)
  • Log rotation may need further adjustments since daemons may need kicked in various ways. Only basics are present now.
  • Wants GUI settings to adjust rotation size vs time requirements (can be either, or both), disable compression or change compression type, PIDs to kick or actions to take, etc.
  • Wants code to allow packages to adjust rotation needs similar to above.
Actions #7

Updated by Jim Pingle about 2 years ago

I split some of those into their own separate issues:

  • #9711: Add GUI options to control log rotation
  • #9712: Add code for packages to set their own log rotation parameters
  • #9713: Review log rotation behavior
Actions #8

Updated by Jim Pingle about 2 years ago

  • Status changed from In Progress to Feedback

This should be ready for general feedback once the latest changes are in snapshots.

Actions #9

Updated by Matt Gilchrist about 2 years ago

This stops MailReports package from working
(not a big deal - just letting you know)

This is a periodic report from your firewall, pfSense.example.com.
Current report: Daily Report
Log output: System (system.log)
Cannot locate clog which is required for reading log files.
Log output: DHCP (dhcpd.log)
Cannot locate clog which is required for reading log files.

Actions #10

Updated by Jim Pingle about 2 years ago

I moved that to its own issue, since it isn't relevant to the base system: #9787

Actions #11

Updated by Mark Rodman about 2 years ago

Testing 2.5.0-dev, found UI does not provide support for TCP syslog forwarding. I believe this results in truncation of large log lines. Example is suricata eve json format. Receiving syslog server only receives partial message.

Can this be addressed please.

Actions #12

Updated by Jim Pingle about 2 years ago

That is not related to this change, so it cannot be considered on this issue. TCP syslog is not yet supported by FreeBSD syslogd, and even if it were, it's not relevant to the clog/plain text switch.

You can use the syslog-ng package.

Actions #13

Updated by Jim Pingle almost 2 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF