Todo #8350

Remove clog in favor of standard syslogd or syslogd alternative with rotation via newsyslog or logrotate

Added by Jim Pingle over 2 years ago. Updated 10 months ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)


Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd

We should remove our dependence on clog and use plain text log files which can be rotated and archived and still maintain a small disk footprint, while not being strictly/exactly limited like clog.

Important points:

  • User log reading/searching must look back in archives for recent messages since rotating will remove them from the current active log file.
  • Log size/rotation schedule/number of past archived log files should be configurable, perhaps more. Log size is already there from clog settings, but for example newsyslog has several options for configuring when rotation happens and what it does at that time (e.g. compress with bzip)
  • Alternatives to syslog (e.g. syslog-ng, rsyslog) could be considered provided there is no loss of functionality
  • Functions dealing with clog files will need replaced/rewritten/removed -- probably not a significant effort since the option already exists today to use plain text logging, may only be a matter of removing dead code


Todo #9711: Add GUI options to control log rotationResolvedJim Pingle

Todo #9712: Add code for packages to set their own log rotation parametersResolvedJim Pingle

Todo #9713: Review log rotation behaviorResolvedJim Pingle

Todo #9714: Add page to view "other" logsResolvedJim Pingle

Bug #9715: Call to undefined function sort_related_log_filesResolvedJim Pingle

Bug #9730: newsyslog cron job not present after every upgradeResolvedJim Pingle

Todo #9734: Re-evaluate log size, line defaults, and limitsResolvedJim Pingle

Associated revisions

Revision f9e8c833 (diff)
Added by Jim Pingle about 1 year ago

Change logging to plain text, deprecate clog. Issue #8350

Revision 3a26e715 (diff)
Added by Jim Pingle about 1 year ago

Move log-related functions to their own file. Issue #8350

Also add a simple shell program that will dump all log entries for a given
log + all rotated/compressed logs in order.

Revision 24b1410a (diff)
Added by Jim Pingle about 1 year ago

Don't add .log to filename twice. Issue #8350

Revision aa2cac18 (diff)
Added by Jim Pingle about 1 year ago

Incorporate into Issue #8350

Now all log-related functions are together.

Revision c63ae216 (diff)
Added by Jim Pingle about 1 year ago

Fix rotation count check. Issue #9711

While here, fixup default static newsyslog entries. Issue #8350


#1 Updated by Darren Spruell over 2 years ago

+1 on this - clog is kind of neat for the use case it addresses, but is fairly inconvenient in terms of modern log analysis/log delivery. For example, clog is not well known and well-documented so there are relatively few references for it (pfSense being one of the best sources today). There is also a complication when considering remote log delivery; syslog works, but it suffers from being syslog. Modern log collection agents like Filebeat and Fluent Bit are used in increasingly more environments today and would benefit from having plaintext, rotated system logs to read from.

Offtopic - It would be good to see this change followed by creation/maintenance of Fluent Bit and Filebeat packages for pfSense to facilitate evolution of log delivery.

#2 Updated by Jim Pingle about 2 years ago

  • Target version changed from 2.4.4 to 48

#3 Updated by Jim Pingle over 1 year ago

  • Target version changed from 48 to 2.5.0

#4 Updated by Jim Pingle about 1 year ago

As a part of this, make sure to check other logs that were not displayed in the GUI before, such as the nginx logs, to help with issues such as #7198

#5 Updated by Jim Pingle about 1 year ago

  • Status changed from New to In Progress

#6 Updated by Jim Pingle about 1 year ago

  • % Done changed from 0 to 70

Most of this is done but there are a few remaining issues:

  • Needs wider testing (obviously)
  • Log rotation may need further adjustments since daemons may need kicked in various ways. Only basics are present now.
  • Wants GUI settings to adjust rotation size vs time requirements (can be either, or both), disable compression or change compression type, PIDs to kick or actions to take, etc.
  • Wants code to allow packages to adjust rotation needs similar to above.

#7 Updated by Jim Pingle about 1 year ago

I split some of those into their own separate issues:

  • #9711: Add GUI options to control log rotation
  • #9712: Add code for packages to set their own log rotation parameters
  • #9713: Review log rotation behavior

#8 Updated by Jim Pingle about 1 year ago

  • Status changed from In Progress to Feedback

This should be ready for general feedback once the latest changes are in snapshots.

#9 Updated by Matt Gilchrist about 1 year ago

This stops MailReports package from working
(not a big deal - just letting you know)

This is a periodic report from your firewall,
Current report: Daily Report
Log output: System (system.log)
Cannot locate clog which is required for reading log files.
Log output: DHCP (dhcpd.log)
Cannot locate clog which is required for reading log files.

#10 Updated by Jim Pingle about 1 year ago

I moved that to its own issue, since it isn't relevant to the base system: #9787

#11 Updated by Mark Rodman about 1 year ago

Testing 2.5.0-dev, found UI does not provide support for TCP syslog forwarding. I believe this results in truncation of large log lines. Example is suricata eve json format. Receiving syslog server only receives partial message.

Can this be addressed please.

#12 Updated by Jim Pingle about 1 year ago

That is not related to this change, so it cannot be considered on this issue. TCP syslog is not yet supported by FreeBSD syslogd, and even if it were, it's not relevant to the clog/plain text switch.

You can use the syslog-ng package.

#13 Updated by Jim Pingle 10 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF