Todo #8350
closed
Remove clog in favor of standard syslogd or syslogd alternative with rotation via newsyslog or logrotate
100%
Description
Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd
We should remove our dependence on clog and use plain text log files which can be rotated and archived and still maintain a small disk footprint, while not being strictly/exactly limited like clog.
Important points:
- User log reading/searching must look back in archives for recent messages since rotating will remove them from the current active log file.
- Log size/rotation schedule/number of past archived log files should be configurable, perhaps more. Log size is already there from clog settings, but for example newsyslog has several options for configuring when rotation happens and what it does at that time (e.g. compress with bzip)
- Alternatives to syslog (e.g. syslog-ng, rsyslog) could be considered provided there is no loss of functionality
- Functions dealing with clog files will need replaced/rewritten/removed -- probably not a significant effort since the option already exists today to use plain text logging, may only be a matter of removing dead code
Updated by Darren Spruell almost 6 years ago
+1 on this - clog is kind of neat for the use case it addresses, but is fairly inconvenient in terms of modern log analysis/log delivery. For example, clog is not well known and well-documented so there are relatively few references for it (pfSense being one of the best sources today). There is also a complication when considering remote log delivery; syslog works, but it suffers from being syslog. Modern log collection agents like Filebeat and Fluent Bit are used in increasingly more environments today and would benefit from having plaintext, rotated system logs to read from.
Offtopic - It would be good to see this change followed by creation/maintenance of Fluent Bit and Filebeat packages for pfSense to facilitate evolution of log delivery.
Updated by Jim Pingle over 4 years ago
As a part of this, make sure to check other logs that were not displayed in the GUI before, such as the nginx logs, to help with issues such as #7198
Updated by Jim Pingle over 4 years ago
- % Done changed from 0 to 70
Most of this is done but there are a few remaining issues:
- Needs wider testing (obviously)
- Log rotation may need further adjustments since daemons may need kicked in various ways. Only basics are present now.
- Wants GUI settings to adjust rotation size vs time requirements (can be either, or both), disable compression or change compression type, PIDs to kick or actions to take, etc.
- Wants code to allow packages to adjust rotation needs similar to above.
Updated by Jim Pingle over 4 years ago
Updated by Jim Pingle over 4 years ago
- Status changed from In Progress to Feedback
This should be ready for general feedback once the latest changes are in snapshots.
Updated by Matt Gilchrist over 4 years ago
This stops MailReports package from working
(not a big deal - just letting you know)
This is a periodic report from your firewall, pfSense.example.com.
Current report: Daily Report
Log output: System (system.log)
Cannot locate clog which is required for reading log files.
Log output: DHCP (dhcpd.log)
Cannot locate clog which is required for reading log files.
Updated by Jim Pingle over 4 years ago
I moved that to its own issue, since it isn't relevant to the base system: #9787
Updated by Mark Rodman over 4 years ago
Testing 2.5.0-dev, found UI does not provide support for TCP syslog forwarding. I believe this results in truncation of large log lines. Example is suricata eve json format. Receiving syslog server only receives partial message.
Can this be addressed please.
Updated by Jim Pingle over 4 years ago
That is not related to this change, so it cannot be considered on this issue. TCP syslog is not yet supported by FreeBSD syslogd, and even if it were, it's not relevant to the clog/plain text switch.
You can use the syslog-ng package.
Updated by Jim Pingle over 4 years ago
- Status changed from Feedback to Resolved