Feature #7926
closed
limit clog -f look-back size
Added by Adam Thompson over 6 years ago.
Updated over 4 years ago.
Description
I've configured the system log files to be substantially larger than normal, in order to get some reasonable retention period on VPN logs, but this creates an unexpected problem:
- running "clog -f /var/log/system.log" takes about 5-10 minutes (on an SG-2440) to catch up to "now" and start tail'ing the log. (Part of that is the SSH pipe, but certainly not all!)
I can't even tell where clog itself is maintained, so this might not be possible, but it would be awesome if clog supported a mechanism for NOT starting all the way back at the beginning of the file. (This would be useful in both "-f" and regular modes, for the same reasons.)
FWIW, I'm thinking of "tail -f"'s behaviour, where it only tail's the last ~10 lines (I think most implementations default to that, anyway). And some implementations let me specify not only "tail -1000" but also "tail -1000 -f".
Either way, for "clog -f" just starting at the current ring position would be helpful.
Some data here:
[2.3.4-RELEASE][root@remote.avant.ca]/root: time sh -c "clog /var/log/system.log | wc -l"
823199
1.490u 0.032s 0:01.50 101.3% 9+167k 0+0io 0pf+0w
the data, unsurprisingly, shows that pushing all the data through a tty and then on through ssh is the biggest problem. SSH compression helps somewhat, but doesn't address the underlying issue.
The way clog reads the records it has to figure out where the start is and then unwind it from there, so it doesn't exactly know how far from the end it is, which complicates this sort of issue. It may be easier to give it a starting offset so it skips X amount from the start of the file but I'm still not sure it's worth it.
clog is doomed anyhow. It's getting harder and harder to keep it and its required patches maintained. We're investigating dropping it entirely in favor of plain text logs + newsyslog or similar in the very near future.
Thanks, Jim. That would be a perfectly acceptable solution, with a whole bunch of side benefits.
Especially since I don't know of a single pfSense firewall still running on write-endurance-limited CF cards anyway. (Wal-mart sells USB sticks that have more write endurance than those old CF disks!)
You'd be surprised, there are a number of them out there on CF, USB sticks and the like, and some of them have opted to have logs in RAM disks which are limited in size. clog was good for that, but that situation is definitely in the minority these days. Good log rotation will keep things in check and open up a lot more possibilities.
- Status changed from New to Closed
Issue will be moot once #8350 is implemented.
Also available in: Atom
PDF