Project

General

Profile

Actions

Feature #7926

closed

limit clog -f look-back size

Added by Adam Thompson over 6 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Very Low
Assignee:
-
Category:
Logging
Target version:
-
Start date:
10/11/2017
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:

Description

I've configured the system log files to be substantially larger than normal, in order to get some reasonable retention period on VPN logs, but this creates an unexpected problem:
  • running "clog -f /var/log/system.log" takes about 5-10 minutes (on an SG-2440) to catch up to "now" and start tail'ing the log. (Part of that is the SSH pipe, but certainly not all!)

I can't even tell where clog itself is maintained, so this might not be possible, but it would be awesome if clog supported a mechanism for NOT starting all the way back at the beginning of the file. (This would be useful in both "-f" and regular modes, for the same reasons.)

Actions #1

Updated by Adam Thompson over 6 years ago

FWIW, I'm thinking of "tail -f"'s behaviour, where it only tail's the last ~10 lines (I think most implementations default to that, anyway). And some implementations let me specify not only "tail -1000" but also "tail -1000 -f".
Either way, for "clog -f" just starting at the current ring position would be helpful.

Some data here:

[2.3.4-RELEASE][root@remote.avant.ca]/root: time sh -c "clog /var/log/system.log | wc -l" 
  823199
1.490u 0.032s 0:01.50 101.3%    9+167k 0+0io 0pf+0w

the data, unsurprisingly, shows that pushing all the data through a tty and then on through ssh is the biggest problem. SSH compression helps somewhat, but doesn't address the underlying issue.

Actions #2

Updated by Jim Pingle over 6 years ago

The way clog reads the records it has to figure out where the start is and then unwind it from there, so it doesn't exactly know how far from the end it is, which complicates this sort of issue. It may be easier to give it a starting offset so it skips X amount from the start of the file but I'm still not sure it's worth it.

clog is doomed anyhow. It's getting harder and harder to keep it and its required patches maintained. We're investigating dropping it entirely in favor of plain text logs + newsyslog or similar in the very near future.

Actions #3

Updated by Adam Thompson over 6 years ago

Thanks, Jim. That would be a perfectly acceptable solution, with a whole bunch of side benefits.
Especially since I don't know of a single pfSense firewall still running on write-endurance-limited CF cards anyway. (Wal-mart sells USB sticks that have more write endurance than those old CF disks!)

Actions #4

Updated by Jim Pingle over 6 years ago

You'd be surprised, there are a number of them out there on CF, USB sticks and the like, and some of them have opted to have logs in RAM disks which are limited in size. clog was good for that, but that situation is definitely in the minority these days. Good log rotation will keep things in check and open up a lot more possibilities.

Actions #5

Updated by Jim Pingle over 4 years ago

  • Status changed from New to Closed

Issue will be moot once #8350 is implemented.

Actions

Also available in: Atom PDF