Bug #11583


dashboard nginx 504 Gateway time-out error

Added by Adam Esslinger over 3 years ago. Updated over 3 years ago.

Not a Bug
Target version:
Start date:
Due date:
% Done:


Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
Affected Architecture:


Ever since upgrading to version 2.5 logging into the firewall takes a really long time. Once logged in and navigating through other menus things are quick, however returning to the dashboard is extremely slow. Sometimes I cant even loginto the web GUI and I get an nginx 504 Gateway Time-out error. I have tried restarting the web configurator and PHP-FPM and even a reboot of the firewall to no avail. I have this same issue happening on 4 different firewalls. When I SSH in and do an htop Im seeing that /usr/bin/bzcat -qf /var/log/filter.log.6.bz2 is a command that is using 100% of a single core.

Actions #1

Updated by Jim Pingle over 3 years ago

  • Status changed from New to Not a Bug

There isn't enough information here to point to one specific issue and this site is not for support or diagnostic discussion.

For assistance in solving problems, please post on the Netgate Forum or the pfSense Subreddit .

See Reporting Issues with pfSense Software for more information.

Actions #2

Updated by Adam Esslinger over 3 years ago

once I finally got logged in I see this in the system logs:

2021/03/01 13:12:17 [error] 88327#100711: *20 upstream timed out (60: Operation timed out) while reading response header from upstream, client:, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "", referrer: ""

note that I replaced the real IPs with xxx for security reasons

Actions #3

Updated by Adam Esslinger over 3 years ago

I was finally able to login by deleting the filter.log.x.bz2 files in the /var/log directory. There were 6 of them and each one was 59MB. This seems to be an issue with the firewall logs and having that widget on the dashboard.

Actions #4

Updated by Jim Pingle over 3 years ago

That could maybe happen with an excessively large log file size (downright huge if it's 59MB compressed) but ultimately that's a settings issue affecting you negatively. You can reduce the size of the logs, disable log compression, or reduce the number of rotated logs that are retained.

If you need more help/guidance, please post on the forum for assistance.

Actions #5

Updated by Adam Esslinger over 3 years ago

I still believe this to be a bug, here is why:
1. When the firewall logs widget is on the dashboard its set to only show the most recent 10 log entries, yet I can see from an htop that a log process is trying to decompress all of the compressed logs
2. As far as log file size nothing was changed on any of these machines, they were an in-place upgrade from 2.4.5-p1 to 2.5.0. I just built a new VM of 2.4.5.-p1 and noticed that the the log settings page doesn't have the section about log compression where as in 2.5.0 logs were set to bzip2.
3. The symptoms of the dashboard got worse as time when on since the upgrade to 2.5.0 which again leads me to believe that its related to the logs being compressed because initially the logs would have be uncompressed in 2.4.5 and new logs as they went through the log rotation would have slowly become compressed.
4. The firewall logs page was set to display the recent 500 entries yet clikcing on that tab would also timeout.

Over all I believe that this is a specific bug with compressed firewall logs being show in any manner (firewall widget or firewall logs tab)

Actions #6

Updated by Jim Pingle over 3 years ago

1. The log parser has no idea how far back it needs to go to find enough usable entries and it has to include rotated log files because of timing issues where the current log may not be enough. There are other issues with trying to search each one individually as well.
2. 2.5.0 moved to plain text logs + compressed and rotated, previously it was clog, which behaves quite differently
3. See previous reply with options to correct that behavior
4. See 1.

It's simple to address this by reducing your log sizes and/or changing compression and retention settings.

Hardware varies significantly so we can't overgeneralize how much load would be caused by doing this, it's up to the user to adjust the settings to suit their hardware and environment.

Actions #7

Updated by Adam Esslinger over 3 years ago

Ultimately disabling bzip2 fixed the issue. Bzip2 is the new format in 2.5.0, prior versions were uncompressed.


Also available in: Atom PDF