Bug #11583
closed
dashboard nginx 504 Gateway time-out error
Added by Adam Esslinger about 3 years ago.
Updated about 3 years ago.
Description
Ever since upgrading to version 2.5 logging into the firewall takes a really long time. Once logged in and navigating through other menus things are quick, however returning to the dashboard is extremely slow. Sometimes I cant even loginto the web GUI and I get an nginx 504 Gateway Time-out error. I have tried restarting the web configurator and PHP-FPM and even a reboot of the firewall to no avail. I have this same issue happening on 4 different firewalls. When I SSH in and do an htop Im seeing that /usr/bin/bzcat -qf /var/log/filter.log.6.bz2 is a command that is using 100% of a single core.
- Status changed from New to Not a Bug
once I finally got logged in I see this in the system logs:
2021/03/01 13:12:17 [error] 88327#100711: *20 upstream timed out (60: Operation timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "xxx.xxx.xxx.xxx", referrer: "https://xxx.xxx.xxx.xxx/status_logs.php"
note that I replaced the real IPs with xxx for security reasons
I was finally able to login by deleting the filter.log.x.bz2 files in the /var/log directory. There were 6 of them and each one was 59MB. This seems to be an issue with the firewall logs and having that widget on the dashboard.
That could maybe happen with an excessively large log file size (downright huge if it's 59MB compressed) but ultimately that's a settings issue affecting you negatively. You can reduce the size of the logs, disable log compression, or reduce the number of rotated logs that are retained.
If you need more help/guidance, please post on the forum for assistance.
I still believe this to be a bug, here is why:
1. When the firewall logs widget is on the dashboard its set to only show the most recent 10 log entries, yet I can see from an htop that a log process is trying to decompress all of the compressed logs
2. As far as log file size nothing was changed on any of these machines, they were an in-place upgrade from 2.4.5-p1 to 2.5.0. I just built a new VM of 2.4.5.-p1 and noticed that the the log settings page doesn't have the section about log compression where as in 2.5.0 logs were set to bzip2.
3. The symptoms of the dashboard got worse as time when on since the upgrade to 2.5.0 which again leads me to believe that its related to the logs being compressed because initially the logs would have be uncompressed in 2.4.5 and new logs as they went through the log rotation would have slowly become compressed.
4. The firewall logs page was set to display the recent 500 entries yet clikcing on that tab would also timeout.
Over all I believe that this is a specific bug with compressed firewall logs being show in any manner (firewall widget or firewall logs tab)
1. The log parser has no idea how far back it needs to go to find enough usable entries and it has to include rotated log files because of timing issues where the current log may not be enough. There are other issues with trying to search each one individually as well.
2. 2.5.0 moved to plain text logs + compressed and rotated, previously it was clog, which behaves quite differently
3. See previous reply with options to correct that behavior
4. See 1.
It's simple to address this by reducing your log sizes and/or changing compression and retention settings.
Hardware varies significantly so we can't overgeneralize how much load would be caused by doing this, it's up to the user to adjust the settings to suit their hardware and environment.
Ultimately disabling bzip2 fixed the issue. Bzip2 is the new format in 2.5.0, prior versions were uncompressed.
Also available in: Atom
PDF
Updated by 
Updated by