Project

General

Profile

Regression #11475

Route tables with many entries can lead to PHP errors and timeouts when looking up routes

Added by Dirk Meyer about 2 months ago. Updated 29 days ago.

Status:
Feedback
Priority:
Normal
Category:
Routing
Target version:
Start date:
02/19/2021
Due date:
% Done:

100%

Estimated time:
Affected Version:
2.5.0
Affected Architecture:
All
Release Notes:
Default

Description

Pfsense with FFR crashes in the web interface after update from 2.4.5-p1 to Pfsense 2.5.0
Right after update the dashboard fails with an empty page.

19-Feb-2021 15:37:26 Europe/Berlin] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/util.inc on line 2632@

I raised the memory limits a few times in /etc/inc/config.inc:

[19-Feb-2021 19:41:01 Europe/Berlin] PHP Fatal error:  Allowed memory size of 1342177280 bytes exhausted (tried to allocate 4096 bytes) in /etc/inc/util.inc on line 2632

Finally the dashboard shows with in /etc/inc/config.inc:
ini_set("memory_limit", "1500M");

But the web interface is still very slow,
often fails with "504 Gateway Time-out"

top shows that "netstat" and "php-fpm" is running with high load.

The PHP code is reading the full routing table
with "/usr/bin/netstat --libxo json -nWr"
just to get the default gateway.

netstat -nr4 | wc -l
  826327
netstat -nr6 | wc -l
  103340

A much faster way to get the default route would be to call:

route -4n get default
route -6n get default

utils.inc.diff (917 Bytes) utils.inc.diff Renato Botelho, 02/22/2021 09:35 AM
169.diff (1.7 KB) 169.diff Viktor Gurov, 03/02/2021 06:57 AM

Associated revisions

Revision 7990de53 (diff)
Added by Viktor Gurov about 1 month ago

route_get() optimization. Fixes #11475

Revision 07b780c8 (diff)
Added by Viktor Gurov about 1 month ago

route_del() optimization. Issue #11475

Revision 7a42c5d0 (diff)
Added by Viktor Gurov about 1 month ago

route_get() optimization. Fixes #11475

(cherry picked from commit 7990de53bfc8267d1dd96636a175929a35cbe664)

Revision 6790dc8c (diff)
Added by Viktor Gurov about 1 month ago

route_del() optimization. Issue #11475

(cherry picked from commit 07b780c84305142e2f3af8587b909bf004f11568)

Revision 46ff02ac (diff)
Added by Viktor Gurov about 1 month ago

Supress route no found error. Issue #11475

Revision f5ff5cdc
Added by Luiz Souza 29 days ago

Merge pull request #187 from viktor/route_get_fix

Supress route no found error. Issue #11475

Revision c9b7ffc3 (diff)
Added by Luiz Souza 29 days ago

Merge pull request #187 from viktor/route_get_fix

Supress route no found error. Issue #11475

(cherry picked from commit f5ff5cdc369b494499db3f7aca4426952add59e3)

History

#2 Updated by Jim Pingle about 2 months ago

  • Subject changed from Pfsense with FFR crashes in the web interface after update to pfsense 2.5.0 to Large routing tables cause PHP errors/timeouts when fetching the default gateway
  • Status changed from New to Pull Request Review
  • Assignee set to Renato Botelho
  • Target version set to 2.5.1

#3 Updated by Renato Botelho about 2 months ago

Dirk,

Can you try attached patch and let me know if it helps?

#4 Updated by Renato Botelho about 2 months ago

  • Status changed from Pull Request Review to In Progress

#5 Updated by Dirk Meyer about 2 months ago

Renato Botelho wrote:

Dirk,

Can you try attached patch and let me know if it helps?

The patch looks like an improved to cut down the search time in PHP.

With the patch applied, the dashboard still fails:

[22-Feb-2021 19:14:46 Europe/Berlin] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /etc/inc/util.inc on line 2632

The problem seems to occur while reading and parsing the long output of netstat in line 2632:

$netstatarr = json_decode(implode(" ", $rawdata), JSON_OBJECT_AS_ARRAY);

Some measurements:

/usr/bin/netstat --libxo json -nWr -4 | wc 
       2 2474365 132563849
/usr/bin/netstat --libxo json -nWr -6 | wc
       2  310186 18411461

For decoding the 132 MB Jason, the function needs more than 1024 MB RAM.

#6 Updated by Renato Botelho about 2 months ago

Dirk Meyer wrote:

Renato Botelho wrote:

Dirk,

Can you try attached patch and let me know if it helps?

The patch looks like an improved to cut down the search time in PHP.

With the patch applied, the dashboard still fails:

[...]

The problem seems to occur while reading and parsing the long output of netstat in line 2632:

[...]

Some measurements:
[...]

For decoding the 132 MB Jason, the function needs more than 1024 MB RAM.

Thanks! I'm going to refactor that code.

#7 Updated by Renato Botelho about 1 month ago

  • Status changed from In Progress to Feedback

PR has been merged. Thanks!

#8 Updated by Viktor Gurov about 1 month ago

  • % Done changed from 0 to 100

#9 Updated by Dirk Meyer about 1 month ago

note: The patch is an improvement, but it doesn't solve the issue.

#10 Updated by Viktor Gurov about 1 month ago

Dirk Meyer wrote:

note: The patch is an improvement, but it doesn't solve the issue.

You need to apply patch ID 7990de53bfc8267d1dd96636a175929a35cbe664
it uses `route -4n get default` to get the default route

#11 Updated by Dirk Meyer about 1 month ago

Thanks, I can verify now the dashboard works again.

But while on system_gateways.php I pressed Save:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/util.inc on line 2632 PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2632, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes) in /etc/inc/notices.inc on line 335

#12 Updated by Viktor Gurov about 1 month ago

Please try this patch

extra fix for route_get() and route_del():
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/169

#13 Updated by Jim Pingle about 1 month ago

  • Status changed from Feedback to Pull Request Review

#14 Updated by Dirk Meyer about 1 month ago

With patch 7990de53bfc8267d1dd96636a175929a35cbe664
and patch 169.diff applied, the errors didn't show anymore.

Thanks.

#15 Updated by Renato Botelho about 1 month ago

  • Status changed from Pull Request Review to Waiting on Merge

PR has been merged. Thanks!

#16 Updated by Renato Botelho about 1 month ago

  • Status changed from Waiting on Merge to Feedback

#17 Updated by Renato Botelho about 1 month ago

Cherry-picked to RELENG_2_5_1

#18 Updated by Jim Pingle about 1 month ago

  • Subject changed from Large routing tables cause PHP errors/timeouts when fetching the default gateway to Route tables with many entries can lead to PHP errors and timeouts when looking up routes

Updating subject for release notes.

#19 Updated by yon Liu about 1 month ago

when running frr bgp route with large full routes system, the gateways.php and system_routes.php edit and save change will happen "504 Gateway Time-out"

tested version:
2.6.0-DEVELOPMENT (amd64)
built on Wed Mar 10 01:03:52 EST 2021
FreeBSD 12.2-STABLE

#20 Updated by Viktor Gurov about 1 month ago

  • Status changed from Feedback to New

now I see many `route: route has not been found` messages on boot:

Starting PC/SC Smart Card Services...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring QinQ interfaces...done.
Configuring WAN interface...done.
Configuring SYNC interface...done.
Configuring WG0 interface...route: route has not been found
route: route has not been found
route: route has not been found
route: route has not been found

fix:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/187

#21 Updated by Jim Pingle 29 days ago

  • Status changed from New to Feedback

Luiz merged the PR and cherry-picked but needs confirmation that the fix is in and working in snapshots.

Also available in: Atom PDF