Project

General

Profile

Actions
Copy link

Regression #11475

closed

Route tables with many entries can lead to PHP errors and timeouts when looking up routes

Added by Dirk Meyer almost 4 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Renato Botelho
Category:
Routing
Target version:
2.5.1
Start date:
02/19/2021
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
Release Notes:
Default
Affected Version:
2.5.0
Affected Architecture:
All

Description

Pfsense with FFR crashes in the web interface after update from 2.4.5-p1 to Pfsense 2.5.0
Right after update the dashboard fails with an empty page.

19-Feb-2021 15:37:26 Europe/Berlin] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/util.inc on line 2632@

I raised the memory limits a few times in /etc/inc/config.inc:

[19-Feb-2021 19:41:01 Europe/Berlin] PHP Fatal error:  Allowed memory size of 1342177280 bytes exhausted (tried to allocate 4096 bytes) in /etc/inc/util.inc on line 2632

Finally the dashboard shows with in /etc/inc/config.inc:
ini_set("memory_limit", "1500M");

But the web interface is still very slow,
often fails with "504 Gateway Time-out"

top shows that "netstat" and "php-fpm" is running with high load.

The PHP code is reading the full routing table
with "/usr/bin/netstat --libxo json -nWr"
just to get the default gateway.

netstat -nr4 | wc -l
  826327
netstat -nr6 | wc -l
  103340

A much faster way to get the default route would be to call:

route -4n get default
route -6n get default

Files

Download all files
utils.inc.diff (917 Bytes) utils.inc.diff Renato Botelho, 02/22/2021 09:35 AM
169.diff (1.7 KB) 169.diff Viktor Gurov, 03/02/2021 06:57 AM
Actions
Copy link
 #2

Updated by Jim Pingle almost 4 years ago

  • Subject changed from Pfsense with FFR crashes in the web interface after update to pfsense 2.5.0 to Large routing tables cause PHP errors/timeouts when fetching the default gateway
  • Status changed from New to Pull Request Review
  • Assignee set to Renato Botelho
  • Target version set to 2.5.1
Actions
Copy link
 #3

Updated by Renato Botelho almost 4 years ago

Dirk,

Can you try attached patch and let me know if it helps?

Actions
Copy link
 #4

Updated by Renato Botelho almost 4 years ago

  • Status changed from Pull Request Review to In Progress
Actions
Copy link
 #5

Updated by Dirk Meyer almost 4 years ago

Renato Botelho wrote:

Dirk,

Can you try attached patch and let me know if it helps?

The patch looks like an improved to cut down the search time in PHP.

With the patch applied, the dashboard still fails:

[22-Feb-2021 19:14:46 Europe/Berlin] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /etc/inc/util.inc on line 2632

The problem seems to occur while reading and parsing the long output of netstat in line 2632:

$netstatarr = json_decode(implode(" ", $rawdata), JSON_OBJECT_AS_ARRAY);

Some measurements:

/usr/bin/netstat --libxo json -nWr -4 | wc 
       2 2474365 132563849
/usr/bin/netstat --libxo json -nWr -6 | wc
       2  310186 18411461

For decoding the 132 MB Jason, the function needs more than 1024 MB RAM.

Actions
Copy link
 #6

Updated by Renato Botelho almost 4 years ago

Dirk Meyer wrote:

Renato Botelho wrote:

Dirk,

Can you try attached patch and let me know if it helps?

The patch looks like an improved to cut down the search time in PHP.

With the patch applied, the dashboard still fails:

[...]

The problem seems to occur while reading and parsing the long output of netstat in line 2632:

[...]

Some measurements:
[...]

For decoding the 132 MB Jason, the function needs more than 1024 MB RAM.

Thanks! I'm going to refactor that code.

Actions
Copy link
 #7

Updated by Renato Botelho almost 4 years ago

  • Status changed from In Progress to Feedback

PR has been merged. Thanks!

Actions
Copy link
 #8

Updated by Viktor Gurov almost 4 years ago

  • % Done changed from 0 to 100
Actions
Copy link
 #9

Updated by Dirk Meyer almost 4 years ago

note: The patch is an improvement, but it doesn't solve the issue.

Actions
Copy link
 #10

Updated by Viktor Gurov almost 4 years ago

Dirk Meyer wrote:

note: The patch is an improvement, but it doesn't solve the issue.

You need to apply patch ID 7990de53bfc8267d1dd96636a175929a35cbe664
it uses `route -4n get default` to get the default route

Actions
Copy link
 #11

Updated by Dirk Meyer almost 4 years ago

Thanks, I can verify now the dashboard works again.

But while on system_gateways.php I pressed Save:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/util.inc on line 2632 PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2632, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes) in /etc/inc/notices.inc on line 335

Actions
Copy link
 #12

Updated by Viktor Gurov almost 4 years ago

Please try this patch

extra fix for route_get() and route_del():
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/169

Actions
Copy link
 #13

Updated by Jim Pingle almost 4 years ago

  • Status changed from Feedback to Pull Request Review
Actions
Copy link
 #14

Updated by Dirk Meyer almost 4 years ago

With patch 7990de53bfc8267d1dd96636a175929a35cbe664
and patch 169.diff applied, the errors didn't show anymore.

Thanks.

Actions
Copy link
 #15

Updated by Renato Botelho almost 4 years ago

  • Status changed from Pull Request Review to Waiting on Merge

PR has been merged. Thanks!

Actions
Copy link
 #16

Updated by Renato Botelho almost 4 years ago

  • Status changed from Waiting on Merge to Feedback
Actions
Copy link
 #17

Updated by Renato Botelho almost 4 years ago

Cherry-picked to RELENG_2_5_1

Actions
Copy link
 #18

Updated by Jim Pingle almost 4 years ago

  • Subject changed from Large routing tables cause PHP errors/timeouts when fetching the default gateway to Route tables with many entries can lead to PHP errors and timeouts when looking up routes

Updating subject for release notes.

Actions
Copy link
 #19

Updated by yon Liu almost 4 years ago

when running frr bgp route with large full routes system, the gateways.php and system_routes.php edit and save change will happen "504 Gateway Time-out"

tested version:
2.6.0-DEVELOPMENT (amd64)
built on Wed Mar 10 01:03:52 EST 2021
FreeBSD 12.2-STABLE

Actions
Copy link
 #20

Updated by Viktor Gurov almost 4 years ago

  • Status changed from Feedback to New

now I see many `route: route has not been found` messages on boot:

Starting PC/SC Smart Card Services...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring QinQ interfaces...done.
Configuring WAN interface...done.
Configuring SYNC interface...done.
Configuring WG0 interface...route: route has not been found
route: route has not been found
route: route has not been found
route: route has not been found

fix:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/187

Actions
Copy link
 #21

Updated by Jim Pingle almost 4 years ago

  • Status changed from New to Feedback

Luiz merged the PR and cherry-picked but needs confirmation that the fix is in and working in snapshots.

Actions
Copy link
 #22

Updated by Jim Pingle over 3 years ago

  • Status changed from Feedback to Closed
Actions
Copy link

Also available in: Atom PDF