Project

General

Profile

Actions

Bug #11605

closed

Suricata can trigger PHP crash on SG-3100

Added by Justin P 5 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Suricata
Target version:
-
Start date:
03/02/2021
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
2.5.x
Affected Plus Version:
Affected Architecture:
SG-3100

Description

Suricata and SNORT won't start on 21.02p1 SG3100. Appears to be an issue related to PHP see the following post for more information:

https://forum.netgate.com/topic/161050/snort-won-t-start-after-upgrade-to-21-02-on-sg-3100/13

Current workaround is to revert to 2.4.5_1

Actions #1

Updated by Marcos Mendoza 5 months ago

Tested on [21.02.2 built on Thu Mar 11 09:10:56 EST 2021] with Suriata 4.1.9_5 on a fresh install.
  1. Enable ETOpen rules and force update of rules
  2. On "Services / Suricata / Interfaces" click "Add"
  3. Default interface is WAN; click "Save"
    Result:
    Page times out with HTTP 502 error
    System log shows:
    Mar 11 19:35:36     nginx         2021/03/11 19:35:36 [error] 67394#100107: *177 upstream prematurely closed connection while reading response header from upstream, client: 10.0.5.50, server: , request: "POST /suricata/suricata_interfaces_edit.php?id=0 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.0.20.103", referrer: "https://10.0.20.103/suricata/suricata_interfaces_edit.php?id=0" 
    Mar 11 19:35:36     kernel         pid 15438 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) 
    

    Crash report on dashboard shows up and says:
    [11-Mar-2021 19:28:27 Etc/UTC] PHP Warning:  count(): Parameter must be an array or an object that implements Countable in /usr/local/pkg/suricata/suricata_check_for_rule_updates.php on line 684
    

It seems this only happens the first time, as deleting and re-adding the interface did not trigger the crash.

Additionally, changing to categories tab without saving interface first can also lead to php crash:
  1. On "Services / Suricata / Interfaces" click "Add"
  2. Default interface is WAN; click "WAN Categories"
  3. Tabs name are now "Iface" instead of "WAN"
  4. Check a rule and click save
    Result:
    Page times out with HTTP 502 error
    System log shows:
    Mar 11 19:48:20     nginx         2021/03/11 19:48:20 [error] 67394#100107: *609 upstream prematurely closed connection while reading response header from upstream, client: 10.0.5.50, server: , request: "POST /suricata/suricata_rulesets.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.0.20.103", referrer: "https://10.0.20.103/suricata/suricata_rulesets.php" 
    Mar 11 19:48:20     kernel         pid 506 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) 
    

Likely related #11466

Actions #2

Updated by Marcos Mendoza 5 months ago

  • Subject changed from Suricata won't start on 21.02p1 SG3100 to Suricata can trigger PHP crash on SG-3100
Actions #3

Updated by Darin May about 2 months ago

Does the PHP temp workaround patch fix this one too?

https://redmine.pfsense.org/issues/11466#note-32

Actions #4

Updated by Jim Pingle about 2 months ago

The patch should fix the behavior, but the package could also implement the fix on its own using ini_set("pcre.jit", "0"); in PHP on 32-bit ARM multi-core systems before performing PCRE operations. Doing it in the package would allow systems without the patch installed to benefit from the change. Same could be done for Snort and pfBlockerNG.

Actions #5

Updated by Jim Pingle about 2 months ago

See also: #12004

Actions #6

Updated by Bill Meeks about 2 months ago

Jim Pingle wrote:

The patch should fix the behavior, but the package could also implement the fix on its own using ini_set("pcre.jit", "0"); in PHP on 32-bit ARM multi-core systems before performing PCRE operations. Doing it in the package would allow systems without the patch installed to benefit from the change. Same could be done for Snort and pfBlockerNG.

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

Actions #7

Updated by Jim Pingle about 2 months ago

Bill Meeks wrote:

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.

You can run the function at any time on a page and it will be active for that page alone, instead of globally.

For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.

Actions #8

Updated by Bill Meeks about 2 months ago

Jim Pingle wrote:

Bill Meeks wrote:

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.

You can run the function at any time on a page and it will be active for that page alone, instead of globally.

For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.

That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.

Actions #9

Updated by Justin P about 2 months ago

Bill Meeks wrote:

Jim Pingle wrote:

Bill Meeks wrote:

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.

You can run the function at any time on a page and it will be active for that page alone, instead of globally.

For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.

That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.

All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.

Actions #10

Updated by Justin P about 2 months ago

Justin P wrote:

Bill Meeks wrote:

Jim Pingle wrote:

Bill Meeks wrote:

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.

You can run the function at any time on a page and it will be active for that page alone, instead of globally.

For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.

That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.

All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.

Justin P wrote:

Bill Meeks wrote:

Jim Pingle wrote:

Bill Meeks wrote:

Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.

The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.

You can run the function at any time on a page and it will be active for that page alone, instead of globally.

For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.

That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.

All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.

Unfortunately suricata is still crashing every day and PHP appears to send an error every night. I am working on log collection to narrow down the root cause.

Seems as if suricata doesn't restart on some interfaces after updates. It leaves the interface in non promiscuous state which stops suricata on the interface since it cannot get the pcap

Actions #11

Updated by Jim Pingle about 1 month ago

  • Status changed from New to Closed

Closing this as it appears to be the same root cause as #11466 which has a workaround applied as #12004 -- Users can wait for that to be integrated into a release or apply it manually using the instructions on that issue.

If you apply that workaround properly and still have problems, report them on the forum at https://forum.netgate.com/topic/164725/netgate-3100-php-crashes and include any error messages displayed in the GUI or log, crash reports, and other relevant data.

Problems unrelated to PHP crashes in suricata should be investigated separately as they are unlikely to be related to this issue, but we also need a lot more detail about how it is failing.

Actions

Also available in: Atom PDF