Bug #11605
closed
Suricata can trigger PHP crash on SG-3100
Added by Justin P almost 4 years ago. Updated over 3 years ago.
0%
Description
Suricata and SNORT won't start on 21.02p1 SG3100. Appears to be an issue related to PHP see the following post for more information:
https://forum.netgate.com/topic/161050/snort-won-t-start-after-upgrade-to-21-02-on-sg-3100/13
Current workaround is to revert to 2.4.5_1
Updated by Marcos M over 3 years ago
- Enable ETOpen rules and force update of rules
- On "Services / Suricata / Interfaces" click "Add"
- Default interface is WAN; click "Save"
Result:
Page times out with HTTP 502 error
System log shows:Mar 11 19:35:36 nginx 2021/03/11 19:35:36 [error] 67394#100107: *177 upstream prematurely closed connection while reading response header from upstream, client: 10.0.5.50, server: , request: "POST /suricata/suricata_interfaces_edit.php?id=0 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.0.20.103", referrer: "https://10.0.20.103/suricata/suricata_interfaces_edit.php?id=0" Mar 11 19:35:36 kernel pid 15438 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Crash report on dashboard shows up and says:[11-Mar-2021 19:28:27 Etc/UTC] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/local/pkg/suricata/suricata_check_for_rule_updates.php on line 684
It seems this only happens the first time, as deleting and re-adding the interface did not trigger the crash.
Additionally, changing to categories tab without saving interface first can also lead to php crash:- On "Services / Suricata / Interfaces" click "Add"
- Default interface is WAN; click "WAN Categories"
- Tabs name are now "Iface" instead of "WAN"
- Check a rule and click save
Result:
Page times out with HTTP 502 error
System log shows:Mar 11 19:48:20 nginx 2021/03/11 19:48:20 [error] 67394#100107: *609 upstream prematurely closed connection while reading response header from upstream, client: 10.0.5.50, server: , request: "POST /suricata/suricata_rulesets.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.0.20.103", referrer: "https://10.0.20.103/suricata/suricata_rulesets.php" Mar 11 19:48:20 kernel pid 506 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Likely related #11466
Updated by Marcos M over 3 years ago
- Subject changed from Suricata won't start on 21.02p1 SG3100 to Suricata can trigger PHP crash on SG-3100
Updated by Loh Phat over 3 years ago
Does the PHP temp workaround patch fix this one too?
Updated by Jim Pingle over 3 years ago
The patch should fix the behavior, but the package could also implement the fix on its own using ini_set("pcre.jit", "0"); in PHP on 32-bit ARM multi-core systems before performing PCRE operations. Doing it in the package would allow systems without the patch installed to benefit from the change. Same could be done for Snort and pfBlockerNG.
Updated by Bill Meeks over 3 years ago
Jim Pingle wrote:
The patch should fix the behavior, but the package could also implement the fix on its own using
ini_set("pcre.jit", "0");in PHP on 32-bit ARM multi-core systems before performing PCRE operations. Doing it in the package would allow systems without the patch installed to benefit from the change. Same could be done for Snort and pfBlockerNG.
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
Updated by Jim Pingle over 3 years ago
Bill Meeks wrote:
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.
You can run the function at any time on a page and it will be active for that page alone, instead of globally.
For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put ini_set("pcre.jit", "0"); in the PHP exec box avove the crash-prone code, it works.
Updated by Bill Meeks over 3 years ago
Jim Pingle wrote:
Bill Meeks wrote:
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.
You can run the function at any time on a page and it will be active for that page alone, instead of globally.
For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put
ini_set("pcre.jit", "0");in the PHP exec box avove the crash-prone code, it works.
That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.
Updated by Justin P over 3 years ago
Bill Meeks wrote:
Jim Pingle wrote:
Bill Meeks wrote:
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.
You can run the function at any time on a page and it will be active for that page alone, instead of globally.
For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put
ini_set("pcre.jit", "0");in the PHP exec box avove the crash-prone code, it works.That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.
All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.
Updated by Justin P over 3 years ago
Justin P wrote:
Bill Meeks wrote:
Jim Pingle wrote:
Bill Meeks wrote:
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.
You can run the function at any time on a page and it will be active for that page alone, instead of globally.
For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put
ini_set("pcre.jit", "0");in the PHP exec box avove the crash-prone code, it works.That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.
All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.
Justin P wrote:
Bill Meeks wrote:
Jim Pingle wrote:
Bill Meeks wrote:
Does this function call work without restarting PHP? I don't have hardware at the moment to test with, but I thought with your patch file posted earlier for the 11466 bug that you had to either restart the PHP process or reboot the firewall for the INI change to take effect in the PHP runtime environment.
The patch requires that because it writes it to the global php.ini file so it needs an extra kick to be active for everything/everywhere.
You can run the function at any time on a page and it will be active for that page alone, instead of globally.
For example your sample code that crashes PHP will crash on Diag > Command on an unpatched system but if you put
ini_set("pcre.jit", "0");in the PHP exec box avove the crash-prone code, it works.That's good news. I guess I will reconsider, then, my position stated in the notes for the #11466 bug report.
All I have applied the patch and will begin testing this weekend. I use Suricata, OPENVPN, and PfBlockerNG so we should be able to get an idea of any errors, if any. I will test throughout the week until Thursday then report back. Thanks for your continued support and help resolving this.
Unfortunately suricata is still crashing every day and PHP appears to send an error every night. I am working on log collection to narrow down the root cause.
Seems as if suricata doesn't restart on some interfaces after updates. It leaves the interface in non promiscuous state which stops suricata on the interface since it cannot get the pcap
Updated by Jim Pingle over 3 years ago
- Status changed from New to Closed
Closing this as it appears to be the same root cause as #11466 which has a workaround applied as #12004 -- Users can wait for that to be integrated into a release or apply it manually using the instructions on that issue.
If you apply that workaround properly and still have problems, report them on the forum at https://forum.netgate.com/topic/164725/netgate-3100-php-crashes and include any error messages displayed in the GUI or log, crash reports, and other relevant data.
Problems unrelated to PHP crashes in suricata should be investigated separately as they are unlikely to be related to this issue, but we also need a lot more detail about how it is failing.