Bug #14498
openphp errors when looking at snort active rules
Added by Jonathan Lee over 1 year ago. Updated about 1 year ago.
0%
Description
Hello Fellow Redmine community members,
I found another php error when I go to look at active rules with Snort for WAN it goes to a all white screen and will not continue. After this error was on the dashboard. I do not know if this is of concern or not. But it matches the other php type errors. In the past it would only buffer what fit in the screen and if you scrolled it would load more.
Crash report begins. Anonymous machine information:
arm64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:31:56 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/aarch64/x3UPdyyG/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/sources/Free
Crash report details:
PHP Errors:
[21-Jun-2023 16:42:05 US/Pacific] PHP Fatal error: str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161
[21-Jun-2023 16:42:40 US/Pacific] PHP Fatal error: str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161
No FreeBSD crash data found.
Files
Updated by Christopher Cope over 1 year ago
We'll need more information to confirm if this is actually a bug. It is possible you are hitting the memory limit in PHP.
Are there any "Allowed memory size of xxx bytes exhausted" log entries in the Diagnostics > System Logs?
The redmine isn't the place for troubleshooting. If the above isn't true please open a ticket at https://www.netgate.com/tac-support-request and reference this redmine.
Updated by Jonathan Lee over 1 year ago
Hello thanks for the reply. This PHP error occurs when I attempt to view the active rules in snort. I only have 20 percent of my 4GB ram in use when I try to view this list. I assumed that a list is just text information when it is viewed and would never require that many GBs to simply display that info. If so the GUI should proper have error handling, would you agree? Leading to why leave that open to a buffer overflow? I am currently a computer science student at Sacramento State and hold an AA in Cyber security from Sierra college. This error concerns me because it is a buffer overflow for displaying active rules. If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again.
Updated by Jonathan Lee over 1 year ago
I do also have custom rules active inside snort. I do not know if that causes it. As custom rules are pasted in and load and function. I use emerging threats 3core sec rules. The rules function and are working and show on the blocked items. It always shows the same PHP error above when you go to check on active rules. It use to work in older versions of pfSense without the custom 3core sec rules loaded. I can attest to that. Have a happy 4th.
Updated by Jonathan Lee over 1 year ago
- File Screenshot 2023-07-01 at 9.53.49 PM.png Screenshot 2023-07-01 at 9.53.49 PM.png added
- File Screenshot 2023-07-01 at 9.54.12 PM.png Screenshot 2023-07-01 at 9.54.12 PM.png added
- File Screenshot 2023-07-01 at 9.55.20 PM.png Screenshot 2023-07-01 at 9.55.20 PM.png added
- File Screenshot 2023-07-01 at 9.58.44 PM.png Screenshot 2023-07-01 at 9.58.44 PM.png added
Per your request in 23.05.1
See attached system goes to blank screen error occurs and no errors in system logs that show memory issues.
I do not think many use custom rules in Snort that may be why this error went unnoticed.
URL to custom rules
https://rules.emergingthreats.net/blockrules/3coresec.rules
Updated by Jonathan Lee over 1 year ago
After sometime I still show no memory errors inside of the SG-2100MAX for this timestamp.
Please let me know if you require anything else for this PHP error
Updated by Jonathan Lee over 1 year ago
Also attached is proof that the custom rules I have in Snort are in use and functional within this regard.
Side thoughts reason for photo:
I do not know if custom rules could cause this PHP error when viewing active rules in Snort. Again if and when custom rules and loaded and they are active when checking "active rules." Leading to, if Snort can not iterate over the custom rules to display that information within active rules could possibly cause this PHP error.
Updated by Jonathan Lee over 1 year ago
TAC ticket open with this referenced copy of config is loaded with my serial number. I hope that provides everything that is needed to help improve this software.
Updated by Jonathan Lee over 1 year ago
The truth is, I really want to fix this PHP software issue, again I am still a student and rather overzealous when I find a bug. The ticket has the status.php file per Netgate requests
Updated by Bill Meeks over 1 year ago
This is a consequence of the PHP process itself running out of memory. Because the output is being buffered in an attempt to improve efficiency, I suspect the PHP "out-of-memory" error is getting swallowed during the buffering and not displayed.
There is no stateful function for reading and outputting very large text files in PHP. The only process currently available is to read the entire text file (the Snort active rules in this case) into PHP's allocated RAM space and store it as a string. The content of that stored string is then written out as HTML to the client. No matter how much RAM is in the firewall, there is a fixed limit on the memory PHP will use for string and other variables storage that is set within the php.ini
file.
When you have lots of enabled rules, then trying to view the "Active Rules" on the RULES tab in Snort or Suricata is likely to crash. Whether or not that happens is strictly up to exactly how many rules you have enabled. If the file that holds the active rules is too large, it won't fit into the PHP string memory space and you will get the error. Custom Rules are not invovled here other than the act of including them (if there are enough of them) in the list along with all the other rules may be enough to push you over the edge and run out of PHP string memory.
The same kind of error occurs when you try to view very large log files in Snort, Suricata, or any of the other packages (pfBlocker and others). The same thing can happen if you try to open any large text file on pfSense (search the forum for examples of this from certain logs).
The error only impacts the currently active PHP process that is outputting to the client display. It does not interrupt the operation of pfSense nor Snort itself. The binary is still running and offering protection and logging. Only that particular GUI session showing the active rules is impacted. You can simply reload the page or move to another tab and the GUI is instantly recovered.
Using web pages created via PHP to implement a GUI is not the same kind of environment as you have in a C, C#, Java or other program running directly in Windows or Linux. You don't have that same level of two-way back and forth between the pieces. With a web-based GUI, you basically do all the processing on the server and attempt to dump it as HTML to a dumb client (your browser in this case). You have no easy and reliable way to "remember" where you are when reading a large file and dumping it out. That's because each PHP GUI client session is essentially stateless. Once it dumps the current output stream, the process exits. That's different from something like a word processor window in a conventional OS GUI where you can precisely control how you read from a file in chunks and output those chunks to the display. Doing that in PHP with a stateless web session is not the same. I've toyed around with trying to implement a type of "paged output" for this situation and similar ones with large log files. But so far I've not come up with something that seems ideal.
So, the TLDR version is this -- it's a known problem with no easy fix. It is the same issue that impacts many other packages (and even pfSense itself) when trying to read very large text files and output their content to the GUI display
Updated by Jonathan Lee over 1 year ago
- File Screenshot 2023-07-02 at 3.20.45 PM.png Screenshot 2023-07-02 at 3.20.45 PM.png added
- File Screenshot 2023-07-02 at 3.23.54 PM.png Screenshot 2023-07-02 at 3.23.54 PM.png added
- File Screenshot 2023-07-02 at 3.25.08 PM.png Screenshot 2023-07-02 at 3.25.08 PM.png added
- File Screenshot 2023-07-02 at 3.32.50 PM.png Screenshot 2023-07-02 at 3.32.50 PM.png added
- File Screenshot 2023-07-02 at 3.32.57 PM.png Screenshot 2023-07-02 at 3.32.57 PM.png added
Thanks for your reply and looking into this at a granular level.
I noticed you said " I've toyed around with trying to implement a type of 'paged output' for this situation and similar ones with large log files. But so far I've not come up with something that seems ideal. "
I wanted to share this with you:
I keep thinking about my dynamic database program I wrote with Java for the class I was TA/Passpeer for at Sierra College and grader for at the University of Sacramento. I had this issue where I wanted to have all the database entry items to be part of a histogram but with each sale or adaption to the database file the histogram would be different. Anyway I had to use what was called image icon to load my Jframe windows with a new image I would create that was already saved when that method was called on. So I am now thinking is there anyway to save the text as an image in a say temp folder to view each page and delete it after?
Here is some photos of my code and program at work. Now drawing connections to PHP it could in theory do the same and just fill a image buffer to PHP's string text limits and save it in a temp folder. Again, make a second image save it and do the same until all pages are created. After use the images of the text for what is displayed in the GUI?
Updated by Jonathan Lee over 1 year ago
I would just make a buffered image and save it everytime that method was called on. It would save the file and open it right on the screen how I wanted each and every time. I thought it was a great simple way to make the database function and be dynamic as well as have dynamic histograms when ever you needed.
How were you attempting to implement a paged output? Was it images that you created and or just accessing sections of the string files as needed?
Thanks again for helping me to understand this and helping my path forward in my education also. I have also seen this error occur when you look at logs within the edit file area of the GUI
Updated by Ryan Coleman over 1 year ago
How were you attempting to implement a paged output? Was it images that you created and or just accessing sections of the string files as needed?
As noted earlier Redmines are not a place for troubleshooting. Please use our forums for that (https://forum.netgate.com/category/53/ids-ips).
Updated by Jonathan Lee over 1 year ago
@Bill Meeks
Thank you for confirming the code issue. As you quoted,
"No matter how much RAM is in the firewall, there is a fixed limit on the memory PHP will use for string and other variables storage that is set within the php.ini file"
"So, the TLDR version is this -- it's a known problem with no easy fix. It is the same issue that impacts many other packages (and even pfSense itself) when trying to read very large text files and output their content to the GUI display"
Thanks for looking into this as it can not be corrected at a configuration level by end users when this error occurs. I like your paged output solution.
Happy 4th
Updated by Jonathan Lee over 1 year ago
@Ryan Coleman
Can you mark my open TAC ticket #1731574435 as closed as it is confirmed this is a code/software issue and not a configuration/troubleshooting issue.
Thanks Happy 4th I think your the same guy in the TAC ticket asking me to reinstall Snort to resolve it.
Please read Bill Meeks notes above. Reinstall will not resolve this.
Updated by Jonathan Lee over 1 year ago
@Christopher Cope
I wanted to also take the time to message you and say I am sorry for the reply with, "If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again." That is not my intention to come across so cross. It is not ok. I am sorry Christopher. I am very overzealous about wanting to find any and all software bugs, or attempt to. I worked in IT for 15 combined years and returned to school to complete my degree. I am here now. Thus, I am very excited about programming and code, especially searching for errors in firewalls. I have been testing out anything and everything within pfSense. I have also learned about pfSense in class, I purchased an official Netgate appliance to learned with and have been testing out every config change I can think of. I think I just got flustered that you said this was "troubleshooting" and not a bug you know. I have seen this bug so many times within logs inside of pfSense. Leading to, software code issues when I find one to me feels like it's a gold mine of educational information. I wanted to let you know I did open a TAC ticket per your request. I am sorry, thanks for being the first to reply to this redmine ticket.
Happy 4th
Updated by Bill Meeks over 1 year ago
In the interest of coming to a resolution on this ticket...
The issue identified here is more of a generic problem with outputting the content of large text files (such as logs) using PHP and stateless sessions. It impacts more than just Snort and Suricata. Other packages have similar issues, and even pfSense itself can similarly fail when trying to output large text files via the GUI.
One suggested resolution is for pfSense to have a native PHP module that can statefully read a large text file in chunks and output each chunk separately to the HTML GUI client using a paged output approach perhaps outputting the content into an iframe??? You could maybe save state by passing some type of file position pointer back and forth using AJAX calls with POST. Packages could then call and use this pfSense-provided module when outputting large text files as HTML to the GUI client.
While it may be legit to consider the current behavior a bug, it could also reasonably be classified as a feature request that has a more universal application in pfSense.
Updated by Christopher Cope over 1 year ago
Jonathan Lee wrote in #note-16:
@Christopher Cope
I wanted to also take the time to message you and say I am sorry for the reply with, "If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again." That is not my intention to come across so cross. It is not ok. I am sorry Christopher. I am very overzealous about wanting to find any and all software bugs, or attempt to. I worked in IT for 15 combined years and returned to school to complete my degree. I am here now. Thus, I am very excited about programming and code, especially searching for errors in firewalls. I have been testing out anything and everything within pfSense. I have also learned about pfSense in class, I purchased an official Netgate appliance to learned with and have been testing out every config change I can think of. I think I just got flustered that you said this was "troubleshooting" and not a bug you know. I have seen this bug so many times within logs inside of pfSense. Leading to, software code issues when I find one to me feels like it's a gold mine of educational information. I wanted to let you know I did open a TAC ticket per your request. I am sorry, thanks for being the first to reply to this redmine ticket.Happy 4th
No offense taken.
Bill Meeks wrote in #note-17:
In the interest of coming to a resolution on this ticket...
The issue identified here is more of a generic problem with outputting the content of large text files (such as logs) using PHP and stateless sessions. It impacts more than just Snort and Suricata. Other packages have similar issues, and even pfSense itself can similarly fail when trying to output large text files via the GUI.
One suggested resolution is for pfSense to have a native PHP module that can statefully read a large text file in chunks and output each chunk separately to the HTML GUI client using a paged output approach perhaps outputting the content into an iframe??? You could maybe save state by passing some type of file position pointer back and forth using AJAX calls with POST. Packages could then call and use this pfSense-provided module when outputting large text files as HTML to the GUI client.
While it may be legit to consider the current behavior a bug, it could also reasonably be classified as a feature request that has a more universal application in pfSense.
I contributed code to the new packet capture page that does AJAX file reads for the live updating of that page. That code could be adapted for the other pages as well, but will take some work to get them adapted for such use.
https://redmine.pfsense.org/issues/13382
Another workaround is the pull request for the option to configure the PHP memory limit in the GUI. https://redmine.pfsense.org/issues/13377 It has a patch file attached that works, but it still needs some polish before it will be merged.
Updated by Jonathan Lee about 1 year ago
Amazing, thanks for sharing I appreciate you.
Updated by Jonathan Lee about 1 year ago
- File Screenshot 2023-08-04 093223.jpg Screenshot 2023-08-04 093223.jpg added
- File 0001-Add-a-setting-for-PHP-memory-limit-in-System-Advance.patch 0001-Add-a-setting-for-PHP-memory-limit-in-System-Advance.patch added
- File PHP_errors.log PHP_errors.log added
@Christopher Cope
I have tested your patch attached here. Strip level 2
set to 512mb
Hover I am still getting this Snort error when viewing active rules. I have this set to half a GB now.
Updated by Jonathan Lee about 1 year ago
[04-Aug-2023 09:30:42 US/Pacific] PHP Fatal error: str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161
With the memory of PHP being set to a larger value this error is still occurring. Per @Christopher Coper this error is not related to PHP memory limit issues, as I have tested his memory software patch to try to resolve this.
Thanks again.
Updated by Lev Prokofev about 1 year ago
The crash was produced in an attempt to grab the status output file, ticket #1936290053 there are no other PHP errors in the logs
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
Crash report details:
PHP Errors:
[28-Sep-2023 13:53:22 Asia/Kolkata] PHP Fatal error: str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161
No FreeBSD crash data found.