pfSense » pfSense Packages

We'll need more information to confirm if this is actually a bug. It is possible you are hitting the memory limit in PHP.

Are there any "Allowed memory size of xxx bytes exhausted" log entries in the Diagnostics > System Logs?

The redmine isn't the place for troubleshooting. If the above isn't true please open a ticket at https://www.netgate.com/tac-support-request and reference this redmine.

Hello thanks for the reply. This PHP error occurs when I attempt to view the active rules in snort. I only have 20 percent of my 4GB ram in use when I try to view this list. I assumed that a list is just text information when it is viewed and would never require that many GBs to simply display that info. If so the GUI should proper have error handling, would you agree? Leading to why leave that open to a buffer overflow? I am currently a computer science student at Sacramento State and hold an AA in Cyber security from Sierra college. This error concerns me because it is a buffer overflow for displaying active rules. If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again.

I do also have custom rules active inside snort. I do not know if that causes it. As custom rules are pasted in and load and function. I use emerging threats 3core sec rules. The rules function and are working and show on the blocked items. It always shows the same PHP error above when you go to check on active rules. It use to work in older versions of pfSense without the custom 3core sec rules loaded. I can attest to that. Have a happy 4th.

TAC ticket open with this referenced copy of config is loaded with my serial number. I hope that provides everything that is needed to help improve this software.

Your ticket number is: 1731574435

The truth is, I really want to fix this PHP software issue, again I am still a student and rather overzealous when I find a bug. The ticket has the status.php file per Netgate requests

This is a consequence of the PHP process itself running out of memory. Because the output is being buffered in an attempt to improve efficiency, I suspect the PHP "out-of-memory" error is getting swallowed during the buffering and not displayed.

There is no stateful function for reading and outputting very large text files in PHP. The only process currently available is to read the entire text file (the Snort active rules in this case) into PHP's allocated RAM space and store it as a string. The content of that stored string is then written out as HTML to the client. No matter how much RAM is in the firewall, there is a fixed limit on the memory PHP will use for string and other variables storage that is set within the php.ini file.

When you have lots of enabled rules, then trying to view the "Active Rules" on the RULES tab in Snort or Suricata is likely to crash. Whether or not that happens is strictly up to exactly how many rules you have enabled. If the file that holds the active rules is too large, it won't fit into the PHP string memory space and you will get the error. Custom Rules are not invovled here other than the act of including them (if there are enough of them) in the list along with all the other rules may be enough to push you over the edge and run out of PHP string memory.

The same kind of error occurs when you try to view very large log files in Snort, Suricata, or any of the other packages (pfBlocker and others). The same thing can happen if you try to open any large text file on pfSense (search the forum for examples of this from certain logs).

The error only impacts the currently active PHP process that is outputting to the client display. It does not interrupt the operation of pfSense nor Snort itself. The binary is still running and offering protection and logging. Only that particular GUI session showing the active rules is impacted. You can simply reload the page or move to another tab and the GUI is instantly recovered.

Using web pages created via PHP to implement a GUI is not the same kind of environment as you have in a C, C#, Java or other program running directly in Windows or Linux. You don't have that same level of two-way back and forth between the pieces. With a web-based GUI, you basically do all the processing on the server and attempt to dump it as HTML to a dumb client (your browser in this case). You have no easy and reliable way to "remember" where you are when reading a large file and dumping it out. That's because each PHP GUI client session is essentially stateless. Once it dumps the current output stream, the process exits. That's different from something like a word processor window in a conventional OS GUI where you can precisely control how you read from a file in chunks and output those chunks to the display. Doing that in PHP with a stateless web session is not the same. I've toyed around with trying to implement a type of "paged output" for this situation and similar ones with large log files. But so far I've not come up with something that seems ideal.

So, the TLDR version is this -- it's a known problem with no easy fix. It is the same issue that impacts many other packages (and even pfSense itself) when trying to read very large text files and output their content to the GUI display

I would just make a buffered image and save it everytime that method was called on. It would save the file and open it right on the screen how I wanted each and every time. I thought it was a great simple way to make the database function and be dynamic as well as have dynamic histograms when ever you needed.

How were you attempting to implement a paged output? Was it images that you created and or just accessing sections of the string files as needed?

Thanks again for helping me to understand this and helping my path forward in my education also. I have also seen this error occur when you look at logs within the edit file area of the GUI

How were you attempting to implement a paged output? Was it images that you created and or just accessing sections of the string files as needed?

As noted earlier Redmines are not a place for troubleshooting. Please use our forums for that (https://forum.netgate.com/category/53/ids-ips).

@Bill Meeks

Thank you for confirming the code issue. As you quoted,

"No matter how much RAM is in the firewall, there is a fixed limit on the memory PHP will use for string and other variables storage that is set within the php.ini file"

"So, the TLDR version is this -- it's a known problem with no easy fix. It is the same issue that impacts many other packages (and even pfSense itself) when trying to read very large text files and output their content to the GUI display"

Thanks for looking into this as it can not be corrected at a configuration level by end users when this error occurs. I like your paged output solution.

Happy 4th

@Ryan Coleman

Can you mark my open TAC ticket #1731574435 as closed as it is confirmed this is a code/software issue and not a configuration/troubleshooting issue.

Thanks Happy 4th I think your the same guy in the TAC ticket asking me to reinstall Snort to resolve it.

Please read Bill Meeks notes above. Reinstall will not resolve this.

@Christopher Cope
I wanted to also take the time to message you and say I am sorry for the reply with, "If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again." That is not my intention to come across so cross. It is not ok. I am sorry Christopher. I am very overzealous about wanting to find any and all software bugs, or attempt to. I worked in IT for 15 combined years and returned to school to complete my degree. I am here now. Thus, I am very excited about programming and code, especially searching for errors in firewalls. I have been testing out anything and everything within pfSense. I have also learned about pfSense in class, I purchased an official Netgate appliance to learned with and have been testing out every config change I can think of. I think I just got flustered that you said this was "troubleshooting" and not a bug you know. I have seen this bug so many times within logs inside of pfSense. Leading to, software code issues when I find one to me feels like it's a gold mine of educational information. I wanted to let you know I did open a TAC ticket per your request. I am sorry, thanks for being the first to reply to this redmine ticket.

Happy 4th

In the interest of coming to a resolution on this ticket...

The issue identified here is more of a generic problem with outputting the content of large text files (such as logs) using PHP and stateless sessions. It impacts more than just Snort and Suricata. Other packages have similar issues, and even pfSense itself can similarly fail when trying to output large text files via the GUI.

One suggested resolution is for pfSense to have a native PHP module that can statefully read a large text file in chunks and output each chunk separately to the HTML GUI client using a paged output approach perhaps outputting the content into an iframe??? You could maybe save state by passing some type of file position pointer back and forth using AJAX calls with POST. Packages could then call and use this pfSense-provided module when outputting large text files as HTML to the GUI client.

While it may be legit to consider the current behavior a bug, it could also reasonably be classified as a feature request that has a more universal application in pfSense.

Jonathan Lee wrote in #note-16:

@Christopher Cope
I wanted to also take the time to message you and say I am sorry for the reply with, "If you do not see this as an error I suggest you not respond to it as it could be out of your capacity and capabilities to resolve it. Thanks again." That is not my intention to come across so cross. It is not ok. I am sorry Christopher. I am very overzealous about wanting to find any and all software bugs, or attempt to. I worked in IT for 15 combined years and returned to school to complete my degree. I am here now. Thus, I am very excited about programming and code, especially searching for errors in firewalls. I have been testing out anything and everything within pfSense. I have also learned about pfSense in class, I purchased an official Netgate appliance to learned with and have been testing out every config change I can think of. I think I just got flustered that you said this was "troubleshooting" and not a bug you know. I have seen this bug so many times within logs inside of pfSense. Leading to, software code issues when I find one to me feels like it's a gold mine of educational information. I wanted to let you know I did open a TAC ticket per your request. I am sorry, thanks for being the first to reply to this redmine ticket.

Happy 4th

No offense taken.

Bill Meeks wrote in #note-17:

In the interest of coming to a resolution on this ticket...

The issue identified here is more of a generic problem with outputting the content of large text files (such as logs) using PHP and stateless sessions. It impacts more than just Snort and Suricata. Other packages have similar issues, and even pfSense itself can similarly fail when trying to output large text files via the GUI.

One suggested resolution is for pfSense to have a native PHP module that can statefully read a large text file in chunks and output each chunk separately to the HTML GUI client using a paged output approach perhaps outputting the content into an iframe??? You could maybe save state by passing some type of file position pointer back and forth using AJAX calls with POST. Packages could then call and use this pfSense-provided module when outputting large text files as HTML to the GUI client.

While it may be legit to consider the current behavior a bug, it could also reasonably be classified as a feature request that has a more universal application in pfSense.

I contributed code to the new packet capture page that does AJAX file reads for the live updating of that page. That code could be adapted for the other pages as well, but will take some work to get them adapted for such use.
https://redmine.pfsense.org/issues/13382

Another workaround is the pull request for the option to configure the PHP memory limit in the GUI. https://redmine.pfsense.org/issues/13377 It has a patch file attached that works, but it still needs some polish before it will be merged.

Amazing, thanks for sharing I appreciate you.

[04-Aug-2023 09:30:42 US/Pacific] PHP Fatal error: str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161

With the memory of PHP being set to a larger value this error is still occurring. Per @Christopher Coper this error is not related to PHP memory limit issues, as I have tested his memory software patch to try to resolve this.

Thanks again.

The crash was produced in an attempt to grab the status output file, ticket #1936290053 there are no other PHP errors in the logs

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/

Crash report details:

PHP Errors:
[28-Sep-2023 13:53:22 Asia/Kolkata] PHP Fatal error:  str_ireplace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 161

No FreeBSD crash data found.