Bug #14648
open
Values obtained from ``sysctl`` are sometimes unexpectedly empty, leading to PHP and other math errors
Added by Steve Wheeler 9 months ago. Updated 2 months ago.
100%
Description
In 23.05.1:
PHP Errors:
[16-Jul-2023 19:44:14 Etc/UTC] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
thrown in /etc/inc/util.inc on line 2479
[27-Jul-2023 21:20:37 Etc/UTC] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/includes/functions.inc.php(35): get_pfstate()
#3 /usr/local/www/getstats.php(40): get_stats(Array)
#4 {main}
thrown in /etc/inc/util.inc on line 2479
The system hitting this reports those sysctls correctly;
[23.05.1-RELEASE][suika@pfSense.pfsense.lan]/home/suika: sysctl hw.physmem hw.physmem: 8288366592 [23.05.1-RELEASE][suika@pfSense.pfsense.lan]/home/suika: sysctl hw.realmem hw.realmem: 8589934592
Files
| Screenshot 2023-08-15 at 9.08.56 PM.png (18.8 KB) Screenshot 2023-08-15 at 9.08.56 PM.png | crash report | C T, 08/22/2023 02:20 AM |
Related issues
Updated by Jim Pingle 9 months ago
Normally I'd say we could just change the lines there to cast to int but I'm curious why it fails to automatically convert for them. It implies it was getting back some kind of non-numeric/nonsense value or other bad data at some point.
Even if I do something that would normally break or act weird (like the same operation on 32-bit ARM but with a value > PHP_INT_MAX), it still doesn't generate that PHP error.
That error implies the function would need some other kind of error checking beyond type casting but it's not clear what would be sufficient given that when they run it by hand the data looks OK.
The large gap between the errors implies it isn't happening constantly, too. Since this is coming from the system info dashboard widget code path I'd expect it to happen every time it polls data unless there is some other factor that contributes to it getting back bad data.
Updated by Michael Clews 9 months ago
I also get similar error:
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2479, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/includes/functions.inc.php(46): get_pfstate(true)
#3 /usr/local/www/getstats.php(40): get_stats(Array)
#4 {main}
hw.physmem: 16878309376
hw.realmem: 17179869184
It happens randomly. If anything you want me to collect or look for plz let me know. This only started after upgrading to 23.05.1-RELEASE (amd64)
Testing I have done:
1st did remove widgets and add them
- no resolve
2nd tried differ ddr5 memory (went through 3 sticks/ N200 cpu so single channel)
- no resolve
3rd resintalled pfsense on a new NVMe
- Didnt have error for 1 week and just returned this morning.
HW im using:
CPU; N200
Memory: CORSAIR Vengeance SODIMM DDR5 RAM 16GB
mobo: cant tell since its a NUC with 4 x i226 intel ports
Updated by Jim Pingle 9 months ago
- Assignee set to Jim Pingle
I doubt it is related to hardware at all, but maybe a timing issue with reading those values from sysctl. It may be hitting some other kind of error condition but it's still not clear what. We can always add more checking/validation here but it would be nice to know what the bad data is exactly. I'll try to work up a patch that will maybe at least log the problematic data when it happens.
Updated by Christian McDonald 9 months ago
var_dump(""/1000); produces the same error
the empty string does not cleanly cast automatically to an int.
get_single_sysctl returns the empty string
Updated by Jim Pingle 9 months ago
Yeah that's what I figured but what I can't figure out is why it would ever come back blank for that OID. I can't make that happen here and it isn't clear why it would happen some times and not others.
Updated by Jim Pingle 9 months ago
- Subject changed from PHP error: /etc/inc/util.inc:2479 to Intermittent PHP error when calculating memory usage
Updated by Jim Pingle 9 months ago
I never could reproduce the error condition but I added several safety belts to ensure the values are sane coming out of that function now. Commit is coming momentarily.
Updated by Jim Pingle 9 months ago
- Status changed from New to Feedback
- % Done changed from 0 to 100
Applied in changeset 054c25418f28bd0afeb1e4a3f07075db76f8f61b.
Updated by Michael Clews 9 months ago
Got a slightly different variant (havent changed anything):
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2479, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
thrown @ 2023-08-10 17:39:11
rash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
Crash report details:
PHP Errors:
[10-Aug-2023 17:39:11 America/New_York] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
thrown in /etc/inc/util.inc on line 2479
I have now patched with 054c25418f28bd0afeb1e4a3f07075db76f8f61b.
Updated by Jim Pingle 9 months ago
Those are the exact same errors as above. You can try the patch above and see if you can reproduce it after.
At this point additional error reports are not necessary unless you can figure out how to reproduce it reliably.
Updated by Michael Clews 9 months ago
It happens in my case after logging into the system based on the time stamp as its the same time as my login.
not sure if helps but I will update if it occurs again
Updated by Michael Clews 9 months ago
Hi
I received the error again
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
Crash report details:
PHP Errors:
[13-Aug-2023 11:26:28 America/New_York] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 219
No FreeBSD crash data found.
Error looks little different than before
Updated by Jim Pingle 9 months ago
Michael Clews wrote in #note-12:
Hi
I received the error again
Is that with the patch applied or without it?
Updated by Suika Ibuki 9 months ago
Jim Pingle wrote in #note-13:
Michael Clews wrote in #note-12:
Hi
I received the error againIs that with the patch applied or without it?
Ok, this happened a week or a bit more after applying the patch 054c25418f28bd0afeb1e4a3f07075db76f8f61b.
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
Crash report details:
PHP Errors:
[18-Aug-2023 13:21:31 Etc/UTC] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 219
No FreeBSD crash data found.
Updated by Jim Pingle 9 months ago
- Subject changed from Intermittent PHP error when calculating memory usage to Values obtained from ``sysctl`` are sometimes unexpectedly empty, leading to PHP and other math errors
- Status changed from Feedback to In Progress
- Priority changed from Normal to High
OK that is in a completely different function, but one which also takes fetches its data from sysctl. Makes no sense how that is coming back empty, that shouldn't be possible, so it's making me wonder what else might be happening there.
I can add some code to check for this case and prevent it from hitting that error, but it doesn't solve the root cause which appears to be the same in both cases. Namely, that the function gathering data from sysctl is somehow failing to obtain data. It's hard to say yet whether it's sysctl itself that has a problem, or something in the function code.
Updated by Suika Ibuki 9 months ago
I don't even know what is triggering that, something in the background of pfsense does, but dunno how to trigger it.
Why not do a patch against that function to dump everything, env and what not? At least maybe dump the systctl values on error? Since that's what is causing the problem.
Or maybe just give me a small php script that invokes the function that fails. Then I can throw it in a loop, running every second and monitor it for failures.
Updated by Jim Pingle 9 months ago
- Category changed from PHP Interpreter to Operating System
- Assignee deleted (
Jim Pingle)
aed18fb07d387c90942b729c02fe460064310f5e should show up on GitHub here in a few minutes with a small fix to avoid that PHP error, but as I said it's no closer to determining the root cause.
I still haven't hit anything like this anywhere in my lab and don't see anything obviously off in the underlying functions fetching info from sysctl, so I'm left to wonder if it's something at the OS level causing sysctl to fail at those times.
Updated by Jim Pingle 9 months ago
Suika Ibuki wrote in #note-16:
Why not do a patch against that function to dump everything, env and what not? At least maybe dump the systctl values on error? Since that's what is causing the problem.
That could end up generating so much data it might be hard to sort through, but it may come to that.
Or maybe just give me a small php script that invokes the function that fails. Then I can throw it in a loop, running every second and monitor it for failures.
Could start with a simple PHP script that checks this:
var_dump(get_sysctl("hw.physmem"));
Or a shell script that checks:
sysctl -iq hw.physmem
If either of those is empty or 0, then it would help to see:
sysctl -a
Updated by Michael Clews 9 months ago
Hi
For the last 2 hrs been running script to keep getting that output every 1 second..
It hasn't come up blank or 0 once.
Will keep running for the next week and will update
Updated by C T 9 months ago
I am repeatedly receiving errors related to this. In addition to errors, crash reports, nearly every day. I just applied aed18fb07d387c90942b729c02fe460064310f5e and will see if any additional errors get through. Please let me know if I can provide any relevant information. Below are some of the errors that I copied out when I first started receiving them and before I added the first patch.
@
PHP errors
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2409, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2409
Stack trace:
#0 /etc/inc/pfsense-utils.inc(1902): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/widgets/widgets/system_information.widget.php(359): get_pfstate(true)
#3 /usr/local/www/index.php(428): include('/usr/local/www/...')
#4 {main}
thrown 2023-08-13 21:02:38
PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 93, Message: Uncaught ValueError: array_combine(): Argument #1 ($keys) and argument #2 ($values) must have the same number of elements in /usr/local/www/includes/functions.inc.php:93
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(93): array_combine(Array, Array)
#1 /usr/local/www/xmlrpc.php(147) : eval()'d code(46): cpu_usage()
#2 /usr/local/www/xmlrpc.php(147): eval()
#3 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#5 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#6 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
#7 {main}
thrown 2023-08-13 23:39:27
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2409, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2409
Stack trace:
#0 /etc/inc/pfsense-utils.inc(1902): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/xmlrpc.php(147) : eval()'d code(42): get_pfstate()
#3 /usr/local/www/xmlrpc.php(147): eval()
#4 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#5 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#6 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#7 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
#8 {main}
thrown 2023-08-14 23:01:09
PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 212, Message: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:212
Stack trace:
#0 /usr/local/www/xmlrpc.php(147) : eval()'d code(80): mem_usage()
#1 /usr/local/www/xmlrpc.php(147): eval()
#2 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#5 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
#6 {main}
thrown 2023-08-15 01:40:36
PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 212, Message: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:212
Stack trace:
#0 /usr/local/www/xmlrpc.php(147) : eval()'d code(80): mem_usage()#1 /usr/local/www/xmlrpc.php(147): eval()
#2 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#5 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
#6 {main}
thrown 2023-08-15 13:54:46
Updated by Jim Pingle 8 months ago
- Status changed from In Progress to Confirmed
- Plus Target Version changed from 23.09 to 24.01
Pushing this ahead since we still can't replicate this and have no leads about how it's happening.
Updated by Jim Pingle 8 months ago
- Plus Target Version changed from 24.01 to 24.03
Updated by Michael Clews 7 months ago
Okay,
So I have been running sysctl -iq hw.physmem for every 10 seconds and it has NEVER returned 0 but today i have had my system crash over 12 times:
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
Crash report details:
PHP Errors:
[14-Oct-2023 17:43:32 America/New_York] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 219
No FreeBSD crash data found.
Is there anyworkaround to this? as to me without functioning workaround all my things stop working during this crash (increase latency)
Updated by Steve Wheeler 7 months ago
Reviewing this it appears everyone hitting this is running an Intel Nxxx CPU. Is anyone hitting it on something else?
Updated by Joel Kåberg 6 months ago
Steve Wheeler wrote in #note-25:
Reviewing this it appears everyone hitting this is running an Intel Nxxx CPU. Is anyone hitting it on something else?
Yeah sure, I'm getting these crashes on a Intel(R) Core(TM) i5-6400T CPU
Updated by Jim Pingle 2 months ago
- Plus Target Version changed from 24.03 to 24.07
Still no great leads on this and we still can't reproduce it reliably. Will continue investigating.