Project

General

Profile

Actions
Copy link

Bug #14648

open

Values obtained from ``sysctl`` are sometimes unexpectedly empty, leading to PHP and other math errors

Added by Steve Wheeler 9 months ago. Updated about 2 months ago.

Status:
Confirmed
Priority:
High
Assignee:
-
Category:
Operating System
Target version:
2.8.0
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
24.07
Release Notes:
Default
Affected Version:
2.7.0
Affected Architecture:

Description

In 23.05.1:

PHP Errors:
[16-Jul-2023 19:44:14 Etc/UTC] PHP Fatal error:  Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
  thrown in /etc/inc/util.inc on line 2479
[27-Jul-2023 21:20:37 Etc/UTC] PHP Fatal error:  Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/includes/functions.inc.php(35): get_pfstate()
#3 /usr/local/www/getstats.php(40): get_stats(Array)
#4 {main}
  thrown in /etc/inc/util.inc on line 2479

The system hitting this reports those sysctls correctly;

[23.05.1-RELEASE][suika@pfSense.pfsense.lan]/home/suika: sysctl hw.physmem
hw.physmem: 8288366592
[23.05.1-RELEASE][suika@pfSense.pfsense.lan]/home/suika: sysctl hw.realmem
hw.realmem: 8589934592

Files

Screenshot 2023-08-15 at 9.08.56 PM.png (18.8 KB) Screenshot 2023-08-15 at 9.08.56 PM.png crash report C T, 08/22/2023 02:20 AM

Related issues

Has duplicate Bug #14814: PHP erroDuplicate

Actions
Actions
Copy link
 #1

Updated by Jim Pingle 9 months ago

Normally I'd say we could just change the lines there to cast to int but I'm curious why it fails to automatically convert for them. It implies it was getting back some kind of non-numeric/nonsense value or other bad data at some point.

Even if I do something that would normally break or act weird (like the same operation on 32-bit ARM but with a value > PHP_INT_MAX), it still doesn't generate that PHP error.

That error implies the function would need some other kind of error checking beyond type casting but it's not clear what would be sufficient given that when they run it by hand the data looks OK.

The large gap between the errors implies it isn't happening constantly, too. Since this is coming from the system info dashboard widget code path I'd expect it to happen every time it polls data unless there is some other factor that contributes to it getting back bad data.

Actions
Copy link
 #2

Updated by Michael Clews 9 months ago

I also get similar error:

PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2479, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/includes/functions.inc.php(46): get_pfstate(true)
#3 /usr/local/www/getstats.php(40): get_stats(Array)
#4 {main}

hw.physmem: 16878309376
hw.realmem: 17179869184

It happens randomly. If anything you want me to collect or look for plz let me know. This only started after upgrading to 23.05.1-RELEASE (amd64)

Testing I have done:
1st did remove widgets and add them
- no resolve

2nd tried differ ddr5 memory (went through 3 sticks/ N200 cpu so single channel)
- no resolve

3rd resintalled pfsense on a new NVMe
- Didnt have error for 1 week and just returned this morning.

HW im using:
CPU; N200
Memory: CORSAIR Vengeance SODIMM DDR5 RAM 16GB
mobo: cant tell since its a NUC with 4 x i226 intel ports

Actions
Copy link
 #3

Updated by Jim Pingle 9 months ago

  • Assignee set to Jim Pingle

I doubt it is related to hardware at all, but maybe a timing issue with reading those values from sysctl. It may be hitting some other kind of error condition but it's still not clear what. We can always add more checking/validation here but it would be nice to know what the bad data is exactly. I'll try to work up a patch that will maybe at least log the problematic data when it happens.

Actions
Copy link
 #4

Updated by Christian McDonald 9 months ago

var_dump(""/1000); produces the same error

the empty string does not cleanly cast automatically to an int.

get_single_sysctl returns the empty string

Actions
Copy link
 #5

Updated by Jim Pingle 9 months ago

Yeah that's what I figured but what I can't figure out is why it would ever come back blank for that OID. I can't make that happen here and it isn't clear why it would happen some times and not others.

Actions
Copy link
 #6

Updated by Jim Pingle 9 months ago

  • Subject changed from PHP error: /etc/inc/util.inc:2479 to Intermittent PHP error when calculating memory usage
Actions
Copy link
 #7

Updated by Jim Pingle 9 months ago

I never could reproduce the error condition but I added several safety belts to ensure the values are sane coming out of that function now. Commit is coming momentarily.

Actions
Copy link
 #8

Updated by Jim Pingle 9 months ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100
Actions
Copy link
 #9

Updated by Michael Clews 9 months ago

Got a slightly different variant (havent changed anything):

PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2479, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
thrown @ 2023-08-10 17:39:11

rash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/

Crash report details:

PHP Errors:
[10-Aug-2023 17:39:11 America/New_York] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2479
Stack trace:
#0 /etc/inc/pfsense-utils.inc(2013): get_memory()
#1 /etc/inc/filter.inc(510): pfsense_default_state_size()
#2 /etc/rc.filter_configure_sync(32): filter_configure_sync()
#3 {main}
thrown in /etc/inc/util.inc on line 2479

I have now patched with 054c25418f28bd0afeb1e4a3f07075db76f8f61b.

Actions
Copy link
 #10

Updated by Jim Pingle 9 months ago

Those are the exact same errors as above. You can try the patch above and see if you can reproduce it after.

At this point additional error reports are not necessary unless you can figure out how to reproduce it reliably.

Actions
Copy link
 #11

Updated by Michael Clews 9 months ago

It happens in my case after logging into the system based on the time stamp as its the same time as my login.
not sure if helps but I will update if it occurs again

Actions
Copy link
 #12

Updated by Michael Clews 9 months ago

Hi
I received the error again

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/

Crash report details:

PHP Errors:
[13-Aug-2023 11:26:28 America/New_York] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 219

No FreeBSD crash data found.

Error looks little different than before

Actions
Copy link
 #13

Updated by Jim Pingle 8 months ago

Michael Clews wrote in #note-12:

Hi
I received the error again

Is that with the patch applied or without it?

Actions
Copy link
 #14

Updated by Suika Ibuki 8 months ago

Jim Pingle wrote in #note-13:

Michael Clews wrote in #note-12:

Hi
I received the error again

Is that with the patch applied or without it?

Ok, this happened a week or a bit more after applying the patch 054c25418f28bd0afeb1e4a3f07075db76f8f61b.

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/

Crash report details:

PHP Errors:
[18-Aug-2023 13:21:31 Etc/UTC] PHP Fatal error:  Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
  thrown in /usr/local/www/includes/functions.inc.php on line 219

No FreeBSD crash data found.
Actions
Copy link
 #15

Updated by Jim Pingle 8 months ago

  • Subject changed from Intermittent PHP error when calculating memory usage to Values obtained from ``sysctl`` are sometimes unexpectedly empty, leading to PHP and other math errors
  • Status changed from Feedback to In Progress
  • Priority changed from Normal to High

OK that is in a completely different function, but one which also takes fetches its data from sysctl. Makes no sense how that is coming back empty, that shouldn't be possible, so it's making me wonder what else might be happening there.

I can add some code to check for this case and prevent it from hitting that error, but it doesn't solve the root cause which appears to be the same in both cases. Namely, that the function gathering data from sysctl is somehow failing to obtain data. It's hard to say yet whether it's sysctl itself that has a problem, or something in the function code.

Actions
Copy link
 #16

Updated by Suika Ibuki 8 months ago

I don't even know what is triggering that, something in the background of pfsense does, but dunno how to trigger it.
Why not do a patch against that function to dump everything, env and what not? At least maybe dump the systctl values on error? Since that's what is causing the problem.

Or maybe just give me a small php script that invokes the function that fails. Then I can throw it in a loop, running every second and monitor it for failures.

Actions
Copy link
 #17

Updated by Jim Pingle 8 months ago

  • Category changed from PHP Interpreter to Operating System
  • Assignee deleted (Jim Pingle)

aed18fb07d387c90942b729c02fe460064310f5e should show up on GitHub here in a few minutes with a small fix to avoid that PHP error, but as I said it's no closer to determining the root cause.

I still haven't hit anything like this anywhere in my lab and don't see anything obviously off in the underlying functions fetching info from sysctl, so I'm left to wonder if it's something at the OS level causing sysctl to fail at those times.

Actions
Copy link
 #18

Updated by Jim Pingle 8 months ago

Suika Ibuki wrote in #note-16:

Why not do a patch against that function to dump everything, env and what not? At least maybe dump the systctl values on error? Since that's what is causing the problem.

That could end up generating so much data it might be hard to sort through, but it may come to that.

Or maybe just give me a small php script that invokes the function that fails. Then I can throw it in a loop, running every second and monitor it for failures.

Could start with a simple PHP script that checks this:

var_dump(get_sysctl("hw.physmem"));

Or a shell script that checks:

sysctl -iq hw.physmem

If either of those is empty or 0, then it would help to see:

sysctl -a
Actions
Copy link
 #19

Updated by Michael Clews 8 months ago

Hi

For the last 2 hrs been running script to keep getting that output every 1 second..
It hasn't come up blank or 0 once.

Will keep running for the next week and will update

Actions
Copy link
 #20

Updated by C T 8 months ago

I am repeatedly receiving errors related to this. In addition to errors, crash reports, nearly every day. I just applied aed18fb07d387c90942b729c02fe460064310f5e and will see if any additional errors get through. Please let me know if I can provide any relevant information. Below are some of the errors that I copied out when I first started receiving them and before I added the first patch.

@
PHP errors

PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2409, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2409
    Stack trace:
    #0 /etc/inc/pfsense-utils.inc(1902): get_memory()
    #1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
    #2 /usr/local/www/widgets/widgets/system_information.widget.php(359): get_pfstate(true)
    #3 /usr/local/www/index.php(428): include('/usr/local/www/...')
    #4 {main}
    thrown  2023-08-13 21:02:38
    PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 93, Message: Uncaught ValueError: array_combine(): Argument #1 ($keys) and argument #2 ($values) must have the same number of elements in /usr/local/www/includes/functions.inc.php:93
    Stack trace:
    #0 /usr/local/www/includes/functions.inc.php(93): array_combine(Array, Array)
    #1 /usr/local/www/xmlrpc.php(147) : eval()'d code(46): cpu_usage()
    #2 /usr/local/www/xmlrpc.php(147): eval()
    #3 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
    #4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
    #5 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
    #6 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
    #7 {main}
    thrown  2023-08-13 23:39:27
    PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 2409, Message: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2409
    Stack trace:
    #0 /etc/inc/pfsense-utils.inc(1902): get_memory()
    #1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
    #2 /usr/local/www/xmlrpc.php(147) : eval()'d code(42): get_pfstate()
    #3 /usr/local/www/xmlrpc.php(147): eval()
    #4 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
    #5 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
    #6 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
    #7 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
    #8 {main}
    thrown  2023-08-14 23:01:09
    PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 212, Message: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:212
    Stack trace:
    #0 /usr/local/www/xmlrpc.php(147) : eval()'d code(80): mem_usage()
    #1 /usr/local/www/xmlrpc.php(147): eval()
    #2 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
    #3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
    #4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
    #5 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
    #6 {main}
    thrown  2023-08-15 01:40:36
    PHP ERROR: Type: 1, File: /usr/local/www/includes/functions.inc.php, Line: 212, Message: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:212
    Stack trace:
    #0 /usr/local/www/xmlrpc.php(147) : eval()'d code(80): mem_usage()
#1 /usr/local/www/xmlrpc.php(147): eval()
    #2 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
    #3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
    #4 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
    #5 /usr/local/www/xmlrpc.php(987): XML_RPC2_Backend_Php_Server->handleCall()
    #6 {main}
    thrown  2023-08-15 13:54:46
Actions
Copy link
 #21

Updated by Jim Pingle 7 months ago

  • Status changed from In Progress to Confirmed
  • Plus Target Version changed from 23.09 to 24.01

Pushing this ahead since we still can't replicate this and have no leads about how it's happening.

Actions
Copy link
 #22

Updated by Jim Pingle 7 months ago

  • Plus Target Version changed from 24.01 to 24.03
Actions
Copy link
 #23

Updated by Jim Pingle 7 months ago

Actions
Copy link
 #24

Updated by Michael Clews 7 months ago

Okay,

So I have been running sysctl -iq hw.physmem for every 10 seconds and it has NEVER returned 0 but today i have had my system crash over 12 times:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/

Crash report details:

PHP Errors:
[14-Oct-2023 17:43:32 America/New_York] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:219
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 219

No FreeBSD crash data found.

Is there anyworkaround to this? as to me without functioning workaround all my things stop working during this crash (increase latency)

Actions
Copy link
 #25

Updated by Steve Wheeler 6 months ago

Reviewing this it appears everyone hitting this is running an Intel Nxxx CPU. Is anyone hitting it on something else?

Actions
Copy link
 #26

Updated by Joel Kåberg 6 months ago

Steve Wheeler wrote in #note-25:

Reviewing this it appears everyone hitting this is running an Intel Nxxx CPU. Is anyone hitting it on something else?

Yeah sure, I'm getting these crashes on a Intel(R) Core(TM) i5-6400T CPU

Actions
Copy link
 #27

Updated by Jim Pingle about 2 months ago

  • Plus Target Version changed from 24.03 to 24.07

Still no great leads on this and we still can't reproduce it reliably. Will continue investigating.

Actions
Copy link

Also available in: Atom PDF