Project

General

Profile

Actions

Bug #9277

closed

MBT-4220/2220: pfSense hangs when running sysctl -a

Added by Adam Gibson over 3 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Hardware / Drivers
Target version:
Start date:
01/18/2019
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Default
Affected Version:
2.4.5-p1
Affected Architecture:
amd64

Description

Running 2.4.4-p2 on MBT-4220

Accessing the WebGUI appears to be causing OS-level hang (no response on WebGUI/SSH/PING).
Occurs after login and dashboard visible, then clicking any link. Traffic Graph is visible stoping after a few seconds.

As long as I don't access WebGUI at any point post reboot WAN network access to LAN all machines works fine.

SSH login in appears to be unaffected until WebGUI is accessed.

Restarting WebGUI or PHP-FPM via SSH console (before hang) does not seem to help.

/var/log/system.log appears only the log successful SSH/WebGUI logins prior to hang and syslogd init post restart.

Last few records to Telegraf/InfluxDB suggest the a small spike in User/System process usage:

System: 5%
User: 6%
Idle: 88%


Files

mbt-2220_sysctl-a_output.txt (73.9 KB) mbt-2220_sysctl-a_output.txt Steve Wheeler, 06/26/2020 07:05 PM

Related issues

Related to Bug #10963: Thermal Sensors widget shows invalid sensorsResolvedSteve Wheeler10/07/2020

Actions
Actions #1

Updated by Jim Pingle over 3 years ago

  • Status changed from New to Not a Bug

That isn't a general issue with pfSense or the MBT-4220. Please contact our support team at https://go.netgate.com and someone can help you with that hardware to ensure it's OK.

Actions #2

Updated by Nano Caiordo over 3 years ago

I'm pretty sure I experienced the same issue on 2.4.4-p1 and or 2.4.4-p2.

It did happen only for the initial few reboots while configuring a fresh install.
Login, login message printed on the console, admin could not pass the login page, ssh not accessible, resetting password or restarting php or the webconfigurator did not help.

After config was finalized it did not happen any more, but I cannot confirm this because few days later it start running dev snapshots.

I always believed was due some misconfiguration introduced by me.

Actions #3

Updated by Steve Wheeler almost 2 years ago

It looks like this might be a problem with the way the dashboard system information widget reads the sysctls when you have powerd enabled.

I see exactly this on an MBT-2220 in 2.4.5p1 after I have enabled powerd. It boots and runs fine until I login. The dashboard is show for a few seconds and then it hangs and has to be rebooted.

That same machine cannot display all the sysctls, running 'sysctl -a' also results in a system hang. It does so whether or not powerd is enabled.

Actions #4

Updated by Viktor Gurov about 1 year ago

  • Related to Bug #10963: Thermal Sensors widget shows invalid sensors added
Actions #5

Updated by Max Leighton 12 months ago

We've received additional reports of issues related to this bug report. The behavior may be related to running sysctl -a. A recent user has seen this on MBT-4220 when pulling a status_output from /status.php. The behavior seems to also affect MBT-2220. There isn't a panic or crash, the system will just hang. status.php?archiveonly didn't trigger the behavior.

Actions #6

Updated by Steve Wheeler 11 months ago

  • Subject changed from MBT-4220: PFSense OS hangs when accessing WebGUI to MBT-4220/2220: pfSense hangs when running sysctl -a
  • Category changed from FreeBSD to Hardware / Drivers
  • Target version set to 2.6.0

This was difficult to pin-down because it only stops responding if the HDMI console is not connected at the time the sysctls are read.

It stops responding at:

hw.dri.0.info.i915_gen6_forcewake_count_info: forcewake count = 0

hw.dri.0.info.i915_context_status: 
hw.dri.0.info.i915_gem_framebuffer: fbcon size: 1024 x 768, depth 24, 32 bpp, obj 0xfffff800054a7600K: p      3072KiB 0041 0000 0 0 0 uncached (pinned x 1) (display) (gtt offset: 0006a000, size: 00300000) (p mappable)

hw.dri.0.info.i915_sr_status: self-refresh: disabled

hw.dri.0.info.i915_fbc_status: FBC unsupported on this chipset

hw.dri.0.info.i915_gfxec: GFXEC: 4294967295

hw.dri.0.info.i915_ring_freq_table: GPU freq (MHz)    Effective CPU freq (MHz)
0        0

Actions #7

Updated by Steve Wheeler 11 months ago

There appear to be two specific sysctls that cause the system to stop responding:

hw.dri.0.info.i915_drpc_info
and
hw.dri.0.info.i915_cur_delayinfo

Again that only happens if HDMI is not connected when they are first read.

Actions #8

Updated by Jim Pingle 11 months ago

Does it still crash if you don't load the i915 module?

Actions #9

Updated by Steve Wheeler 11 months ago

No. Those OIDs don't exist to be read if the i915 module is not loaded:

[2.5.2-RC][admin@2220.stevew.lan]/root: sysctl hw.dri.0
sysctl: unknown oid 'hw.dri.0'

Actions #10

Updated by Mateusz Guzik 4 months ago

  • Status changed from New to Resolved
  • Assignee set to Mateusz Guzik

I committed a patch to hide the problematic sysctls when running sysctl -a, which should be good enough for the time being.

Actions #11

Updated by Steve Wheeler 4 months ago

  • Status changed from Resolved to Feedback

Tested this against 2.6 beta:

[2.6.0-BETA][admin@4220.stevew.lan]/root: kldstat
Id Refs Address                Size Name
 1   29 0xffffffff80200000  3aed878 kernel
 2    1 0xffffffff83cee000    e0b68 i915kms.ko
 3    2 0xffffffff83dcf000     4bf0 iicbb.ko
 4    5 0xffffffff83dd4000     9c28 iicbus.ko
 5    2 0xffffffff83dde000     2f70 iic.ko
 6    2 0xffffffff83de1000    71220 drm2.ko
 7    1 0xffffffff84111000     1000 cpuctl.ko
 8    1 0xffffffff84112000     8e10 aesni.ko
 9    1 0xffffffff8411b000      bf8 coretemp.ko
[2.6.0-BETA][admin@4220.stevew.lan]/root: uname -a
FreeBSD 4220.stevew.lan 12.3-STABLE FreeBSD 12.3-STABLE i915-n226733-6d2f07003239 pfSense  amd64

sysctl -a no longer hangs. Calling the offending sysctls dircetly still does but that should never be done outside testing.

Actions #12

Updated by Jim Pingle 4 months ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF