Project

General

Profile

Actions

Bug #6177

closed

pkg update checking with no Internet access kills web GUI

Added by Chris Buechler over 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
High
Category:
Operating System
Target version:
Start date:
04/15/2016
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.3
Affected Architecture:

Description

When a system can't reach the Internet and is checking for updates, pkg processes start piling up, eventually killing the GUI (nginx returning 504 because php-fpm doesn't return).

pkg processes like the following start piling up:

root    25840   0.0  1.5  45144  7268  -  I     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root    25984   0.0  1.6  45144  7676  -  S     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root    63921   0.0  1.5  45144  7328  -  S     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root    64216   0.0  1.6  45144  7640  -  S     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root    66597   0.0  1.5  45144  7328  -  S     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root    66628   0.0  1.6  45144  7640  -  S     2:43AM 0:00.00 /usr/sbin/pkg search --raw-format json-compact pfSense-base

It appears these processes never timeout.


Files

pfsense.PNG (82.7 KB) pfsense.PNG Jan Jurkus, 04/23/2016 05:36 PM
Actions #1

Updated by robi robi over 8 years ago

A quick suggestion for the routine which starts re-installing packages after the first reboot following the update:
- there should be a test/check of connectivity with pfsense packages servers before trying to download anything.
- test should be able to pass through the preconfigured proxy if that's the case, so pinging the server would not be appropriate. Perhaps downloading a dummy/test-file from the server(s) would assume that there is a connection available to packages servers
- If no connection is available, print something useful to the user, and continue booting normally
- also put something on the TTY console and the Web Dashboard, that package re-installation didn't succeed - and let the user the chance to trigger package reinstall later, when connection is made available.

Actions #2

Updated by Jan Jurkus over 8 years ago

Well, it's not only the GUI that wasn't working any more. I could not get an IP with DHCP from pfSense. Look at the load average in the attached screenshot.

After a night's sleep it was still broken, so I opted to restart pfSense from the console. It would not complete that, and apparently it kept waiting for something. I did not know how to return to a command line and kill any remaining processes, so I reset the thing. Luckily everything worked afterwards.

As a sidenote: it says 'retrieving overview data' under ipsec on the dashboard. Didn't this just show up instantly in 2.2? The ntp status on the dashboard also showed a large error about a gateway, something I was unable to make a screenshot of.

Actions #3

Updated by Renato Botelho over 8 years ago

  • Status changed from Confirmed to Feedback
  • % Done changed from 0 to 100

GUI also fixed by sbeaver, should be fine now

Actions #4

Updated by Chris Buechler over 8 years ago

  • Status changed from Feedback to Confirmed

What's been done so far all works and definitely helps.

But, pkg processes will still pile up and hang the GUI, if you refresh dashboard a few times. Then a 'killall pkg' brings it back immediately.

pkg_update probably should skip the pkg_call if it's already running.

Actions #5

Updated by Dominic S over 8 years ago

Disabling the 'automatic dashboard auto-update check' at /system_update_settings.php seems to mitigate the issue for now.

Actions #6

Updated by Anonymous over 8 years ago

Which widgets are enabled on you dashboard?

Actions #7

Updated by Dominic S over 8 years ago

Steve Beaver wrote:

Which widgets are enabled on you dashboard?

- System Information
- Interfaces
- Firewall Logs
- Snort Alerts

Actions #8

Updated by Renato Botelho over 8 years ago

  • Status changed from Confirmed to Feedback
Actions #9

Updated by Chris Buechler over 8 years ago

  • Status changed from Feedback to Resolved

all fixed now

Actions #10

Updated by Glenn Provoost over 8 years ago

this fixed my issue on 2.3, thanks

Actions #11

Updated by Nicola Bressan over 8 years ago

hello,
I've experienced a similar issue.

pfSense 2.3.1_5
IPv6 tunnel configured

pfSense in GUI was checking for updates and was resolving pkg.pfsense.org as IPv6 address (AAAA record) and couldn't check for updates.
This caused issues and 100% cpu for pkg process.

Checking " System > Advanced on the Networking tab > Prefer IPv4 over IPv6" was a good workaround, but this indicates that there's still that 100% cpu bug when pkg.pfsense.org is not responding correctly...right?

can you have a look in it?
or maybe fix IPv6 answering of pkg.pfsense.org :)

Actions #12

Updated by Chris Buechler over 8 years ago

Nicola Bressan wrote:

I've experienced a similar issue.

can you have a look in it?
or maybe fix IPv6 answering of pkg.pfsense.org :)

IPv6 works just fine on pkg.pfsense.org. You're not hitting the issue here, please start a forum thread to discuss.

Actions #13

Updated by Nicola Bressan over 8 years ago

Chris Buechler wrote:

IPv6 works just fine on pkg.pfsense.org. You're not hitting the issue here, please start a forum thread to discuss.

ok I'll do it, but the fact that with IPv6 preference enabled, package update was not working correctly it's a matter of fact.

When I don't enable "Prefer to use IPv4 even if IPv6 is available" I get 100% CPU usage for pkg search:


2.3.1-RELEASE][root@pfSense.nbr.local]/root: ps aux | grep pkg
root    31701 93.0  1.8  45172  8980  -  R     2:26PM    0:44.03 /usr/sbin/pkg search --raw-format json-compact pfSense-base
root     7399  0.0  6.7 225160 32624  -  S     9:19PM    0:39.61 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
root    31683  0.0  1.6  45172  7624  -  I     2:26PM    0:00.01 /usr/sbin/pkg search --raw-format json-compact pfSense-base

last pid: 19104;  load averages:  0.91,  0.48,  0.25                                                                                  up 0+17:10:31  14:28:17
47 processes:  4 running, 43 sleeping
CPU: 68.2% user,  0.0% nice, 31.8% system,  0.0% interrupt,  0.0% idle
Mem: 19M Active, 127M Inact, 103M Wired, 38M Buf, 214M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
31701 root          1 102    0 45172K  8980K RUN      2:00 100.00% pkg
35155 unbound       1  20    0 38604K 13856K kqread   0:42   0.00% unbound

This is what's happening:

with IPv6 [before hitting Google to show it's working...]

[2.3.1-RELEASE][root@pfSense.nbr.local]/root: fetch -v -6 https://www.google.it
looking up www.google.it
connecting to www.google.it:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-ECDSA-AES128-GCM-SHA256
Certificate subject: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
Certificate issuer: /C=US/O=Google Inc/CN=Google Internet Authority G2
requesting https://www.google.it/
fetch: https://www.google.it: size of remote file is not known
www.google.it                                           10 kB  863 kBps 00m00s

Now the test:
[2.3.1-RELEASE][root@pfSense.nbr.local]/root: fetch -v -6 https://pkg.pfsense.org/pfSense_v2_3_1_amd64-core/meta.txz
looking up pkg.pfsense.org
connecting to pkg.pfsense.org:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem

and it hangs here...

over IPv4 it works:

[2.3.1-RELEASE][root@pfSense.nbr.local]/root: fetch -v -4 https://pkg.pfsense.org/pfSense_v2_3_1_amd64-core/meta.txz
looking up pkg.pfsense.org
connecting to pkg.pfsense.org:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES256-GCM-SHA384
Certificate subject: /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.pfsense.org
Certificate issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
requesting https://pkg.pfsense.org/pfSense_v2_3_1_amd64-core/meta.txz
remote size / mtime: 944 / 1466102812
meta.txz                                      100% of  944  B 2265 kBps 00m00s
[2.3.1-RELEASE][root@pfSense.nbr.local]/root:

the problem that arises is always the same...when pkg update/fetching is not getting done correctly, the GUI hang with 100% cpu power being collected by the pkg process.

Actions #14

Updated by Nicola Bressan over 8 years ago

edit

Actions #15

Updated by Nicola Bressan over 8 years ago

edit.

Actions #16

Updated by José Jorge over 7 years ago

I can just say I have the same bug with 2.3.2 version. Forcing Ipv4 as indicated in comment 13 worked around this bug.

Actions

Also available in: Atom PDF