Project

General

Profile

Actions

Bug #4344

closed

package (re)installation loop after upgrading from 2.1.5-RELEASE to 2.2-RELEASE

Added by Vinícius Zavam over 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
High
Assignee:
-
Category:
Upgrade
Target version:
-
Start date:
01/29/2015
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
Affected Architecture:
amd64

Description

this issue was originally reported by "WolfSec-Support" <support at wolfsec.ch> on the pfSense's mailing list.

how to reproduce?

packages related to this issue
  • squid
  • lightsquid
  • open-vm-tools
  • cron
  • autoconfigbackup
  • mtr-nox11
  • sudo
  • suricata
  • nmap
  • iperf
system logs? high priority?
  • as suricata is not working after the upgrade, i decided to tag it as 'high priority'.
about the images (screenshots)
  • warning_packages_being_reinstalled: the warning you will see after the first login after the upgrade, right on your dashboard.
  • services_suricata: shows a page where the suricata package/service should live. suricata can't be enabled/started.
misc
  • WolfSec tells that you may see some "crashes" and some upgrades weren't able to bring the system back and running again. some of the WolfSec's systems were running on a VMware ESXi 5.1 and just one, running on ALIX, did the upgrade with success. there's another reported system with the same issue running on baremetal (physical machine).

Files

warning_packages_being_reinstalled.png (29.2 KB) warning_packages_being_reinstalled.png Vinícius Zavam, 01/29/2015 11:54 AM
services_suricata.png (55.7 KB) services_suricata.png Vinícius Zavam, 01/29/2015 11:54 AM
Actions #1

Updated by Vinícius Zavam over 9 years ago

http://pastebin.com/8ni6F2Tb was scheduled to expire in 2 weeks, so... here's its content:

Jan 29 09:35:45     SuricataStartup[85541]: Suricata START for DMZ(44312_re0)...
Jan 29 04:34:06     root: rc.update_bogons.sh is ending the update cycle.
Jan 29 04:34:06     root: Bogons V6 file downloaded: 56040 addresses added.
Jan 29 04:34:06     root: Bogons V4 file downloaded: 48 addresses deleted.
Jan 29 04:34:06     root: Bogons V4 file downloaded: 33 addresses added.
Jan 29 04:34:01     root: rc.update_bogons.sh is beginning the update cycle.
Jan 29 03:01:00     root: rc.update_bogons.sh is sleeping for 5581
Jan 29 03:01:00     root: rc.update_bogons.sh is starting up.
Jan 29 01:37:06     check_reload_status: Syncing firewall
Jan 29 01:37:05     php: suricata_check_for_rule_updates.php: [Suricata] The Rules update has finished.
Jan 29 01:37:05     php: suricata_check_for_rule_updates.php: [Suricata] Building new sid-msg.map file for DMZ...
Jan 29 01:37:05     php: suricata_check_for_rule_updates.php: [Suricata] Enabling any flowbit-required rules for: DMZ...
Jan 29 01:37:04     php: suricata_check_for_rule_updates.php: [Suricata] Updating rules configuration for: DMZ ...
Jan 29 01:37:03     php: suricata_check_for_rule_updates.php: [Suricata] Emerging Threats Open rules file update downloaded successfully
Jan 29 01:37:00     php: suricata_check_for_rule_updates.php: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Jan 28 15:50:59     SuricataStartup[51885]: Suricata START for DMZ(44312_re0)...
Jan 28 15:50:59     check_reload_status: Syncing firewall
Jan 28 15:50:59     check_reload_status: Syncing firewall
Jan 28 15:50:58     php-fpm[17653]: /rc.start_packages: Checked cron job for /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/suricata/suricata_check_cron_misc.inc, no change needed
Jan 28 15:50:58     kernel: done.
Jan 28 15:50:58     php-fpm[17653]: /rc.start_packages: Restarting/Starting all packages.
Jan 28 15:50:58     kernel: Starting CRON...
Jan 28 15:50:54     kernel: 100%
Jan 28 15:50:54     kernel: 90%
Jan 28 15:50:53     kernel: 80%
Jan 28 15:50:53     kernel: 70%
Jan 28 15:50:53     kernel: 60%
Jan 28 15:50:52     kernel: 50%
Jan 28 15:50:52     kernel: 40%
Jan 28 15:50:51     kernel: 30%
Jan 28 15:50:51     kernel: 20%
Jan 28 15:50:50     kernel: 10%
Jan 28 15:50:50     kernel: 7%
Jan 28 15:50:50     kernel: 3%
Jan 28 15:50:50     kernel: 2%
Jan 28 15:50:49     kernel: 0%
Jan 28 15:50:48     kernel:
Jan 28 15:50:48     check_reload_status: Syncing firewall
Jan 28 15:50:48     kernel: 100%
Jan 28 15:50:47     php: rc.bootup: Beginning package installation for nmap .
Jan 28 15:50:47     php: rc.bootup: Reinstalling package nmap
Jan 28 15:50:47     php: rc.bootup: Finished uninstalling package nmap
Jan 28 15:50:47     check_reload_status: Syncing firewall
Jan 28 15:50:46     check_reload_status: Syncing firewall
Jan 28 15:50:37     php: rc.bootup: Uninstalling package nmap
Jan 28 15:50:37     kernel: Done.
Jan 28 15:50:37     php: rc.bootup: List of packages to reinstall: nmap, mtr-nox11, sudo, suricata

Suricata still doesn't work!

Shell Access

vinicius.zavam at egypcio:~ % ssh -6 -l noc -p 22085 2001:aaaa:bbbb:cccc::
Last login: Wed Jan 28 16:26:47 2015 from 2001:aaaa:bbbb:cccc::fb5d
export: Command not found.
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(1) id
uid=2000(noc) gid=65534(nobody) groups=65534(nobody),1999(admins)
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(2) su -
su: Sorry
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(3) sudo su -
Shared object "libutil.so.8" not found, required by "sudo" 
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(4) pkg info
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]:
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(5) ls -1 /var/db/pkg
bsdinstaller-2.0.2012.1207
gettext-0.18.1.1
iperf-2.0.5 
libiconv-1.14
libpcap-1.2.1
lua-5.1.5_4
mtr-nox11-0.82
nmap-6.01
openssl-1.0.1_10
pcre-8.30_2
pkg-config-0.25_1
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(6) nmap --version

Nmap version 6.47 ( http://nmap.org )
Platform: amd64-portbld-freebsd10.0
Compiled with: liblua-5.2.3 openssl-1.0.1g-freebsd libpcre-8.35
libpcap-1.4.0 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: kqueue poll select
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(7) mtr --version
mtr 0.85
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(8) iperf
iperf: Command not found.
[2.2-RELEASE][noc at hostname.dn.tld]/home/noc(9) whereis iperf
whereis: Command not found.

PS: iperf shouldn't be here, I think. I've deinstalled it long time ago.
PS2: a quick test scan with nmap was okay. mtr-nox11 is also in good shape.

Actions #2

Updated by Vinícius Zavam over 9 years ago

Piba-NL's (##pfsense@freenode) pointed out that would be nice to try this procedures:
  • clear installation's lock;
  • reinstall all packages.

the 'unlock' and 'reinstallation' features are available through Diagnostics > Backup/Restore.

https://github.com/pfsense/pfsense/blob/RELENG_2_1/usr/local/www/diag_backup.php#L763

Actions #3

Updated by Vinícius Zavam over 9 years ago

Vinícius Zavam wrote:

Piba-NL's (##pfsense@freenode) pointed out that would be nice to try this procedures:
  • clear installation's lock;
  • reinstall all packages.

the 'unlock' and 'reinstallation' features are available through Diagnostics > Backup/Restore.

https://github.com/pfsense/pfsense/blob/RELENG_2_1/usr/local/www/diag_backup.php#L763

it works!

Actions #4

Updated by Vinícius Zavam over 9 years ago

FYI
there's another package that was reported as unable to be 100% reinstalled after the upgrade procedure from 2.1.5-RELEASE to 2.2-RELEASE.
"zabbix_agentd" needed manual (re)installation/upgrade.

Actions #5

Updated by Chris Buechler over 8 years ago

  • Status changed from New to Closed
  • Affected Version deleted (2.2)

It's not a loop, it can get stuck in the process because of issues in package code hanging up PHP. 2.3 changed the package reinstall process to use a separate PHP instance so bad package code can't hang up the entire reinstall process.

Actions

Also available in: Atom PDF