Bug #7954
open
Package upgrade/reinstall gets stuck on deinstall if the package-provided service is not running
0%
Description
So you have a package and the service is not running. Trying to upgrade/reinstall produces the following:
Installed packages to be UPGRADED:
pfSense-pkg-squid: 0.4.41 -> 0.4.42 [pfSense]
Number of packages to be upgraded: 1
57 KiB to be downloaded.
[1/1] Fetching pfSense-pkg-squid-0.4.42.txz: ........ done
Checking integrity... done (0 conflicting)
[1/1] Upgrading pfSense-pkg-squid from 0.4.41 to 0.4.42...
Extracting pfSense-pkg-squid-0.4.42: .......... done
Removing squid components...
Menu items... done.
Services... Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services...
There, it gets stuck till you kill the /usr/local/bin/php -f /etc/rc.packages pfSense-pkg-squid DEINSTALL process. This is logged by syslog:
Oct 16 22:02:56 php /etc/rc.packages: The command '/usr/local/etc/rc.d/clamd.sh stop' returned exit code '1', the output was '' Oct 16 22:02:51 php /etc/rc.packages: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'squid: ERROR: Could not send signal 15 to process 78563: (3) No such process squid: ERROR: Could not send signal 9 to process 78563: (3) No such process'
Once you've killed the process, the reinstall/upgrade finishes.
Services... done. Writing configuration... done. Message from pfSense-pkg-squid-0.4.42: Please visit Services - Squid Proxy Server menu to configure the package and enable the proxy. >>> Cleaning up cache... done.
Trying with reinstall instead:
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be REINSTALLED:
pfSense-pkg-squid-0.4.42 [pfSense]
Number of packages to be reinstalled: 1
[1/1] Reinstalling pfSense-pkg-squid-0.4.42...
[1/1] Extracting pfSense-pkg-squid-0.4.42: .......... done
Removing squid components...
Menu items... done.
Services...
Zzzzzzzzzzz... Killing the deinstall process makes it move:
Services... Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Executing custom_php_resync_config_command()...done. Menu items... done. Services... done. Writing configuration... done. Message from pfSense-pkg-squid-0.4.42: Please visit Services - Squid Proxy Server menu to configure the package and enable the proxy. >>> Cleaning up cache... done.
In general, the pkg things gets badly confused here, spawning multiple processes for some reason:
$ ps auxww | grep squid root 72763 0.0 0.1 13084 2604 - I 22:02 0:00.00 /bin/sh /usr/local/sbin/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f root 73512 0.0 0.0 6172 1940 - I 22:02 0:00.00 /usr/bin/lockf -s -t 5 /tmp/pfSense-upgrade.lock /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f root 73670 0.0 0.1 13084 2756 - I 22:02 0:00.01 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f root 82869 0.0 0.1 13084 2756 - I 22:02 0:00.00 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f root 82886 0.0 0.1 13084 2756 - I 22:02 0:00.00 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f root 83293 0.0 0.0 8224 1980 - I 22:02 0:00.00 tee -a /cf/conf/pkg_log_pfSense-pkg-squid.txt root 83926 0.0 0.1 10264 3912 - I 22:02 0:00.00 pkg-static -o EVENT_PIPE=/tmp/pfSense-upgrade.sock upgrade -f pfSense-pkg-squid root 84039 0.0 0.3 26868 13444 - I 22:02 0:02.15 pkg-static -o EVENT_PIPE=/tmp/pfSense-upgrade.sock upgrade -f pfSense-pkg-squid root 85417 0.0 0.8 254228 34852 - I 22:02 0:00.24 /usr/local/bin/php -f /etc/rc.packages pfSense-pkg-squid DEINSTALL
Huh???
Updated by Jim Pingle over 6 years ago
- Status changed from New to Feedback
- Target version changed from 2.4.1 to 2.4.2
I can't seem to replicate this as-is but there could be something I haven't quite triggered yet.
I setup squid and enabled clamav, so I have squid, clamav, and c-icap services running. I manually stopped the services, then I tried to do a package reinstall.
It does pause for a few moments on the "Services..." line but then it continues on and reinstalls the package as expected, and at the end the services are running again. This was on an 8860 so it is a faster box, maybe that is a factor.
Does that approximate the same test you made? Or did you do something different?
Updated by Kill Bill over 6 years ago
Jim Pingle wrote:
I setup squid and enabled clamav, so I have squid, clamav, and c-icap services running. I manually stopped the services, then I tried to do a package reinstall.
Does that approximate the same test you made?
Well I don't think so when you look at the system log:
Oct 16 22:02:56 php /etc/rc.packages: The command '/usr/local/etc/rc.d/clamd.sh stop' returned exit code '1', the output was '' Oct 16 22:02:51 php /etc/rc.packages: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'squid: ERROR: Could not send signal 15 to process 78563: (3) No such process squid: ERROR: Could not send signal 9 to process 78563: (3) No such process'
So, perhaps try with creating some bogus PID (as if the service crashed and left a stale PID file behind).
Jim Pingle wrote:
It does pause for a few moments on the "Services..." line but then it continues
A few moments would be a severe understatement, didn't move anywhere for 10+ minutes, I don't think waiting would help. (HW: APU2C4, mSATA SSD).
Updated by Jim Pingle over 6 years ago
- Project changed from pfSense to pfSense Packages
- Category changed from Package System to Squid
- Status changed from Feedback to Confirmed
- Target version deleted (
2.4.2)
OK, that does make a difference. If there are stale PID files it seems to get stuck because "/bin/sh /usr/local/etc/rc.d/c-icap.sh stop" gets stuck.
Running it by hand also fails:
: /bin/sh /usr/local/etc/rc.d/c-icap.sh stop [I Hit ^T] load: 0.41 cmd: sh 93344 [fifoow] 1.71r 0.00u 0.00s 0% 2568k
The other parts all work, if I kill the PID of just the sh processes trying to run the stop script, it continues.
This appears to be specific to c-icap. The "stop" action in its service script does this:
/bin/echo -n "stop" > /var/run/c-icap/c-icap.ctl
And if c-icap isn't running, that echo gets blocked trying to write to that fifo. If you run it a few times while c-icap is running, the same thing happens.
Looks like the c-icap sh script needs to be more careful about what it's doing, maybe check pgrep output to see if it's actually running before attempting the echo. Looks like even using timeout won't stop it from blocking on that fifo write, and using & would only make those echos pile up and never die.
Updated by Jim Pingle over 6 years ago
Looks like one viable method might be to echo with &, capture the pid of that process, sleep for a moment, and then kill the pid if it's still running.
Updated by Kill Bill over 6 years ago
Well yes I think there's something broken about c-icap in general, the named pipe (fifo) should vanish once the service has stopped, doesn't appear to be the case so the script removes it manually after stopping the service [1]. So the script actually already does check whether the service is running, it just doesn't work that way.
Updated by Kill Bill over 6 years ago
Jim Pingle wrote:
Looks like one viable method might be to echo with
&, capture the pid of that process, sleep for a moment, and then kill the pid if it's still running.
Well, how about this? https://github.com/pfsense/FreeBSD-ports/pull/435