Project

General

Profile

Bug #7954

Package upgrade/reinstall gets stuck on deinstall if the package-provided service is not running

Added by Kill Bill over 2 years ago. Updated over 2 years ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Squid
Target version:
-
Start date:
10/16/2017
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.x
Affected Architecture:
All

Description

So you have a package and the service is not running. Trying to upgrade/reinstall produces the following:

Installed packages to be UPGRADED:
        pfSense-pkg-squid: 0.4.41 -> 0.4.42 [pfSense]

Number of packages to be upgraded: 1

57 KiB to be downloaded.
[1/1] Fetching pfSense-pkg-squid-0.4.42.txz: ........ done
Checking integrity... done (0 conflicting)
[1/1] Upgrading pfSense-pkg-squid from 0.4.41 to 0.4.42...
Extracting pfSense-pkg-squid-0.4.42: .......... done
Removing squid components...
Menu items... done.
Services... Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services...

There, it gets stuck till you kill the /usr/local/bin/php -f /etc/rc.packages pfSense-pkg-squid DEINSTALL process. This is logged by syslog:

Oct 16 22:02:56    php        /etc/rc.packages: The command '/usr/local/etc/rc.d/clamd.sh stop' returned exit code '1', the output was ''
Oct 16 22:02:51    php        /etc/rc.packages: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'squid: ERROR: Could not send signal 15 to process 78563: (3) No such process squid: ERROR: Could not send signal 9 to process 78563: (3) No such process'

Once you've killed the process, the reinstall/upgrade finishes.

Services... done.
Writing configuration... done.
Message from pfSense-pkg-squid-0.4.42:
Please visit Services - Squid Proxy Server menu to configure the package and enable the proxy.
>>> Cleaning up cache... done.

Trying with reinstall instead:

The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        pfSense-pkg-squid-0.4.42 [pfSense]

Number of packages to be reinstalled: 1
[1/1] Reinstalling pfSense-pkg-squid-0.4.42...
[1/1] Extracting pfSense-pkg-squid-0.4.42: .......... done
Removing squid components...
Menu items... done.
Services... 

Zzzzzzzzzzz... Killing the deinstall process makes it move:


Services... Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
Message from pfSense-pkg-squid-0.4.42:
Please visit Services - Squid Proxy Server menu to configure the package and enable the proxy.
>>> Cleaning up cache... done.

In general, the pkg things gets badly confused here, spawning multiple processes for some reason:

$ ps auxww | grep squid
root    72763   0.0  0.1   13084   2604  -  I    22:02     0:00.00 /bin/sh /usr/local/sbin/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f
root    73512   0.0  0.0    6172   1940  -  I    22:02     0:00.00 /usr/bin/lockf -s -t 5 /tmp/pfSense-upgrade.lock /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f
root    73670   0.0  0.1   13084   2756  -  I    22:02     0:00.01 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f
root    82869   0.0  0.1   13084   2756  -  I    22:02     0:00.00 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f
root    82886   0.0  0.1   13084   2756  -  I    22:02     0:00.00 /bin/sh /usr/local/libexec/pfSense-upgrade -y -l /cf/conf/pkg_log_pfSense-pkg-squid.txt -p /tmp/pfSense-upgrade.sock -i pfSense-pkg-squid -f
root    83293   0.0  0.0    8224   1980  -  I    22:02     0:00.00 tee -a /cf/conf/pkg_log_pfSense-pkg-squid.txt
root    83926   0.0  0.1   10264   3912  -  I    22:02     0:00.00 pkg-static -o EVENT_PIPE=/tmp/pfSense-upgrade.sock upgrade -f pfSense-pkg-squid
root    84039   0.0  0.3   26868  13444  -  I    22:02     0:02.15 pkg-static -o EVENT_PIPE=/tmp/pfSense-upgrade.sock upgrade -f pfSense-pkg-squid
root    85417   0.0  0.8  254228  34852  -  I    22:02     0:00.24 /usr/local/bin/php -f /etc/rc.packages pfSense-pkg-squid DEINSTALL

Huh???

History

#1 Updated by Jim Pingle over 2 years ago

  • Status changed from New to Feedback
  • Target version changed from 2.4.1 to 2.4.2

I can't seem to replicate this as-is but there could be something I haven't quite triggered yet.

I setup squid and enabled clamav, so I have squid, clamav, and c-icap services running. I manually stopped the services, then I tried to do a package reinstall.

It does pause for a few moments on the "Services..." line but then it continues on and reinstalls the package as expected, and at the end the services are running again. This was on an 8860 so it is a faster box, maybe that is a factor.

Does that approximate the same test you made? Or did you do something different?

#2 Updated by Kill Bill over 2 years ago

Jim Pingle wrote:

I setup squid and enabled clamav, so I have squid, clamav, and c-icap services running. I manually stopped the services, then I tried to do a package reinstall.
Does that approximate the same test you made?

Well I don't think so when you look at the system log:

Oct 16 22:02:56    php        /etc/rc.packages: The command '/usr/local/etc/rc.d/clamd.sh stop' returned exit code '1', the output was ''
Oct 16 22:02:51    php        /etc/rc.packages: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'squid: ERROR: Could not send signal 15 to process 78563: (3) No such process squid: ERROR: Could not send signal 9 to process 78563: (3) No such process'

So, perhaps try with creating some bogus PID (as if the service crashed and left a stale PID file behind).

Jim Pingle wrote:

It does pause for a few moments on the "Services..." line but then it continues

A few moments would be a severe understatement, didn't move anywhere for 10+ minutes, I don't think waiting would help. (HW: APU2C4, mSATA SSD).

#3 Updated by Jim Pingle over 2 years ago

  • Project changed from pfSense to pfSense Packages
  • Category changed from Package System to Squid
  • Status changed from Feedback to Confirmed
  • Target version deleted (2.4.2)

OK, that does make a difference. If there are stale PID files it seems to get stuck because "/bin/sh /usr/local/etc/rc.d/c-icap.sh stop" gets stuck.

Running it by hand also fails:

: /bin/sh /usr/local/etc/rc.d/c-icap.sh stop
[I Hit ^T]
load: 0.41  cmd: sh 93344 [fifoow] 1.71r 0.00u 0.00s 0% 2568k

The other parts all work, if I kill the PID of just the sh processes trying to run the stop script, it continues.

This appears to be specific to c-icap. The "stop" action in its service script does this:

    /bin/echo -n "stop" > /var/run/c-icap/c-icap.ctl

And if c-icap isn't running, that echo gets blocked trying to write to that fifo. If you run it a few times while c-icap is running, the same thing happens.

Looks like the c-icap sh script needs to be more careful about what it's doing, maybe check pgrep output to see if it's actually running before attempting the echo. Looks like even using timeout won't stop it from blocking on that fifo write, and using & would only make those echos pile up and never die.

#4 Updated by Jim Pingle over 2 years ago

Looks like one viable method might be to echo with &, capture the pid of that process, sleep for a moment, and then kill the pid if it's still running.

#5 Updated by Kill Bill over 2 years ago

Well yes I think there's something broken about c-icap in general, the named pipe (fifo) should vanish once the service has stopped, doesn't appear to be the case so the script removes it manually after stopping the service [1]. So the script actually already does check whether the service is running, it just doesn't work that way.

[1] https://github.com/pfsense/FreeBSD-ports/blob/devel/www/pfSense-pkg-squid/files/usr/local/pkg/squid_antivirus.inc#L714

#6 Updated by Kill Bill over 2 years ago

Jim Pingle wrote:

Looks like one viable method might be to echo with &, capture the pid of that process, sleep for a moment, and then kill the pid if it's still running.

Well, how about this? https://github.com/pfsense/FreeBSD-ports/pull/435

#7 Updated by Jim Pingle over 2 years ago

Sure, that should work fine

Also available in: Atom PDF