Project

General

Profile

Bug #6001

Not all packages are started during boot time

Added by Greg M about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Category:
Package System
Target version:
Start date:
03/15/2016
Due date:
% Done:

100%

Estimated time:
Affected Version:
2.3
Affected Architecture:
All

Description

Hi!

In snapshots from (I think so) 11. march some packages are not started durin boot/reboot.
Example is Lightsquid and Avahi.
See here: https://forum.pfsense.org/index.php?topic=108148.0

When I edited /etc/rc.start_packages and changed "if (time() - $stat['mtime'] >= 90) {" to "if (time() - $stat['mtime'] >= 5) {" problem was resolved.

There is one more thing I noticed...
When all was working just before console shown up, there was text displayed:
Starting package "packagename" for all packages installed.
Now there is no more and I think it was removed somwhere in the proccess.

Funny is that I have 2 affected and 1 not affected systems so it`s not common.

I can provide more logs if needed.

History

#1 Updated by Chris Buechler about 3 years ago

what's in your /tmp/bootup_messages file?

#2 Updated by Greg M about 3 years ago

I have no such file.
But I can see .rc.start_packages.running file which shouldnt be there right?

#3 Updated by Greg M about 3 years ago

After I manually run rc.start_packages file is created and it has content:

2016-03-15 22:59:48: (network.c.416) can't bind to port: 0.0.0.0 7445 Address already in use
But lightsquid is started and working after this manual intervention.
Also file .rc.start_packages.running is gone.
So it seems that .sh is not running correctly...

#4 Updated by Dmitriy K about 3 years ago

Squid does not start after boot too. Not only Squid.
https://forum.pfsense.org/index.php?topic=108425

#5 Updated by Dmitriy K about 3 years ago

2016/03/17 10:14:00 kid1| /var/run/squid/access.log: (2) No such file or directory

/tmp/cache.log
2016/03/17 10:24:58 kid1| /var/run/squid/squid.pid: (2) No such file or directory

Looks like links issue

#6 Updated by Greg M about 3 years ago

BTW I guess package startup gets interrupted because after boot (like 10 minutes after) there is still /tmp/.rc.start_packages.running file present...
It should be gone but it`s not.
After further testing I think that pppoe might interrupt start process somehow but I`m not sure.

@Dmitriy K:
You have different issue not related to this one.

#7 Updated by Greg M about 3 years ago

And found cause.

So I have snort running on 3 ifaces. It takes a long time to start it up. It starts before lightsquid and because of that lightsquid never starts. I disabled snort on all 3 ifaces, rebooted and lightsquid is up and running.

#8 Updated by Chris Buechler about 3 years ago

  • Status changed from New to Not a Bug
  • Affected Version deleted (2.3)

Doesn't appear there is any general problem with package startup (that will be addressed with the status quo of package startup at least). The issue of one package taking ages to startup causing issues for other packages will be addressed by switching to FreeBSD's stock package rc.d in the future. Dmitriy's squid issue is probably something else entirely, please follow up on your forum thread re: that.

#9 Updated by Greg M about 3 years ago

Ok, I understand. But still 10 days ago all was started just fine I don't know what changed, maybe package startup was moved to earlier point in boot time.

Is there a way to change package startup order? So that snort or other heavy packages start last?

EDIT:
Found workaround that works, see: https://forum.pfsense.org/index.php?topic=108148.msg604474#msg604474

#10 Updated by Chad Wagner about 3 years ago

I was not having issues until the same snapshot timeframe as Greg. The workaround Greg mentioned works for Lightsquid but Avahi is still not starting automatically. I am not using snort. Just Squid, Lightsquid, Avahi and openvpn.
Chad

#11 Updated by Jim Pingle about 3 years ago

  • Status changed from Not a Bug to New
  • Assignee set to Chris Buechler
  • Target version set to 2.3
  • Affected Version set to 2.3

I finally managed to reproduce this in a 100% reliable fashion. In my test case, WAN is static, WAN2 is DHCP. If WAN2 is enabled, the rc.newwanip call that happens during the boot sequence disrupts the package restart progress, and some services end up not running. If WAN2 is disabled, all of the expected services start.

Chris is digging into it, he's got access to the test system in question.

#12 Updated by Chris Buechler about 3 years ago

  • Status changed from New to Confirmed
  • Assignee changed from Chris Buechler to Marc Dye
  • Priority changed from Normal to High
  • Affected Architecture changed from amd64 to All

Marc's going to pick up on this. We've discussed it at length.

TLDR version: The issue is when rc.start_packages runs a second time (and exits because it's already running), it somehow stops further processing of this array in the first rc.start_packages instance.
https://github.com/pfsense/pfsense/blob/master/src/etc/rc.start_packages#L65

#13 Updated by Chris Buechler about 3 years ago

  • Assignee changed from Marc Dye to Chris Buechler

#14 Updated by Renato Botelho about 3 years ago

  • Status changed from Confirmed to Feedback
  • Assignee changed from Chris Buechler to Renato Botelho
  • % Done changed from 0 to 100

Fixed in check_reload_status 0.0.7

#15 Updated by Chris Buechler about 3 years ago

Looks to be fixed. The 3 cases I had, and one of JimP's, all work fine now. Same root cause likely fixed every other possible combination that was a problem previously.

Others who were seeing issues, working now once you're up to latest?

#16 Updated by Chad Wagner about 3 years ago

All of my services are starting now as well.

#17 Updated by Greg M about 3 years ago

Same here, all OK.

Thanks!

#18 Updated by Renato Botelho about 3 years ago

  • Status changed from Feedback to Resolved

Fixed

Also available in: Atom PDF