Bug #1534
closed
rc.newwanip issues (CARP slave problems, package issues)
0%
Description
Quoted from here http://forum.pfsense.org/index.php/topic,36802.0.html
I've mentioned some rc.d problems in other thread but decided to create new one as I've debugged problem a little more. Feel free to merge threads if you think they should be merged :-)
The bugs are...
1. Function restart_packages() from rc.newwanip doesn't do restart properly
2. Packages aren't stopped on system reboot/halt
3. Backup router forks lots of php causing OOM error and system failureNow... rc.newwanip has function restart_packages which calls rc.start_packages (in background). Restarting packages should first stop them waiting for processes to finish and then start them again. Unfortunately calling "stop" was removed from rc.start_packages long long time ago and placed in rc.stop_packages. Its fine, but rc.stop_packages is never called or grep is cheating me :-) Because of that few thing doesn't work as they should.
1. User scripts may missbehave, ie. I've placed in /usr/localt/etc/rc.d simple script that launches tcpdump into background to log some traffic. Script has proper handling of start, stop and restart commands. After reboot I have lots and lots instances of tcpdump running because stop wasn't called before start when exectued from restart_packages().
2. System services aren't restarted. Calling just "start" to restart service won't work if it checks if its already running. Thus no restart ever occurs for such services.
3. Services aren't properly stopped on system reboot/halt. That means processes are simply killed at some point. This may lead to data corruption depending on services used and how they handle signals.
4. Any service related operation (start/stop/restart) shoudn't be lanuched into background. If there are many interfaces used in system rc.start_packages is called in background many times. Even with proper implementation of start/stop it won't work because script processes will interfere with each other (one stopping, second starting, third stopping and so on).Here is little patch that fixed bugs #1 and #2: http://pastebin.com/ARVfDvLs
Finally bug #3 about which I wrote in other thread, but back then I didn't knew reason of such behavior. Now after little research I know exactly what happens. My setup has total of 51 network interfaces reported by ifconfig. After each click on "Apply changes" in any part of pfSense web configurator on my master router following things happen on backup router:
1. /usr/local/sbin/check_reload_status is run
2. for each network interface /etc/rc.newwanip is called, by now we have 51 php processes spawned
3. each copy of /etc/rc.newwanip calls /etc/rc.start_packages shell script, we have 51 of those too but they quickly disappear because...
4. each copy of /etc/rc.start_packages executes each *.sh file from /usr/local/etc/rc.d in background so each *.sh script is spawned 51 times almost simultaneouslyLets say I'm working on adding aliases, rules, configuring VPNs etc. and lets say I'll click "Apply changes" 5 times in 5 minutes. I'll have 255 php processes which will launch each script from /usr/local/etc/rc.d 255 times. Usually thats enough to hog C2D CPU, eat 4 gigs of ram, 8 gigs of swap effectively killing machine.
I'm wondering if sources of check_reload_status are available somewhere? In worst case (no fix in mainstream pfSense) I'll be able to fix this problem on my own.
Files
Updated by Ermal Luçi over 13 years ago
Can you please test with latest snapshot from tomorrow and let me know.
I have done some fixes that should prevent this.
Updated by Marcin Krol over 13 years ago
Can you please test with latest snapshot from tomorrow and let me know.
I have done some fixes that should prevent this.
Sure. I did update today but changes from diffs were not applied on my install. Probably because in meantime I switched to IPv6 build. I'm currently running:
2.0-RC1-IPv6 (amd64)
built on Thu May 19 22:21:03 EDT 2011
Will these changes be available for IPv6 builds too or do I need to add them by hand?
Updated by Marcin Krol over 13 years ago
Sure. I did update today
Err... before weekend :-)
Updated by Jim Pingle over 13 years ago
Did you adjust your gitsync URL to point at the github location? The IPv6 tree is up-to-date with 2.0 mainline right now, if you are behind, your gitsync URL is probably still pointing at the old gitweb.pfsense.org/rcs.pfsense.org location.
Updated by Marcin Krol over 13 years ago
Indeed I was using old gitsync URL. I've performed few tests yesterday on updated systems. Unfortunately applied changes have not fixed issues, but its a little bit better. I'm now running 2.0-RC2-IPv6 (amd64) built on Tue May 24 04:45:10 EDT 2011
First thing - rc scripts:
1. Still there is no "stop" issued when system is going down.
2. There is also no "stop" issued before "start" in rc.start_packages.
2. rc.newwanip still launches "start" into background.
As previously I had to create little patch to get things starting/stopping correctly (attached).
Also problem with backup router still exists. I've clicked "edit" -> "save" -> "apply changes" on 10 firewall rules (without modifying them). It took me less than 1 minute. That caused over 200 php processes on backup router. Whole ram was eaten and about half of the swap. Router recovered to normal cpu/memory/swap usage in about 20 minutes, but during that time it was hardly usable (web configurator timing out, lagged shell).
Updated by Marcin Krol over 13 years ago
Disabling and re-enabling HAVP on master router causes even more severe load on backup router. It also causes other isssue, but I'll report it in separate ticket.
Updated by Marcin Krol over 13 years ago
I just updated my backup router, but commits from 5 days ago are not included yet. I'm now running:
2.0-RC2-IPv6 (amd64)
built on Tue May 31 12:13:03 EDT 2011
and my gitsync URL points to github.
Updated by Jim Pingle over 13 years ago
That is because you are on the IPv6 branch which hadn't been merged in a while. I just synced it back up with mainline so it should be up to date now, either by a gitsync or your next firmware update it should be ok.
Updated by Marcin Krol over 13 years ago
I've updated yesterday and now I'm running 2.0-RC2-IPv6 (amd64) built on Wed Jun 1 18:03:37 EDT 2011 and last commits are included in this version. It seems that packages start/stop related problems are all gone, however slave router is still being killed by forking dozens of "/usr/local/bin/php /etc/rc.filter_configure_sync" processes on every "Apply changes" I do in firewall aliases or rules on master router.
Updated by Jim Pingle over 13 years ago
How many interfaces / VIPs do you have on that box? When I apply changes on my VM, I only see 1-2 of those and they immediately exit. Something about your config must be triggering more syncs, but without more information it's hard to say what.
Updated by Marcin Krol over 13 years ago
19 VIPs, 20 permanent OpenVPN tunnels, 1 OpenVPN for users, 2 gigabit NICs aggregated into 1 lagg0 interface, 14 VLANs defined on lagg0 for 2 WAN links and 12 LAN segments, 1 dedicated 100Mbit NIC for CARP link between master and slave router. Grand total is 52 network interfaces reported with ifconfig -a.
Updated by Ermal Luçi over 13 years ago
Latest snapshot has more fixes in this regard.
Updated by Marcin Krol over 13 years ago
After upgrade I'm now on 2.0-RC2-IPv6 (amd64) built on Fri Jun 10 01:43:14 EDT 2011 and its even more broken. Whatever changes I'll make on master router they are not synchronized to backup router. I tried forced config sync on master and it finished successully, but changes are still not visible on backup router. Even after reboot backup router still doesn't see changes.
Updated by Jim Pingle over 13 years ago
That is a known issue. Fixed in the next snapshot (building now) try again with that.
Updated by Marcin Krol over 13 years ago
I've upgraded to 2.0-RC2-IPv6 (amd64) built on Sat Jun 11 23:14:29 EDT 2011. Had severe problems with this update. After reboot my slave router claimed to be master and I ended up with two masters fighting for VIPs. I finally managed to get it working but some other issues have emerged in the meantime. For example after adding host alias with single IP I'm getting error on both routers saying that firewall was unable to load filter rules due to "invalid port 192.168.1.15". HAVP stopped working on backup router and doesn't start at all with no errors in logs, dhcpd started logging following error many many times on both routers "dhcpd: failover: listener: no matching state"...
So far network seems to be working and when I'm changing something on master router backup one is no longer flooded with php process, but I'm not sure if bug was fixed or simply sync doesn't occur correctly due to errors mentioned above. I also hope that next upgrade won't totally break my installation :)
Updated by Ermal Luçi over 13 years ago
I do not know the state of the IPv6 code but can you try with latest snapshot of pfSense since there were some binary updates in base for which i am unsure the status in the IPv6 snapshots.
Updated by Marcin Krol over 13 years ago
Unfortuntely I can't drop IPv6 support on my machines. I'm currently running 2.0-RC3-IPv6 (amd64) built on Mon Jun 27 06:02:37 EDT 2011 and it seems that php forking problem is gone. Only few php processes appear on backup router and they quickly disappear.
Updated by Chris Buechler over 13 years ago
- Status changed from Feedback to Resolved