Bug #3951
closed
Processes like filterdns and ipfw-classifyd accumulate many open file handles
Added by Phillip Davis over 9 years ago.
Updated over 9 years ago.
Category:
Operating System
Description
Forum: https://forum.pfsense.org/index.php?topic=63357.msg453040#msg453040
and surrounding discussion and results in that thread.
e.g. I have a 2.1.5 production system that has been up for only 7 days and has:
249 openvpn
1535 php
1861 filterdns
as the top 3 offenders for open files.
The magic command to see the open file counts is:
fstat | awk '\!/CMD/{print $2}' | sort | uniq -c | sort -n
filterdns is under control of pfSense project, so it should be easy to check this and see why the open file count keeps increasing.
ipfw-classifyd is the other main offender reported in the thread - I don't use that so do not have an example.
I am surprised that my "php" on 2.1.5 has 1535 things open - why could that be?
And openvpn looks a bit high to believe, but maybe it does that, I have a bunch of site-to-site servers, road warrior server...
Anyway, IMHO this is worth investigating, and particularly making sure it does not happen on 2.2
Files
My case: pfSense 2.1.5 running on i386 embedded version. Clean install, only modification apart from UI is password file for OpenVPN client connection.
I saw the open files grow to an extend that DNS resolver and DHCP server stopped working. I've tracked down the growth to gateway alarm, which caused reload of the rules.
I had a dodgy monitor IP with short alarm period, which meant that the connection alarm was triggered on/off every 10-15m. This means that every 10-15m, I would see the number of open files grow. Even my system with very few aliases locked up in a week's time.
I have worked-around the issue by disabling the gateway alarms on the VPN connection (I was not using them to trigger anything anyway) and now my system seems more stable!
could anyone attach the full output of fstat from an affected system? Spot checked some using filterdns and they seem sane.
- Assignee set to Chris Buechler
- Target version set to 2.2
I think i found the cause.
Please test with new snapshots.
- Status changed from New to Feedback
My main 2.1.5 production system is the big offender with this - it has over 4000 in filterdns fstat. But I can't upgrade that to test. A sample of its fstat output is in filterdns-fstat-215-2014-11-02.txt
I added an alias with 15 FQDNs in (one I regularly use) to my test 2.2 system. The filterdns fstat output grows every 5minutes or so, i.r. every time filterdns wakes up and does its checks - sample is in attached filterdns-fstat-22B-2014-11-02.txt
Now I am upgrading the 2.2 test system to the latest snapshot. Will report back in a few hours how it goes.
Updated to:
2.2-BETA (amd64)
built on Sat Nov 01 21:36:28 CDT 2014
FreeBSD 10.1-RC4
Now filterdns has just 8 things open after starting up the first time, and after 45 minutes uptime there are still only 8 things open.
For the record, some fstat output is attached - looks good.
The filterdns issue seems to be fixed.
@Petr - are you able to test a 2.2 system with ipfw-classifyd and see if that exhibits similar issues to your 2.1.5 system?
If it does, then perhaps a separate issue should be raised for that, and this one can be closed - as there are probably 2 completely separate issues here.
@Eraml and . . - thanks for looking at this in a timely manner - it is good to cleanup any issues that leak system resources.
- Status changed from Feedback to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by Ermal Luçi 
Updated by