Project

General

Profile

Actions

Bug #3951

closed

Processes like filterdns and ipfw-classifyd accumulate many open file handles

Added by Phillip Davis about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Category:
Operating System
Target version:
Start date:
10/21/2014
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.1.x
Affected Architecture:

Description

Forum: https://forum.pfsense.org/index.php?topic=63357.msg453040#msg453040
and surrounding discussion and results in that thread.
e.g. I have a 2.1.5 production system that has been up for only 7 days and has:
249 openvpn
1535 php
1861 filterdns
as the top 3 offenders for open files.

The magic command to see the open file counts is:
fstat | awk '\!/CMD/{print $2}' | sort | uniq -c | sort -n

filterdns is under control of pfSense project, so it should be easy to check this and see why the open file count keeps increasing.
ipfw-classifyd is the other main offender reported in the thread - I don't use that so do not have an example.
I am surprised that my "php" on 2.1.5 has 1535 things open - why could that be?
And openvpn looks a bit high to believe, but maybe it does that, I have a bunch of site-to-site servers, road warrior server...

Anyway, IMHO this is worth investigating, and particularly making sure it does not happen on 2.2


Files

fstat_out.txt (275 KB) fstat_out.txt Petr Klus, 10/21/2014 03:18 PM
filterdns-fstat-22B-2014-11-02.txt (7.79 KB) filterdns-fstat-22B-2014-11-02.txt Phillip Davis, 11/02/2014 05:46 AM
filterdns-fstat-215-2014-11-02.txt (256 KB) filterdns-fstat-215-2014-11-02.txt Phillip Davis, 11/02/2014 05:46 AM
filterdns-fstat-22B-2014-11-02a.txt (1.39 KB) filterdns-fstat-22B-2014-11-02a.txt Phillip Davis, 11/02/2014 08:58 AM
Actions #1

Updated by Petr Klus about 7 years ago

My case: pfSense 2.1.5 running on i386 embedded version. Clean install, only modification apart from UI is password file for OpenVPN client connection.

I saw the open files grow to an extend that DNS resolver and DHCP server stopped working. I've tracked down the growth to gateway alarm, which caused reload of the rules.

I had a dodgy monitor IP with short alarm period, which meant that the connection alarm was triggered on/off every 10-15m. This means that every 10-15m, I would see the number of open files grow. Even my system with very few aliases locked up in a week's time.

I have worked-around the issue by disabling the gateway alarms on the VPN connection (I was not using them to trigger anything anyway) and now my system seems more stable!

Actions #2

Updated by Chris Buechler about 7 years ago

could anyone attach the full output of fstat from an affected system? Spot checked some using filterdns and they seem sane.

Actions #3

Updated by Petr Klus about 7 years ago

Here it is!

Actions #4

Updated by Chris Buechler about 7 years ago

  • Assignee set to Chris Buechler
  • Target version set to 2.2
Actions #5

Updated by Ermal Luçi about 7 years ago

I think i found the cause.
Please test with new snapshots.

Actions #6

Updated by Ermal Luçi about 7 years ago

  • Status changed from New to Feedback
Actions #7

Updated by Phillip Davis about 7 years ago

My main 2.1.5 production system is the big offender with this - it has over 4000 in filterdns fstat. But I can't upgrade that to test. A sample of its fstat output is in filterdns-fstat-215-2014-11-02.txt
I added an alias with 15 FQDNs in (one I regularly use) to my test 2.2 system. The filterdns fstat output grows every 5minutes or so, i.r. every time filterdns wakes up and does its checks - sample is in attached filterdns-fstat-22B-2014-11-02.txt
Now I am upgrading the 2.2 test system to the latest snapshot. Will report back in a few hours how it goes.

Actions #8

Updated by Phillip Davis about 7 years ago

Updated to:
2.2-BETA (amd64)
built on Sat Nov 01 21:36:28 CDT 2014
FreeBSD 10.1-RC4

Now filterdns has just 8 things open after starting up the first time, and after 45 minutes uptime there are still only 8 things open.
For the record, some fstat output is attached - looks good.

The filterdns issue seems to be fixed.

@Petr - are you able to test a 2.2 system with ipfw-classifyd and see if that exhibits similar issues to your 2.1.5 system?
If it does, then perhaps a separate issue should be raised for that, and this one can be closed - as there are probably 2 completely separate issues here.

@Eraml and . . - thanks for looking at this in a timely manner - it is good to cleanup any issues that leak system resources.

Actions #9

Updated by Ermal Luçi about 7 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF