pfSense

duplicate of #433, but this has better info so I closed that one.

"traceroute" and "mtr -u" work, but the torrents are also continuing to work.

And now the system log many messages are written:

Jun 3 15:07:11     ipfw-classifyd: sent from divert socket
Jun 3 15:07:11     ipfw-classifyd: received from divert socket

and process ipfw-classifyd utilize 100% 1 CPU:

# top -P

last pid: 50483;  load averages:  1.05,  0.91,  0.52                                                                                    up 0+00:08:21  09:48:53
73 processes:  2 running, 71 sleeping
CPU 0: 31.5% user,  0.0% nice,  0.6% system,  0.0% interrupt, 67.8% idle
CPU 1: 70.3% user,  0.0% nice,  0.0% system,  0.0% interrupt, 29.7% idle
Mem: 46M Active, 11M Inact, 46M Wired, 48K Cache, 19M Buf, 2897M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
14207 root        5  44    0  9200K  5296K kqread  0   7:04 100.00% ipfw-classifyd
31595 root        1  44    0   115M 16812K nanslp  0   0:01  0.00% php
...........

My system:

# uname -a
FreeBSD totoro.pht 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #1: Thu Jun  3 04:05:46 EDT 2010     sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8  i386

You do not have any Found protocol in there?!
And can you please try latest snapshot to know that you have the latest one since i put some more fixes yesterday.

Please a snapshot that is later than this post.
You will find it even because you will not have the repeated errors on the log.

Strange, there can be a port ipfw-classifyd in last snapshot was not refreshed?
The behavior has not varied from the last check.

current system:

Current version: 2.0-BETA2
       Built On: Fri Jun 4 17:07:35 EDT 2010

MD5 (/usr/local/sbin/ipfw-classifyd) = 0133f82bbad75f3cb544a22fd5a458c8

As far as i know it should be ok on latest snapshots.

Can you please before upgrading make sure no ipfw-classifyd process is running?

the md5 shown above matches the binary in the latest snapshot as of now, 20100604-1650.

Ermal Luçi wrote:

As far as i know it should be ok on latest snapshots.

Can you please before upgrading make sure no ipfw-classifyd process is running?

At upgrade process ipfw-classifyd has been running.

I saw that in last commits changes have been made and debug messages are removed.
http://redmine.pfsense.org/projects/pfsense-tools/repository/revisions/master/show/pfPorts/ipfw-classifyd
But just as has told earlier, in the last snapshot, the behaviour has not varied (as in comment 3).

Please try again latest snapshot many things have been fixed and this seems to work as expected now.

trying to enable ipfw-classifyd in this mornings snapshot gives me this output in my system logs :

Jun 10 16:24:31 ipfw-classifyd: Loaded Protocol: ftp (rule action block)
Jun 10 16:24:34 ipfw-classifyd: Something went wrong waiting on kqueue.

Now periodically there are other messages in system.log:

Jun 10 12:30:37 totoro ipfw-classifyd: Loaded Protocol: bittorrent (rule action block)
Jun 10 12:30:41 totoro ipfw-classifyd: Something went wrong waiting on kqueue.
Jun 10 12:31:03 totoro ipfw-classifyd: Something went wrong waiting on kqueue.
Jun 10 12:31:26 totoro ipfw-classifyd: Something went wrong waiting on kqueue.
Jun 10 12:32:28 totoro ipfw-classifyd: Something went wrong waiting on kqueue.

Load on CPU the normal.

But the torrents are also continuing to work, again :)

PF log (for bittorrent packets):

00:00:00.000029 rule 120/0(match): pass in on em0: (tos 0x0, ttl 64, id 24678, offset 0, flags [DF], proto UDP (17), length 93)
    192.168.192.4.6881 > 188.16.58.135.13518: UDP, length 65
00:00:00.000022 rule 120/0(match): pass in on em0: (tos 0x0, ttl 64, id 48415, offset 0, flags [DF], proto UDP (17), length 93)
    192.168.192.4.6881 > 188.17.188.117.13518: UDP, length 65
00:00:00.000253 rule 118/0(match): pass in on em0: (tos 0x0, ttl 64, id 17201, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.192.4.54950 > 212.40.56.138.26691: Flags [S], cksum 0x5157 (correct), seq 3258275224, win 5840, options [mss 1460,sackOK,TS val 123466854 ecr 0,nop,wscale 6], length 0
00:00:00.000289 rule 118/0(match): pass in on em0: (tos 0x0, ttl 64, id 39699, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.192.4.58184 > 212.13.99.157.41635: Flags [S], cksum 0xf4d9 (correct), seq 3267903368, win 5840, options [mss 1460,sackOK,TS val 123466854 ecr 0,nop,wscale 6], length 0
00:00:00.000787 rule 118/0(match): pass in on em0: (tos 0x0, ttl 64, id 48559, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.192.4.42739 > 195.64.255.124.24110: Flags [S], cksum 0x2168 (correct), seq 3254126979, win 5840, options [mss 1460,sackOK,TS val 123466855 ecr 0,nop,wscale 6], length 0

PF rules (118, 120):

@118 pass in log quick on em0 route-to { (em1 33.222.44.1), (rl0 17.119.27.193) } round-robin inet proto tcp from 192.168.192.0/24 to any port 1023 >< 49152 flags S/SA keep state label "USER_RULE: Allow High TCP ports" divert 52226
@120 pass in log quick on em0 route-to { (em1 33.222.44.1), (rl0 17.119.27.193) } round-robin inet proto udp from 192.168.192.0/24 to any port 1023 >< 49152 keep state label "USER_RULE: Allow High UDP ports (Traceroute and MTR)" divert 52226

For the test I have created more simple l7 rule:

Block-FTP -> ftp, action, block

And an appropriate firewall rule:

Rule Proto Source  Port Destination Port     Gateway     layer7
Pass TCP   LAN net *    *           21 - 22  LoadBalance Block-FTP

Too has not received desirable result:

$ ftp ftp.microsoft.com
Connected to ftp.microsoft.akadns.net.
220 Microsoft FTP Service
Name (ftp.microsoft.com:mike): ftp
331 Anonymous access allowed, send identity (e-mail name) as password.
Password:
230-Welcome to FTP.MICROSOFT.COM. Also visit http://www.microsoft.com/downloads.
230 User logged in.
Remote system type is Windows_NT.
ftp> quit
221 Thank you for using Microsoft products.

working processes ipfw-classifyd:

# ps auxww|grep ipfw-
root   14180  0.0  0.2  9200  5512  ??  Ss    8:55AM   0:00.08 /usr/local/sbin/ipfw-classifyd -n 5 -q 700 -c /tmp/Block-P2P.l7 -p 52226 -P /usr/local/share/protocols
root   42089  0.0  0.2  9200  5332  ??  SNs   9:02AM   0:00.02 /usr/local/sbin/ipfw-classifyd -n 5 -q 700 -c /tmp/Block-FTP.l7 -p 40050 -P /usr/local/share/protocols

And divert sockets:

# netstat -lna|grep div
div4       0      0 *.40051                *.*                    
div4       0      0 *.40050                *.*                    
div4       0      0 *.52227                *.*                    
div4       0      0 *.52226                *.*

There are no new ideas of solution of the task?
The error still is not corrected :(

At the big load on the channel, additional reports in a system log began to appear:

ipfw-classifyd: unable to write to divert socket: Invalid argument

I've started to get these today as well. We use the L7 containers.

ipfw-classifyd: Something went wrong waiting on kqueue.

I'm also getting these quite often:

ipfw-classifyd: unable to write to divert socket: No buffer space available

under my side i can see the blocked rules in filter logs but torrent working

This issue has not been resolved, including for me personally (no Layer 7 rules applied), per discussion at http://forum.pfsense.org/index.php?topic=30125

not working for me either. december 2 snapshot i386

Not working for me either. I have tried every combination of firewall rules I can think of as per conversation noted by David.

I committed a change, please test newer snapshots.

Using 2.0-BETA4 (i386) built on Sat Dec 18 09:51:58 EST 2010, I re-applied the Layer 7 rule I'd created before (which was disabled; I enabled it first) to the Default LAN to All rule with no other rules active. I have it set to block HTTP, DNS, and HTML. Web browsing still works from a host on the LAN after this change was saved and applied, so I assume it is still not working, or I am testing incorrectly.

2.0-BETA4 (i386) built on Sun Dec 19 06:36:15 EST 2010

I have tried all my same combination of firewall rules (default LAN, seperate LAN, floating, etc) and it is still failing to block the torrents.

Did you clear states?

Yes, after every firewall rule change I make, I close uTorrent, clear the states, and start uTorrent back up.

Committed another fix can you please try again with newer snapshots.

It will be worth to try even blocking http with bitorrent to see if possibly bittorrent goes on https or the regex is not good enough for most of it.

Just updated to 2.0-BETA4 (i386) built on Thu Dec 23 15:39:33 EST 2010. Created a new L7 group for HTTP. Applied the new L7 group to the default LAN rule. Reset states. HTTP still works.

I committed changes in kernel yet again which should impact even layer7.
Feedback from new snapshot is welcomed.

2.0-BETA5 (i386) built on Wed Jan 5 12:00:59 EST 2011

Still doesn't appear to be working. Opened up the default LAN rule, switched it to TCP/UDP from Any, added my HTTP layer 7 container, and reset the states. I can still browse HTTP no problem.

So when you are committing these changes, the layer 7 stuff is working fine on your box? Maybe we aren't applying it right?

I just tried it with ssh.
I created a new Layer7 group, enabled it, gave it a name, selected ssh as protocol action block.

Then just to make sure I added a floating rule to the top allowing all tcp traffic src any, dst any, ports any, direction out, pass on quick and selected the previously created layer 7 group.
Then I added another rule on top of my lan rules allowing all tcp traffic src any dst any ports any and selected the layer 7 group.

After that I still could open an ssh session to any remote host from the lan side. I didn't find anything regarding layer 7 in the system logs..

Max Riedel wrote:

I just tried it with ssh.
I created a new Layer7 group, enabled it, gave it a name, selected ssh as protocol action block.

Then just to make sure I added a floating rule to the top allowing all tcp traffic src any, dst any, ports any, direction out, pass on quick and selected the previously created layer 7 group.
Then I added another rule on top of my lan rules allowing all tcp traffic src any dst any ports any and selected the layer 7 group.

After that I still could open an ssh session to any remote host from the lan side. I didn't find anything regarding layer 7 in the system logs..

Sorry forgot to mention:

I'm running (full) 2.0-BETA5 (i386)
built on Thu Jan 6 02:48:15 EST 2011

I currently get:
ipfw-classifyd: unable to write to divert socket: No buffer space available

If I have an l7 classification for p2p w/ just bittorrent, /tmp/p2p.l7:
bittorrent = queue qP2P

and I attach this classification to the main TCP/UDP floating rule.

Is there some way to adjust buffer space? Is it a bad idea to have all traffic going through this?

Any update?

Bump.

Does this ticket's status need to be changed to Feedback?

Still not working for me, Clear states and can still browse http. I only have one layer 7 container, and http is selected to block..

I have a rule under my LAN interface for PASS TCP/UDP any any with the layer 7 container. It is the first rule in the list.

I am identical to Adam. Running 2.0-RC1 (i386) built on Wed Mar 2 12:33:12 EST 2011.

I have a L7 container for blocking HTTP. I create a rule on the LAN interface above the default allow rule. I set it to Pass, TCP/UDP, any, any, and apply the HTTP L7 container. Reset states and can still browse HTTP.

Same problem here - Layer 7 filter isn't working :-(

It would be great if dev team could fix that issue for RC2! Please :-)

I also am able to replicate this issue, tested the same as Seth. L7 containers don't block traffic.
Happy to help in any way.

I was testing and getting the same behaviour as you.
Can you please restart your machines before testing if http will get blocked?

Also please apply the patch i last committed and check if it helps.
It makes the max-packets in the generated rule 10 from 5.

It will help on moving this forward.

Please test even with

diff --git a/etc/inc/filter.inc b/etc/inc/filter.inc
index 36e7624..18f2d66 100644
--- a/etc/inc/filter.inc
+++ b/etc/inc/filter.inc
@@ -1881,7 +1881,7 @@ function filter_generate_user_rule($rule) {
                } else
                        $aline['flags'] .= "keep state ";

-               if($noadvoptions == false || $l7_present)
+               if($noadvoptions == false)
                        if( (isset($rule['source-track']) and $rule['source-track'] <> "") or
                            (isset($rule['max']) and $rule['max'] <> "") or
                            (isset($rule['max-src-nodes']) and $rule['max-src-nodes'] <> "") or
@@ -1890,7 +1890,7 @@ function filter_generate_user_rule($rule) {
                            (isset($rule['max-src-conn-rates']) and $rule['max-src-conn-rates'] <> "") or
                            (isset($rule['max-src-states']) and $rule['max-src-states'] <> "") or
                            (isset($rule['statetimeout']) and $rule['statetimeout'] <> "") or
-                           isset($rule['sloppy']) or $l7_present) {
+                           isset($rule['sloppy'])) {
                                        $aline['flags'] .= "( ";
                                        if (isset($rule['sloppy']))
                                                $aline['flags'] .= "sloppy ";
@@ -1913,8 +1913,6 @@ function filter_generate_user_rule($rule) {
                                                $aline['flags'] .= "max-src-conn-rate " . $rule['max-src-conn-rate'] . " ";
                                                $aline['flags'] .= "/" . $rule['max-src-conn-rates'] . ", overload <virusprot> flush global ";
                                        }
-                                       if(!empty($aline['divert']))
-                                               $aline['flags'] .= "max-packets 5 ";

                                        $aline['flags'] .= " ) ";
                                }

Hi there,

I tested it with 2.0-RC1 (i386)
built on Tue Mar 22 16:13:31 EDT 2011

Did a restart and got this error on the filter reload:

Mar 22 20:14:28 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was '/tmp/rules.debug:214: syntax error pfctl: Syntax error in config file: pf rules not loaded'
Mar 22 20:14:28 php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:214: syntax error pfctl: Syntax error in config file: pf rules not loaded The line in question reads [214]: pass in quick on $LAN proto tcp from 192.168.1.0/24 to any divert 49064 flags S/SA keep state ( ) label "USER_RULE: L7test"
Mar 22 20:14:28 php: : There were error(s) loading the rules: /tmp/rules.debug:214: syntax error pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [214]: pass in quick on $LAN proto tcp from 192.168.1.0/24 to any divert 49064 flags S/SA keep state ( ) label "USER_RULE: L7test"

L7test is the rule I applied the L7 http block container to.
Removed the L7 container only and the rules loaded up no prob.

I'll try the code next.

So, same build...

Applied the first 2 changes in your code there, the 3rd seemed to have already been applied in my build..

It loaded the rules with no errors this time, I rebooted, no errors but... no L7 blocking, just like before.

Using 2.0-RC1 (i386) built on Wed Mar 23 20:46:01 EDT 2011, I get the following error when I set one of my LAN rules to use a Layer7 container:

There were error(s) loading the rules: /tmp/rules.debug:286: syntax error/tmp/rules.debug:287: syntax error pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [286]: pass in quick on $LAN_STAFF proto tcp from 192.168.20.0/22 to flags S/SA keep state ( ) label "NEGATE_ROUTE: Negate policy route for vpn(s)" ...

I don't know how to apply that patch, I'm afraid.

I committed that patch and fixed the error above which was accidental breakage by me.
So try latest snapshot with the changes and lets see how it works.

Using 2.0-RC1 (i386) built on Mon Mar 28 00:15:15 EDT 2011, and the error message is now gone (thanks!), but I am still not seeing the layer 7 rules have any effect. I've tried queue and block actions, on BitTorrent and Skype traffic, and none of them appear to be having any effect.

Do you have on your logs any information as 'identified proto(http)' ?

I can't tell. My system log is entirely saturated with "ipfw-classifyd: unable to write to divert socket: No buffer space available"

+1 for this not working.

I have a bridged interface (one WLAN and one LAN) but tried it without the bridged interface and also
with the suggested settings in tunables:

net.link.bridge.pfil_member = 0
net.link.bridge.pfil_bridge = 1

Other Traffic Shaping stuff seems to be applying when I try setting it up in the Wizards.

Am using the Captive Portal with per user bandwidth limits, but switched captive portal off and it doesn't make a difference.

Am on snapshot version:

2.0-RC1 (i386)
built on Sun Apr 3 02:54:42 EDT 2011

As supplied by auto update.

Still not working here either (2.0-RC1 (i386) built on Mon Apr 18 10:01:33 EDT 2011). L7 container set to block HTTP. Have it applied on the very first rule on the LAN interface (Pass, LAN, TCP/UDP, ANY/ANY). Reset the states and tried on two different machines, both can still browse HTTP.

I put a patch yesterday in the layer7 daemon used for classification.
It was forgetting the protocols during reload.

Please test tomorrows snapshots and report back.

I've tested with a snapshot from the evening of the 3rd. It appears to be partially working now. If I create a simple rule to block HTTP, it blocks as expected, so that's good! However I have a more detailed rule that passes traffic to queues and blocks certain traffic... (prioritize Skype, block Bittorrent, etc.) and these rules are not being applied at all. (Bittorrent traffic is not blocked, if I change it to any of my queues it's also not effective at queueing the traffic).

I haven't tried any advance layer 7 rules yet but I do agree with Jonathan that a simple rule to block traffic(I blocked pop3) is working as expected now.

Okay,

On image 2.0-RC1 (i386)
built on Tue May 3 10:51:27 EDT 2011

Confirmed that it works as previous comments in simple config.(Block SSH etc.)

When captive portal is enabled, it stops working, however. (Passes all traffic for an authenticated user, or if they have a MAC pass-through enabled in the CP.)

Rob

@Jonathan,

can you show any picture of your configuration and the system log with the relevant layer7 logs(they should be like 'Found protocol (http)'.

@Rob,

layer7 should not be impacted from CP being on or not.
Possibly you need to check your rules more.

Same as several others. HTTP block is working now, but it still does nothing for BitTorrent.

can you do a packet trace for bittorrent?
You sure its not encrypted?

Keep in mind that these are regex from http://l7-filter.sourceforge.net/protocols

You need to show the generated config file and you have not shown your layer7 from your config.

You have a stray <containter/> tag what happens if you remove that?

I've removed that, and there's no improvement.

Can you upload the generated config file i am not seeing it.

<l7shaper>
<container>
<name>test</name>
<enabled>on</enabled>
<description/>
<divert_port>41744</divert_port>
<l7rules>
<protocol>100bao</protocol>
<structure>queue</structure>
<behaviour>wan</behaviour>
</l7rules>
<l7rules>
<protocol>aim</protocol>
<structure>queue</structure>
<behaviour>qVoIP</behaviour>
</l7rules>
</container>
</l7shaper>

Pretty basic, as you can see. And the system logs still display:

Jun 20 23:49:07 ipfw-classifyd: Loaded Protocol: aim (rule altq)
Jun 20 23:49:07 ipfw-classifyd: Loaded Protocol: 100bao (rule action block)

Hrm i see.
Thank you for the info.
Actually you cannot use the root queue in there and i will try to fix the interface for this.

Can you please test with latest snapshots?

Updated to the latest version, but logging is busted, so I can't say what's happening. (There's a bunch of comments in the forum on this logging issue)

Starting syslog...
/libexec/ld-elf.so.1:
Shared object "libczmq.so.0" not found, required by "syslogd"

Latest version has the same problem.

Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: citrix (rule altq)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: chikka (rule altq)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: audiogalaxy (rule altq)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: ares (rule action block)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: aimwebcontent (rule altq)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: aim (rule action block)
Jul 4 02:15:14 ipfw-classifyd: Loaded Protocol: 100bao (rule action block)

<l7shaper>
        <container>
            <name>layer7shaper</name>
            <enabled>on</enabled>
            <description/>
            <divert_port>46150</divert_port>
            <l7rules>
                <protocol>aimwebcontent</protocol>
                <structure>queue</structure>
                <behaviour>qVoIP</behaviour>
            </l7rules>
            <l7rules>
                <protocol>audiogalaxy</protocol>
                <structure>queue</structure>
                <behaviour>qOthersLow</behaviour>
            </l7rules>
            <l7rules>
                <protocol>citrix</protocol>
                <structure>queue</structure>
                <behaviour>qOthersMed</behaviour>
            </l7rules>
            <l7rules>
                <protocol>chikka</protocol>
                <structure>queue</structure>
                <behaviour>qInternet</behaviour>
            </l7rules>
            <l7rules>
                <protocol>100bao</protocol>
                <structure>queue</structure>
                <behaviour>qACK</behaviour>
            </l7rules>
            <l7rules>
                <protocol>aim</protocol>
                <structure>queue</structure>
                <behaviour>qP2P</behaviour>
            </l7rules>
            <l7rules>
                <protocol>ares</protocol>
                <structure>queue</structure>
                <behaviour>qOthersHigh</behaviour>
            </l7rules>
        </container>
    </l7shaper>

Is there any progress on this? I appreciate your hard work, Ermal.

Try with latest snaps.
It was only a cosmetic issue.

Still seeing the same issue. Using 2.0-RC3 (i386) built on Tue Aug 2 19:07:41 EDT 2011

Try a snapshot from tomorrow since the port was not rebuild it seems.

Hi all,
I tried with version:
2.0-RC1-IPv6 (amd64)
built on Mon Aug 15 22:32:41 EDT 2011

This seems definitely to work very well now.
I tried easy configs with l7 blocking of smtp, ssh and ftp.
It is important to clear the state table before testing the l7 blocking.

Greetings
Peter

Strangely, I am updated to the latest 2.0 RELEASE (nanobsd) and have defined a firewall rule with Layer7 to block
ssh and telnet just as a test, and it does not work still.

It is a bridged interface although I also tried without bridged interface with a blank system and I also couldn't get it to work (still need to try that with 2.0-RELEASE. Other tests were on earlier version.)

Any ideas?

We find it generally works, however adding just a few rules sends our CPU usage to 100%. We're still on one of the RC releases. We'll reinstall on the release soon.