Project

General

Profile

Actions

Bug #12982

open

FreeRadius RadReply table entries missing from pf

Added by Frank Lee 3 months ago. Updated 9 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
FreeRADIUS
Target version:
-
Start date:
03/24/2022
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
2.6.0
Affected Plus Version:
Affected Architecture:
All

Description

It seems when there are too many entries (per user) in the Radreply table (using MySQL) of FreeRadius package, pfsense (or maybe is FreeRadius) will start ignoring the rules. It seems the max entries per user (for Cisco AVPair Rule) is around 70. Pfsense will start ignoring any rules (per user) above 70 or so.

Just to clarify the issue a bit more, for example, in the radreply table, if I have these entries (total of 100 entries) for the user ATMMSOpenVPN,

ATMMSOpenVPN Cisco-AVPair += ip:inacl#1=permit any host 10.7.0.30 host 10.100.0.120
ATMMSOpenVPN Cisco-AVPair += ip:inacl#2=permit any host 10.7.0.30 host 10.100.10.57
ATMMSOpenVPN Cisco-AVPair += ip:inacl#3=permit any host 10.7.0.30 host 10.100.55.22
...
ATMMSOpenVPN Cisco-AVPair += ip:inacl#100=permit any host 10.7.0.30 host 10.100.0.53

1. In entry #1 it is allowing 10.7.0.30 to access 10.100.0.120
2. In entry #100 it is allowing 10.7.0.30 host to access 10.100.0.53

Both 10.100.0.120 and 10.100.0.53 are pingable. But when the user ATMMSOpenVPN login to OpenVPN, he will ONLY be able to ping 10.100.0.120, NOT 10.100.0.53, because 10.100.0.53 is rule #100 (which is above 70)

BUT, if I switch these 2 IP around where now #1 is 10.100.0.53 and #100 is 10.100.0.120 (in the MySQL database), now he will be able tto ping 10.100.0.53 BUT NOT 10.100.0.120 anymore. So almost seems like pfsense only loading the first 70 or so rules and render the rest of the entries (beyind 70 rows) to be "useless".


Files

reply.txt (43.9 KB) reply.txt Marcos Mendoza, 05/01/2022 11:30 AM
_DEBUG1ovpn_ovpns1_ATMMSOpenVPN_54334.txt (5 KB) _DEBUG1ovpn_ovpns1_ATMMSOpenVPN_54334.txt Frank Lee, 05/31/2022 06:06 PM
_DEBUG2ovpn_ovpns1_ATMMSOpenVPN_54334.txt (126 Bytes) _DEBUG2ovpn_ovpns1_ATMMSOpenVPN_54334.txt Frank Lee, 05/31/2022 06:06 PM
Actions #1

Updated by Marcos Mendoza 3 months ago

  • File 562.diff added

There's an MR that changes the way AVPair rules are handled with OpenVPN users (for a different feature request). Would you test the following patch and provide feedback on whether anything breaks/gets fixed after the patch?

Use the System Patches package to apply the patch. After applying the patch, the following commands need to be run (can be done from Diagnostics / Command Prompt):

chmod 755 /usr/local/sbin/openvpn.connect_async.sh
chmod 755 /usr/local/sbin/openvpn_connect_async.php

Actions #2

Updated by Marcos Mendoza 3 months ago

  • Subject changed from FreeRadius RadReply table entries issue (using MySQL) to FreeRadius RadReply table entries missing from pf
  • Description updated (diff)
Actions #3

Updated by Marcos Mendoza 3 months ago

Tested with the patch applied, but the issue remains. Note: the rule syntax originally tested was incorrect - this test is with the correct syntax.

There are 300+ rules for the user, but only 65 get loaded into pf:

[2.6.0-RELEASE][root@pfSense.localdomain]/root: pfSsh.php playback pfanchordrill

ipsec rules/nat contents:

miniupnpd rules/nat contents:

natearly rules/nat contents:

natrules rules/nat contents:

openvpn rules/nat contents:

openvpn/ovpns1_ATMMSOpenVPN_61575 rules/nat contents:
pass in quick on ovpns1 inet from 10.7.0.30 to <__automatic_b3692641_0> flags S/SA keep state

tftp-proxy rules/nat contents:

userrules rules/nat contents:
2.6.0-RELEASE][root@pfSense.localdomain]/root: cat /tmp/ovpn_ovpns1_ATMMSOpenVPN_61575.rules | wc -l
      65

Actions #4

Updated by Jim Pingle 3 months ago

The number that works is too conveniently close to 64 to be a coincidence. It sounds like it's hitting a limit somewhere but it's not clear where. I don't see anything in the RADIUS auth code that would be limited to 64 attributes, but it could be anywhere in that chain. Will need to debug more, including a packet capture of the RADIUS auth exchange to see if more than 64 rules are being sent back in reply attributes. And look in the file that gets the rule content to see if more are there and just not being loaded in the anchor.

Actions #5

Updated by Marcos Mendoza 3 months ago

To add some details from the test:
The file contents did have just 65 rules. I also tried increasing the php max_input_vars (and restarted php-fpm/webconfig) in case that came into play somewhere, but got the same result.

Actions #6

Updated by Frank Lee 2 months ago

  • File LAN_Capture(NoFilter).cap added
  • File WAN_Capture.cap added

Hello,

the support tech told me you guys wanted a packet capture. I assume that you need packet capture between the Radius server (internal radius server) and the MySQL? I'm not sure if I did this right, but here is what I did:

1. I turn on packet capture on LAN with NO FILTER on IP or port.
2. I restarted the OpenVPN server (to kick all user out)
3. Then I waited until the user ATMMSOpenVPN to log on, he is the one with few hundred AVPair rules
4. Then I stop the capture.

Attached is the file. I do see some MySQL packets going back and forth, hopefully this helps.

I've also capture a WAN interface data between the OpenVPN client and pfsense (WAN_Capture.cap).

I was just wondering how long will someone start work on this bug as I need to know if I need to find another product to replace pfSense for this feature.

If you need anything, you can always email me at x or call me on x.

Thank you!

Frank

Actions #7

Updated by Marcos Mendoza about 2 months ago

  • File deleted (WAN_Capture.cap)
Actions #8

Updated by Marcos Mendoza about 2 months ago

  • File deleted (LAN_Capture(NoFilter).cap)
Actions #10

Updated by Marcos Mendoza about 2 months ago

Note: removed potentially sensitive info from comment.

Thanks for the test. I presume 10.1.10.50 is pfSense and 10.1.10.207 is the sql server. It looks like the correct data was sent to pfSense (495 rules for user ATMMSOpenVPN). Hence, the issue seems to lie on the freeradius package or pfSense itself.

Actions #11

Updated by Frank Lee about 2 months ago

Yes, you are correct, 10.1.10.50 is pfSense and 10.1.10.207 is MySQL which provide the table for the FreeRadius on pfsense. It almost like there is some limit set (amount of "rows") that pfsense will allow from the radius server. But like you said, either is the freeradius package limiting that or pfsense itself.

Thank you for looking into this.

Like I said, anytime you want to do more "live" testing or capturing, if you think is easier, you can always contact me.

Actions #12

Updated by Jim Pingle about 2 months ago

I've looked at the RADIUS code we have but didn't see anything that would suggest any kind of limit. I also briefly looked at the underlying RADIUS code from upstream and didn't see anything that looked like it might be imposing a limit, but I didn't dig too deeply in the C code. Though if the rules were all in the RADIUS reply then it has to be hitting a limit and stopping somewhere. It may be a limit of PHP itself but it's hard to say for sure without narrowing it down further.

Actions #13

Updated by Frank Lee about 2 months ago

Hello team,

Any further progress? Sorry keep bugging you guys, but I just need to know, if possible, any rough ETA as to when this problem may get resolved? Because we have customer waiting to get this issue resolve and if we can't do it in a reasonable time frame, I really need to look for another solution. Does netgate have another product that can do this? Or do you guys have any suggestion?

Thank you!

Actions #14

Updated by Frank Lee about 1 month ago

Any new update?

Actions #15

Updated by Marcos Mendoza 30 days ago

  • File debug1.patch added

The issue needs to be narrowed down further. Apply the following patch, reproduce the issue, then submit the /tmp/_DEBUG* files. You can run ls -l /tmp/_DEBUG* from Diagnostics / Command Prompt to view the names, and use the same page to download them.

Actions #16

Updated by Frank Lee 29 days ago

Hello Marcos, I sent you an email on the result, but It seems it is not outputing the information you need. Not sure what is wrong, maybe the way I apply the patch?

Actions #18

Updated by Frank Lee 28 days ago

Hello Marcos, after using your patch, I took the log as requested. It seems it loaded 63 rows and stopped.

Actions #19

Updated by Marcos Mendoza 16 days ago

I can't find where [ciscoavpair] is being set in the code - the only reference I could find was in pear-Auth_RADIUS in the FreeBSD ports. This needs someone with more knowledge than I have.

Actions #20

Updated by Frank Lee 16 days ago

Ok, so do you know roughly when "someone" can look at this issue further?

Actions #21

Updated by Marcos Mendoza 9 days ago

I was able to replicate this with a simpler setup by adding a custom option to the Additional RADIUS Attributes (REPLY-ITEM) field at the bottom of a FreeRADIUS user config entry. For example, add the following up to 10.0.0.99:

Cisco-AVPair += "ip:inacl#1=permit ip host {clientip} host 10.0.0.1",|Cisco-AVPair += "ip:inacl#2=permit ip host {clientip} host 10.0.0.2",|Cisco-AVPair += "ip:inacl#3=permit ip host {clientip} host 10.0.0.3" 

The retrieved ciscoavpair attribute will only contain up to 68 entries. This seems to be an issue with the Auth_RADIUS PHP extension (or perhaps with PHP itself, though I think that's less likely). https://github.com/pear/Auth_RADIUS

For now, I'd recommend finding an alternative to needing more than 68 rules per user.

Actions #22

Updated by Marcos Mendoza 9 days ago

  • File deleted (debug1.patch)
Actions #23

Updated by Marcos Mendoza 9 days ago

  • File deleted (562.diff)
Actions #24

Updated by Frank Lee 9 days ago

So are you saying that pfsense/freeRadius will not be able to go more then 68 rules? any software you know would be able to do this?

Actions

Also available in: Atom PDF