Project

General

Profile

Actions

Feature #12982

closed

Add support for RFC7499 in RADIUS library.

Added by Frank Lee almost 2 years ago. Updated 11 months ago.

Status:
Rejected
Priority:
Normal
Category:
Operating System
Target version:
-
Start date:
03/24/2022
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Default

Description

It seems when there are too many entries (per user) in the Radreply table (using MySQL) of FreeRadius package, pfsense (or maybe is FreeRadius) will start ignoring the rules. It seems the max entries per user (for Cisco AVPair Rule) is around 70. Pfsense will start ignoring any rules (per user) above 70 or so.

Just to clarify the issue a bit more, for example, in the radreply table, if I have these entries (total of 100 entries) for the user ATMMSOpenVPN,

ATMMSOpenVPN Cisco-AVPair += ip:inacl#1=permit any host 10.7.0.30 host 10.100.0.120
ATMMSOpenVPN Cisco-AVPair += ip:inacl#2=permit any host 10.7.0.30 host 10.100.10.57
ATMMSOpenVPN Cisco-AVPair += ip:inacl#3=permit any host 10.7.0.30 host 10.100.55.22
...
ATMMSOpenVPN Cisco-AVPair += ip:inacl#100=permit any host 10.7.0.30 host 10.100.0.53

1. In entry #1 it is allowing 10.7.0.30 to access 10.100.0.120
2. In entry #100 it is allowing 10.7.0.30 host to access 10.100.0.53

Both 10.100.0.120 and 10.100.0.53 are pingable. But when the user ATMMSOpenVPN login to OpenVPN, he will ONLY be able to ping 10.100.0.120, NOT 10.100.0.53, because 10.100.0.53 is rule #100 (which is above 70)

BUT, if I switch these 2 IP around where now #1 is 10.100.0.53 and #100 is 10.100.0.120 (in the MySQL database), now he will be able tto ping 10.100.0.53 BUT NOT 10.100.0.120 anymore. So almost seems like pfsense only loading the first 70 or so rules and render the rest of the entries (beyind 70 rows) to be "useless".


Files

Actions #1

Updated by Marcos M almost 2 years ago

  • File 562.diff added

There's an MR that changes the way AVPair rules are handled with OpenVPN users (for a different feature request). Would you test the following patch and provide feedback on whether anything breaks/gets fixed after the patch?

Use the System Patches package to apply the patch. After applying the patch, the following commands need to be run (can be done from Diagnostics / Command Prompt):

chmod 755 /usr/local/sbin/openvpn.connect_async.sh
chmod 755 /usr/local/sbin/openvpn_connect_async.php

Actions #2

Updated by Marcos M almost 2 years ago

  • Subject changed from FreeRadius RadReply table entries issue (using MySQL) to FreeRadius RadReply table entries missing from pf
  • Description updated (diff)
Actions #3

Updated by Marcos M almost 2 years ago

Tested with the patch applied, but the issue remains. Note: the rule syntax originally tested was incorrect - this test is with the correct syntax.

There are 300+ rules for the user, but only 65 get loaded into pf:

[2.6.0-RELEASE][root@pfSense.localdomain]/root: pfSsh.php playback pfanchordrill

ipsec rules/nat contents:

miniupnpd rules/nat contents:

natearly rules/nat contents:

natrules rules/nat contents:

openvpn rules/nat contents:

openvpn/ovpns1_ATMMSOpenVPN_61575 rules/nat contents:
pass in quick on ovpns1 inet from 10.7.0.30 to <__automatic_b3692641_0> flags S/SA keep state

tftp-proxy rules/nat contents:

userrules rules/nat contents:
2.6.0-RELEASE][root@pfSense.localdomain]/root: cat /tmp/ovpn_ovpns1_ATMMSOpenVPN_61575.rules | wc -l
      65

Actions #4

Updated by Jim Pingle almost 2 years ago

The number that works is too conveniently close to 64 to be a coincidence. It sounds like it's hitting a limit somewhere but it's not clear where. I don't see anything in the RADIUS auth code that would be limited to 64 attributes, but it could be anywhere in that chain. Will need to debug more, including a packet capture of the RADIUS auth exchange to see if more than 64 rules are being sent back in reply attributes. And look in the file that gets the rule content to see if more are there and just not being loaded in the anchor.

Actions #5

Updated by Marcos M almost 2 years ago

To add some details from the test:
The file contents did have just 65 rules. I also tried increasing the php max_input_vars (and restarted php-fpm/webconfig) in case that came into play somewhere, but got the same result.

Actions #6

Updated by Frank Lee almost 2 years ago

  • File LAN_Capture(NoFilter).cap added
  • File WAN_Capture.cap added

Hello,

the support tech told me you guys wanted a packet capture. I assume that you need packet capture between the Radius server (internal radius server) and the MySQL? I'm not sure if I did this right, but here is what I did:

1. I turn on packet capture on LAN with NO FILTER on IP or port.
2. I restarted the OpenVPN server (to kick all user out)
3. Then I waited until the user ATMMSOpenVPN to log on, he is the one with few hundred AVPair rules
4. Then I stop the capture.

Attached is the file. I do see some MySQL packets going back and forth, hopefully this helps.

I've also capture a WAN interface data between the OpenVPN client and pfsense (WAN_Capture.cap).

I was just wondering how long will someone start work on this bug as I need to know if I need to find another product to replace pfSense for this feature.

If you need anything, you can always email me at x or call me on x.

Thank you!

Frank

Actions #7

Updated by Marcos M almost 2 years ago

  • File deleted (WAN_Capture.cap)
Actions #8

Updated by Marcos M almost 2 years ago

  • File deleted (LAN_Capture(NoFilter).cap)
Actions #10

Updated by Marcos M almost 2 years ago

Note: removed potentially sensitive info from comment.

Thanks for the test. I presume 10.1.10.50 is pfSense and 10.1.10.207 is the sql server. It looks like the correct data was sent to pfSense (495 rules for user ATMMSOpenVPN). Hence, the issue seems to lie on the freeradius package or pfSense itself.

Actions #11

Updated by Frank Lee almost 2 years ago

Yes, you are correct, 10.1.10.50 is pfSense and 10.1.10.207 is MySQL which provide the table for the FreeRadius on pfsense. It almost like there is some limit set (amount of "rows") that pfsense will allow from the radius server. But like you said, either is the freeradius package limiting that or pfsense itself.

Thank you for looking into this.

Like I said, anytime you want to do more "live" testing or capturing, if you think is easier, you can always contact me.

Actions #12

Updated by Jim Pingle almost 2 years ago

I've looked at the RADIUS code we have but didn't see anything that would suggest any kind of limit. I also briefly looked at the underlying RADIUS code from upstream and didn't see anything that looked like it might be imposing a limit, but I didn't dig too deeply in the C code. Though if the rules were all in the RADIUS reply then it has to be hitting a limit and stopping somewhere. It may be a limit of PHP itself but it's hard to say for sure without narrowing it down further.

Actions #13

Updated by Frank Lee almost 2 years ago

Hello team,

Any further progress? Sorry keep bugging you guys, but I just need to know, if possible, any rough ETA as to when this problem may get resolved? Because we have customer waiting to get this issue resolve and if we can't do it in a reasonable time frame, I really need to look for another solution. Does netgate have another product that can do this? Or do you guys have any suggestion?

Thank you!

Actions #14

Updated by Frank Lee almost 2 years ago

Any new update?

Actions #15

Updated by Marcos M almost 2 years ago

  • File debug1.patch added

The issue needs to be narrowed down further. Apply the following patch, reproduce the issue, then submit the /tmp/_DEBUG* files. You can run ls -l /tmp/_DEBUG* from Diagnostics / Command Prompt to view the names, and use the same page to download them.

Actions #16

Updated by Frank Lee almost 2 years ago

Hello Marcos, I sent you an email on the result, but It seems it is not outputing the information you need. Not sure what is wrong, maybe the way I apply the patch?

Actions #18

Updated by Frank Lee almost 2 years ago

Hello Marcos, after using your patch, I took the log as requested. It seems it loaded 63 rows and stopped.

Actions #19

Updated by Marcos M almost 2 years ago

I can't find where [ciscoavpair] is being set in the code - the only reference I could find was in pear-Auth_RADIUS in the FreeBSD ports. This needs someone with more knowledge than I have.

Actions #20

Updated by Frank Lee almost 2 years ago

Ok, so do you know roughly when "someone" can look at this issue further?

Actions #21

Updated by Marcos M over 1 year ago

I was able to replicate this with a simpler setup by adding a custom option to the Additional RADIUS Attributes (REPLY-ITEM) field at the bottom of a FreeRADIUS user config entry. For example, add the following up to 10.0.0.99:

Cisco-AVPair += "ip:inacl#1=permit ip host {clientip} host 10.0.0.1",|Cisco-AVPair += "ip:inacl#2=permit ip host {clientip} host 10.0.0.2",|Cisco-AVPair += "ip:inacl#3=permit ip host {clientip} host 10.0.0.3" 

The retrieved ciscoavpair attribute will only contain up to 68 entries. This seems to be an issue with the Auth_RADIUS PHP extension (or perhaps with PHP itself, though I think that's less likely). https://github.com/pear/Auth_RADIUS

For now, I'd recommend finding an alternative to needing more than 68 rules per user.

Actions #22

Updated by Marcos M over 1 year ago

  • File deleted (debug1.patch)
Actions #23

Updated by Marcos M over 1 year ago

  • File deleted (562.diff)
Actions #24

Updated by Frank Lee over 1 year ago

So are you saying that pfsense/freeRadius will not be able to go more then 68 rules? any software you know would be able to do this?

Actions #25

Updated by Christian McDonald over 1 year ago

I've been working on the radius code quite a bit over the past few weeks. The radius client library used in pfSense doesn't support RFC7499 (https://www.rfc-editor.org/rfc/rfc7499.html) and so RADIUS request/response payloads have an upper limit of 4096 bytes. Thus, there is a limit to the maximum number of attributes one can receive. I will read the RADIUS packet fragmentation RFC over the next few days and see what it would take to support this.

In the meantime, we need to document this limitation.

Actions #26

Updated by Christian McDonald over 1 year ago

  • Assignee set to Christian McDonald
Actions #27

Updated by Frank Lee over 1 year ago

Hello Christian,

thank you VERY MUCH for looking into this. Any sort of workaround or patch would be GREATLY appreciated!

Actions #28

Updated by Frank Lee over 1 year ago

Any update? thank you!

Actions #29

Updated by Frank Lee over 1 year ago

Hello Christian. Ok, I finally understand what you are saying. I've install a stand along FreeRadius (as one of your over the phone tech support told me to try). But then, like you said, it seems pfsense does not support RFC7499. thus then the returned "reply rules" are too many (bigger then 4096 bytes), it will get truncated by pfsense.

Because when I run FreeRadius in -x (debug mode) I saw the warning " WARNING: Packet is large, and possibly truncated - 4048 vs max 4096" whenever user login which has large amount of Cisco AV rules in the RadReply table.

Thus, my question, have you figure out if that pfsense can support RFC7499? And if it can, roughly what is the time frame? Or, is there a "hardware" version of netgate or pfsense I can buy that support this feature?

Thank you!

Actions #30

Updated by Frank Lee over 1 year ago

Hello, so new information. I was able to install a separate stand-alone Ubuntu server which includes both FreeRadius and OpenVPN. And like you said, the library (radiusplugin.so) is fairly old. All the plugins wrote by varies author are all around 10 years or older. Thus like you said, the library simply won't handle that many entries. HOWEVER, what I did is "bypass" the radius server on Radreply. Meaning that I DO NOT PUT the Cisco-AVpair rows in the radreply table. Instead I created a table (in the Freeradius database), for example, call "Routes". And when the script calls the "client-connect" event, I simply have it call a bash script which will read all the rows from the MySQL table with the username (since username at that point is in an enviroment variable already $commonname). And then I simply manipulate the iptables to insert all those entries. Although for pfsense, you guys probably simply insert those rows into your firewall rules. So, maybe you guys can modify your pfsense to have some sort workaround like this. But anyway, I will be using the stand alone server I've created. Thank you VERY MUCH for all the information so I was able to come up with a work-around for myself.

Actions #31

Updated by Marcos M over 1 year ago

  • Tracker changed from Bug to Feature
  • Project changed from pfSense Packages to pfSense
  • Subject changed from FreeRadius RadReply table entries missing from pf to Add support for RFC7499 in RADIUS library.
  • Category changed from FreeRADIUS to Operating System
  • Affected Version deleted (2.6.0)
  • Release Notes set to Default
  • Affected Architecture deleted (All)
Actions #32

Updated by Christian McDonald 11 months ago

  • Status changed from New to Rejected

RFC7499 is experimental. Thus, it is unlikely we will support RFC7499.

Actions

Also available in: Atom PDF