Bug #8507
closed
FreeBSD 11.2-BETA dhclient always uses server MTU value
0%
Description
I hit this while looking into #8506, it may not be related since it happens on other hardware. It also started around the switch to 11-stable.
Something is setting the MTU too low, and it's not in the configuration.
May 11 07:26:34 blooper kernel: nd6_setmtu0: new link MTU on mvneta0 (576) is too small for IPv6
That message happens on both an igb box and SG-3100. The routes for this interface only have that same low MTU:
: netstat -4rnW | egrep '(Mtu|mvneta0)' Destination Gateway Flags Use Mtu Netif Expire default 216.252.41.1 UGS 21213 576 mvneta0 8.8.8.8 216.252.41.1 UGHS 55825 576 mvneta0 208.123.73.7 216.252.41.1 UGHS 182429 576 mvneta0 209.51.181.2 216.252.41.1 UGHS 489319 576 mvneta0 216.252.41.0/24 link#1 U 0 576 mvneta0
The interface configuration is a DHCP WAN:
<opt1>
<descr><![CDATA[Cable]]></descr>
<if>igb2</if>
<enable></enable>
<alias-address></alias-address>
<alias-subnet>32</alias-subnet>
<spoofmac>00:xx:xx:xx:xx:xx</spoofmac>
<monitorip>x.x.x.1</monitorip>
<ipaddr>dhcp</ipaddr>
<dhcphostname></dhcphostname>
<dhcprejectfrom>192.168.100.1</dhcprejectfrom>
<adv_dhcp_pt_timeout></adv_dhcp_pt_timeout>
<adv_dhcp_pt_retry></adv_dhcp_pt_retry>
<adv_dhcp_pt_select_timeout></adv_dhcp_pt_select_timeout>
<adv_dhcp_pt_reboot></adv_dhcp_pt_reboot>
<adv_dhcp_pt_backoff_cutoff></adv_dhcp_pt_backoff_cutoff>
<adv_dhcp_pt_initial_interval></adv_dhcp_pt_initial_interval>
<adv_dhcp_pt_values>SavedCfg</adv_dhcp_pt_values>
<adv_dhcp_send_options></adv_dhcp_send_options>
<adv_dhcp_request_options></adv_dhcp_request_options>
<adv_dhcp_required_options></adv_dhcp_required_options>
<adv_dhcp_option_modifiers></adv_dhcp_option_modifiers>
<adv_dhcp_config_advanced></adv_dhcp_config_advanced>
<adv_dhcp_config_file_override></adv_dhcp_config_file_override>
<adv_dhcp_config_file_override_path></adv_dhcp_config_file_override_path>
</opt1>
It does have a spoofed MAC, and there is a GIF interface on top of that interface as well as an OpenVPN and IPsec, but attempting to configure those on a fresh lab install doesn't trigger the issue, and removing them or disabling them here doesn't seem to affect it.
Other interfaces (PPPoE WAN, multiple local LANs) are unaffected.
I commented out every call to pfSense_interface_mtu() in interfaces.inc and the low MTU still happened, so it doesn't appear to be related to that function at least.
Files
Updated by Jim Pingle almost 6 years ago
Same thing happens on a factory default configuration, so looking deeper at packet captures of the DHCP packets the ISP is sending an MTU of 576. This was not requested on 11.1, but is being requested and applied now.
/usr/local/sbin/pfSense-dhclient-script doesn't have any code to process the MTU, so it may be getting set by dhclient directly.
Comparing 2.4.3-p1 and 2.4.4, the only difference in the request is that the 2.4.4 client requests option 26 (Interface MTU) in addition to the other parameters.
Updated by Jim Pingle almost 6 years ago
Looks like this is a recent change in FreeBSD dhclient to add support for the MTU:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206721
https://svnweb.freebsd.org/base?view=revision&revision=331179
https://reviews.freebsd.org/D5675
https://svnweb.freebsd.org/base?view=revision&revision=239564
It seems they were aware it could cause link bounces, and changed dhclient so it doesn't exit when the link drops, but it appears to still trip up our link handling scripts.
Based on that, this could end up being the same root cause as #8506 but it manifested differently depending on the hardware.
Updated by Jim Pingle almost 6 years ago
I tried setting an explicit request list in the generated dhclient configuration which does not send a request for the MTU. Didn't help, if the server sends the MTU it still is taken and used.
I also tried setting a supersede statement for interface-mtu with 0, 1, and other values (e.g. 1402), all of which appear to be ignored in favor of the server-side MTU value every time.
It looks like the way it was coded into dhclient in https://reviews.freebsd.org/D5675 that it can't be ignored or overridden, which is rather awful. We may need to back that patch out or convince FreeBSD that it needs fixed to at least have a way to turn that off or respect the supersede. Preferably disable it entirely.
Updated by Jim Pingle almost 6 years ago
- Subject changed from Interface MTU being set incorrectly in some cases to FreeBSD 11.2-BETA dhclient always uses server MTU value
Updated the subject to be more accurate.
I also dropped a note on https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206721#c10
Updated by Jim Pingle almost 6 years ago
- Status changed from New to Feedback
Renato committed a patch that was added to the FreeBSD PR that should let supesede work, next snapshots should be better.
Updated by Jim Pingle almost 6 years ago
The patch looks good. Setting a supersede of 0 in the dhclient config now allows the MTU change to be ignored. The test system I could replicate the MTU issue on is OK now, and there are multiple confirmations that the latest snapshot is working as expected from forum users.
We need to keep an eye on https://reviews.freebsd.org/D15484 as it evolves and is eventually committed to 11.2. Once it makes it into the tree, the manually added patch can be removed.
Updated by Jim Pingle almost 6 years ago
- Status changed from Feedback to Assigned
- Assignee set to Renato Botelho
- Priority changed from Very High to Normal
- Affected Architecture All added
- Affected Architecture deleted (
)
The supersede change was committed and now has been MFC'd as well:
https://svnweb.freebsd.org/base?view=revision&revision=334787
We should be able to remove the patch we added manually in favor of the version now the tree.
Updated by Jim Pingle almost 6 years ago
- Status changed from Assigned to Resolved
We're on 11.2-RELEASE now with stock patches, working as expected.
Updated by Bennett Feitell over 5 years ago
This is still causing problems in pfSense 2.4.4-RELEASE.
I believe that the upstream patches to FreeBSD introduce the ability to supersede the option 26 interface-mtu value being returned by the DHCP server. The user must affirmatively add "supersede interface-mtu 0", or the system will set the bad MTU in the DHCP server response. Option 26 information is still being requested, and if it is returned by the server with the lease it will be enforced to the point of overriding an explicitly set MTU. If the MTU given in the lease is bad, things break.
Please see these threads:
https://forum.netgate.com/topic/136089/solved-and-revised-2-4-4-release-arpresolve-can-t-allocate-llinfo-for-gateway-on-interface0-dhcp-mtu-576
https://forum.netgate.com/topic/136121/wan-interfaces-fail-to-return-after-power-outage
Updated by Jim Pingle over 5 years ago
We already set that (see the linked commit above).
My edge firewall failed horribly because my ISP sent the MTU even when not requested, but it is working now. So there is something else happening here that isn't necessarily related to the dhclient config, but perhaps the presence of the proper code/patch in the binary.
Updated by Bennett Feitell over 5 years ago
All I know is that I needed to affirmatively set "supersede interface-mtu 0" in the option modifiers for the WAN config. The incantation does not seem to be issued by default. It takes user action from what I have seen.
Updated by Jim Pingle over 5 years ago
https://github.com/pfsense/pfsense/blob/master/src/etc/inc/interfaces.inc#L4990
The line should always be there by default, unless there is some other customization that prevented it from working. Check your copy of /etc/inc/interfaces.inc and make sure it matches the above line.
If you have any other custom settings in the WAN DHCP client settings, I'd be interested to know what they are.
Updated by Bennett Feitell over 5 years ago
I suspect that a prior hard setting of MTU on the interface may be interfering with the propagation of the fix in dhclient.conf on upgrades. I know that I have routinely hard set an MTU of 1500 on the boxes I adminster. Is it possible that the hard setting of MTU is preventing dhclient.conf from picking up the change on upgrade? If so, it would explain the behavior I have seen.
Updated by Bennett Feitell over 5 years ago
My copy of /etc/inc/interfaces.inc matches, and contains the line.
Updated by Jim Pingle over 5 years ago
- File supersede-advanced.diff supersede-advanced.diff added
Looks like there may be two ways this could happen:
#1: If you used advanced DHCP client options that clobber the default values. This case would be fixed by the attached patch.
#2: A custom dhclient configuration file, which would need to be fixed manually. Injecting automatic edits into that seems wrong, though. I'll add a note to the upgrade guide about it. Someone might conceivably need this behavior and not letting them override it seems like a bad idea.
Updated by Bennett Feitell over 5 years ago
I think I see a typo in your diff. "tsupersede".
Updated by Bennett Feitell over 5 years ago
Got it, thank you.
By advanced DHCP client options, do you mean specifying an MTU for the interface? If that would break the patch, I will guess that is what has been happening. The specified MTU is ignored, and the bad value in the lease is enforced.
Updated by Jim Pingle over 5 years ago
No, I mean settings in the DHCP Client Configuration box on the WAN when Advanced Configuration is set, like Protocol Timings, Presets, or Lease Requirements and Requests.
Updated by Bennett Feitell over 5 years ago
So a setting to reject leases from the cable modem itself in DHCP Client Configuration might cause this too, as would any other setting in that box. The machine in question definitely had reject leases from 192.168.100.1 set in that section.
Updated by Jim Pingle over 5 years ago
No, that setting would not do it on its own, as it is handled in a different way.
It only happens when adv_dhcp_config_advanced is set, which is the Advanced Configuration checkbox, or when Configuration Override is set pointing to a custom file, which is a different case.
Updated by Bennett Feitell over 5 years ago
Is it possible that just toggling the visibility of the advanced settings box to on and then saving and applying would interfere with the fix?
Updated by Jim Pingle over 5 years ago
If you check the box and save, it will overwrite the default DHCP values with what is in the boxes for Protocol Timings, Presets, or Lease Requirements and Requests, even if they are empty. The other settings do not require checking that box. So if someone did check it, they would lose the automatic supersede without my last patch, even if they made no other changes.
Updated by Bennett Feitell over 5 years ago
I suspect that might be happening to people. Thank you for being so attentive on this.
Updated by Jim Pingle over 5 years ago
I added a note to the upgrade guide pointing back here and offering a workaround. Thanks for confirming the details!