Bug #10960
closedBring up VXLANs correctly at boot
Added by Steve Wheeler over 4 years ago. Updated over 4 years ago.
100%
Description
With the patches from 10898 and 10899 VXLAN interfaces can be added and are created at boot. VXLANs on WAN are configured correctly but throw an error:
Oct 6 00:58:31 php 432 rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'
This doesn't seem to affect the interface.
VXLANs on other parent interfaces try to come up before the parent resulting in:
Oct 6 00:58:31 php 432 rc.bootup: The command '/sbin/ifconfig 'vxlan3' description 'VXLAN4'' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist' Oct 6 00:58:31 php 432 rc.bootup: The command '/sbin/ifconfig 'vxlan3' -staticarp ' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist' Oct 6 00:58:31 php 432 rc.bootup: The command '/usr/sbin/arp -d -i 'vxlan3' -a > /dev/null 2>&1 ' returned exit code '1', the output was ''
And incomplete configuration. Missing the description and MTU values.
Files
Screenshot from 2020-12-06 23-16-04.png (89.1 KB) Screenshot from 2020-12-06 23-16-04.png | Steve Wheeler, 12/06/2020 05:16 PM |
Updated by Steve Wheeler over 4 years ago
This addresses the 2nd issue here:
https://github.com/pfsense/pfsense/pull/4474
Updated by Renato Botelho over 4 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 100
PR has been merged. Thanks!
Updated by Steve Wheeler over 4 years ago
- Status changed from Feedback to Confirmed
Tested:
2.5.0-DEVELOPMENT (amd64) built on Sun Oct 11 01:01:59 EDT 2020 FreeBSD 12.2-STABLE
The interface configuration ordering is now correct, it no longer tries to configyre interfaces that do not yet exist.
However vxlans on WAN (only) still throw an error at boot:
Oct 11 12:14:48 kernel vtnet0: link state changed to UP Oct 11 12:14:50 sshd 8764 Server listening on :: port 22. Oct 11 12:14:50 sshd 8764 Server listening on 0.0.0.0 port 22. Oct 11 12:14:52 check_reload_status 1127 rc.newwanip starting vtnet0 Oct 11 12:14:52 php 1143 rc.bootup: calling interface_dhcpv6_configure. Oct 11 12:14:52 php 1143 rc.bootup: Accept router advertisements on interface vtnet0 Oct 11 12:14:52 php 1143 rc.bootup: Starting rtsold process Oct 11 12:14:53 kernel vxlan0: Ethernet address: 58:9c:fc:10:cb:30 Oct 11 12:14:53 kernel vxlan0: link state changed to UP Oct 11 12:14:53 kernel vxlan1: Ethernet address: 58:9c:fc:10:ff:9f Oct 11 12:14:53 kernel vxlan1: link state changed to UP Oct 11 12:14:53 check_reload_status 1127 Linkup starting vxlan0 Oct 11 12:14:53 check_reload_status 1127 Linkup starting vxlan1 Oct 11 12:14:54 php 1143 rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy' Oct 11 12:14:54 php 1143 rc.bootup: The command '/sbin/ifconfig 'vxlan1' vxlanid '54321' vxlanremote '172.21.16.222' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy' Oct 11 12:14:54 check_reload_status 1127 Linkup starting vtnet1 Oct 11 12:14:54 kernel vtnet1: link state changed to UP Oct 11 12:14:54 check_reload_status 1127 Linkup starting vxlan3 Oct 11 12:14:54 kernel vxlan3: Ethernet address: 58:9c:fc:10:ff:f5 Oct 11 12:14:54 kernel vxlan3: link state changed to UP
Though the resulting interfaces are given a vxlanid correctly:
vxlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450 description: VXLAN0 options=80000<LINKSTATE> ether 58:9c:fc:10:cb:30 inet6 fe80::5a9c:fcff:fe10:cb30%vxlan0 prefixlen 64 scopeid 0x7 groups: vxlan vxlan vni 12345 local 172.21.16.167:4789 remote 172.21.16.187:4789 media: Ethernet autoselect (autoselect <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vxlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450 description: VXLAN3 options=80000<LINKSTATE> ether 58:9c:fc:10:ff:9f inet6 fe80::5a9c:fcff:fe10:ff9f%vxlan1 prefixlen 64 scopeid 0x8 groups: vxlan vxlan vni 54321 local 172.21.16.167:4789 remote 172.21.16.222:4789 media: Ethernet autoselect (autoselect <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vxlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450 description: VXLAN4 options=80000<LINKSTATE> ether 58:9c:fc:10:ff:f5 inet6 fe80::5a9c:fcff:fe10:fff5%vxlan3 prefixlen 64 scopeid 0x9 groups: vxlan vxlan vni 54322 local 192.168.167.1:4789 remote 192.168.180.56:4789 media: Ethernet autoselect (autoselect <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Updated by Viktor Gurov over 4 years ago
first issue fix:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/11
Updated by Steve Wheeler over 4 years ago
With that change the vxlans on WAN are created then destroyed and re-created at boot. Somehow the vxlans on WAN are being created before that code is run. Which I imagine is also triggering that error when you don't destroy them.
Oct 14 18:32:32 kernel vtnet0: link state changed to UP Oct 14 18:32:33 sshd 8015 Server listening on :: port 22. Oct 14 18:32:33 sshd 8015 Server listening on 0.0.0.0 port 22. Oct 14 18:32:35 check_reload_status 416 rc.newwanip starting vtnet0 Oct 14 18:32:35 php 432 rc.bootup: calling interface_dhcpv6_configure. Oct 14 18:32:35 php 432 rc.bootup: Accept router advertisements on interface vtnet0 Oct 14 18:32:35 php 432 rc.bootup: Starting rtsold process Oct 14 18:32:36 kernel vxlan0: Ethernet address: 58:9c:fc:10:cb:30 Oct 14 18:32:36 kernel vxlan0: link state changed to UP Oct 14 18:32:36 kernel vxlan1: Ethernet address: 58:9c:fc:10:ff:9f Oct 14 18:32:36 kernel vxlan1: link state changed to UP Oct 14 18:32:36 check_reload_status 416 Linkup starting vxlan0 Oct 14 18:32:36 check_reload_status 416 Linkup starting vxlan1 Oct 14 18:32:37 check_reload_status 416 Linkup starting vtnet1 Oct 14 18:32:37 kernel vtnet1: link state changed to UP Oct 14 18:32:37 kernel vxlan0: link state changed to DOWN Oct 14 18:32:37 rtsold 17653 <rtsock_input_ifannounce> interface vxlan0 removed Oct 14 18:32:37 kernel vxlan0: Ethernet address: 58:9c:fc:10:cb:30 Oct 14 18:32:37 rtsold 17653 <rtsock_input_ifannounce> interface vxlan1 removed Oct 14 18:32:37 kernel vxlan0: link state changed to UP Oct 14 18:32:37 kernel vxlan3: Ethernet address: 58:9c:fc:10:ff:f5 Oct 14 18:32:37 kernel vxlan3: link state changed to UP Oct 14 18:32:37 kernel vxlan1: link state changed to DOWN Oct 14 18:32:37 kernel vxlan1: Ethernet address: 58:9c:fc:10:ff:9f Oct 14 18:32:37 check_reload_status 416 Linkup starting vxlan0 Oct 14 18:32:37 check_reload_status 416 Linkup starting vxlan3 Oct 14 18:32:37 kernel vxlan1: link state changed to UP Oct 14 18:32:37 kernel vxlan2: Ethernet address: 58:9c:fc:10:ff:99 Oct 14 18:32:37 kernel vxlan2: link state changed to UP Oct 14 18:32:37 check_reload_status 416 Linkup starting vxlan1 Oct 14 18:32:37 kernel done. Oct 14 18:32:37 check_reload_status 416 Linkup starting vxlan1 Oct 14 18:32:37 kernel done. Oct 14 18:32:37 check_reload_status 416 Linkup starting vxlan2
Updated by Renato Botelho over 4 years ago
- Status changed from Confirmed to In Progress
- Assignee changed from Steve Wheeler to Renato Botelho
I'll work on this one
Updated by Jim Pingle over 4 years ago
- Status changed from In Progress to Pull Request Review
Updated by Renato Botelho over 4 years ago
- Status changed from Pull Request Review to Feedback
- Assignee changed from Renato Botelho to Viktor Gurov
PR has been merged. Thanks!
Updated by Viktor Gurov over 4 years ago
VXLAN restart on newwanip event fix:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/62
Updated by Jim Pingle over 4 years ago
- Status changed from Feedback to Pull Request Review
Updated by Renato Botelho over 4 years ago
- Status changed from Pull Request Review to Feedback
PR has been merged. Thanks!
Updated by Viktor Gurov over 4 years ago
- Status changed from Feedback to Resolved
works as expected on 2.5.0.a.20201127.1850:
Nov 28 19:26:35 pf41 check_reload_status[373]: Linkup starting vxlan0 Nov 28 19:26:35 pf41 kernel: vxlan0: Ethernet address: 58:9c:fc:10:cd:26 Nov 28 19:26:35 pf41 kernel: vxlan0: link state changed to UP
Updated by Steve Wheeler over 4 years ago
- File Screenshot from 2020-12-06 23-16-04.png Screenshot from 2020-12-06 23-16-04.png added
- Status changed from Resolved to Needs Patch
With the current patch set you cannot create new VXLAN interfaces. The following error is generated:
Warning: escapeshellarg() expects parameter 1 to be string, array given in /etc/inc/interfaces.inc on line 1546 Warning: pfSense_interface_flags() expects parameter 1 to be string, array given in /etc/inc/interfaces.inc on line 40 Warning: stristr() expects parameter 1 to be string, array given in /usr/local/www/interfaces_vxlan_edit.php on line 117
Tested:
2.5.0-DEVELOPMENT (amd64) built on Sun Dec 06 03:01:20 EST 2020 FreeBSD 12.2-STABLE
Looks like this: https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/11#note_26713
Updated by Steve Wheeler over 4 years ago
- Status changed from Needs Patch to Confirmed
Updated by Viktor Gurov over 4 years ago
Updated by Jim Pingle over 4 years ago
- Status changed from Confirmed to Pull Request Review
Updated by Renato Botelho over 4 years ago
- Status changed from Pull Request Review to Feedback
PR has been merged. Thanks!
Updated by Steve Wheeler over 4 years ago
- Status changed from Feedback to Resolved
- % Done changed from 50 to 100
This looks good now.
Can create new interfaces. Can create interfaces out of sequence: where vxlan0 and vxlan2 exist a new vxlan is created as vxlan1.
Interfaces come up correctly at boot. vxlans on WAN are still brought up and down twice but that is true of all tunnel type interfaces.
Updated by Jim Pingle over 4 years ago
- Target version deleted (
2.5.0) - Affected Version deleted (
2.5.0)