Project

General

Profile

Bug #10960

Bring up VXLANs correctly at boot

Added by Steve Wheeler about 2 months ago. Updated 3 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Interfaces
Target version:
Start date:
10/06/2020
Due date:
% Done:

100%

Estimated time:
Affected Version:
2.5.0
Affected Architecture:
All

Description

With the patches from 10898 and 10899 VXLAN interfaces can be added and are created at boot. VXLANs on WAN are configured correctly but throw an error:

Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy' 

This doesn't seem to affect the interface.

VXLANs on other parent interfaces try to come up before the parent resulting in:

Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan3' description 'VXLAN4'' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist'
Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan3' -staticarp ' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist'
Oct 6 00:58:31     php     432     rc.bootup: The command '/usr/sbin/arp -d -i 'vxlan3' -a > /dev/null 2>&1 ' returned exit code '1', the output was '' 

And incomplete configuration. Missing the description and MTU values.

Associated revisions

Revision 967d16ae (diff)
Added by Viktor Gurov 23 days ago

VXLAN bootup fix. Issue #10960

Revision cf3096df (diff)
Added by Viktor Gurov 8 days ago

VXLAN restart on newwanip event. Bug #10960

History

#1 Updated by Steve Wheeler about 2 months ago

This addresses the 2nd issue here:
https://github.com/pfsense/pfsense/pull/4474

#2 Updated by Renato Botelho about 2 months ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

PR has been merged. Thanks!

#3 Updated by Steve Wheeler about 2 months ago

  • Status changed from Feedback to Confirmed

Tested:

2.5.0-DEVELOPMENT (amd64)
built on Sun Oct 11 01:01:59 EDT 2020
FreeBSD 12.2-STABLE

The interface configuration ordering is now correct, it no longer tries to configyre interfaces that do not yet exist.
However vxlans on WAN (only) still throw an error at boot:

Oct 11 12:14:48     kernel         vtnet0: link state changed to UP
Oct 11 12:14:50     sshd     8764     Server listening on :: port 22.
Oct 11 12:14:50     sshd     8764     Server listening on 0.0.0.0 port 22.
Oct 11 12:14:52     check_reload_status     1127     rc.newwanip starting vtnet0
Oct 11 12:14:52     php     1143     rc.bootup: calling interface_dhcpv6_configure.
Oct 11 12:14:52     php     1143     rc.bootup: Accept router advertisements on interface vtnet0
Oct 11 12:14:52     php     1143     rc.bootup: Starting rtsold process
Oct 11 12:14:53     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 11 12:14:53     kernel         vxlan0: link state changed to UP
Oct 11 12:14:53     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 11 12:14:53     kernel         vxlan1: link state changed to UP
Oct 11 12:14:53     check_reload_status     1127     Linkup starting vxlan0
Oct 11 12:14:53     check_reload_status     1127     Linkup starting vxlan1
Oct 11 12:14:54     php     1143     rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'
Oct 11 12:14:54     php     1143     rc.bootup: The command '/sbin/ifconfig 'vxlan1' vxlanid '54321' vxlanremote '172.21.16.222' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'
Oct 11 12:14:54     check_reload_status     1127     Linkup starting vtnet1
Oct 11 12:14:54     kernel         vtnet1: link state changed to UP
Oct 11 12:14:54     check_reload_status     1127     Linkup starting vxlan3
Oct 11 12:14:54     kernel         vxlan3: Ethernet address: 58:9c:fc:10:ff:f5
Oct 11 12:14:54     kernel         vxlan3: link state changed to UP 

Though the resulting interfaces are given a vxlanid correctly:

vxlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN0
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:cb:30
    inet6 fe80::5a9c:fcff:fe10:cb30%vxlan0 prefixlen 64 scopeid 0x7
    groups: vxlan
    vxlan vni 12345 local 172.21.16.167:4789 remote 172.21.16.187:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vxlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN3
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:9f
    inet6 fe80::5a9c:fcff:fe10:ff9f%vxlan1 prefixlen 64 scopeid 0x8
    groups: vxlan
    vxlan vni 54321 local 172.21.16.167:4789 remote 172.21.16.222:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vxlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN4
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:f5
    inet6 fe80::5a9c:fcff:fe10:fff5%vxlan3 prefixlen 64 scopeid 0x9
    groups: vxlan
    vxlan vni 54322 local 192.168.167.1:4789 remote 192.168.180.56:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

#5 Updated by Steve Wheeler about 2 months ago

With that change the vxlans on WAN are created then destroyed and re-created at boot. Somehow the vxlans on WAN are being created before that code is run. Which I imagine is also triggering that error when you don't destroy them.

Oct 14 18:32:32     kernel         vtnet0: link state changed to UP
Oct 14 18:32:33     sshd     8015     Server listening on :: port 22.
Oct 14 18:32:33     sshd     8015     Server listening on 0.0.0.0 port 22.
Oct 14 18:32:35     check_reload_status     416     rc.newwanip starting vtnet0
Oct 14 18:32:35     php     432     rc.bootup: calling interface_dhcpv6_configure.
Oct 14 18:32:35     php     432     rc.bootup: Accept router advertisements on interface vtnet0
Oct 14 18:32:35     php     432     rc.bootup: Starting rtsold process
Oct 14 18:32:36     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 14 18:32:36     kernel         vxlan0: link state changed to UP
Oct 14 18:32:36     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 14 18:32:36     kernel         vxlan1: link state changed to UP
Oct 14 18:32:36     check_reload_status     416     Linkup starting vxlan0
Oct 14 18:32:36     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     check_reload_status     416     Linkup starting vtnet1
Oct 14 18:32:37     kernel         vtnet1: link state changed to UP
Oct 14 18:32:37     kernel         vxlan0: link state changed to DOWN
Oct 14 18:32:37     rtsold     17653     <rtsock_input_ifannounce> interface vxlan0 removed
Oct 14 18:32:37     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 14 18:32:37     rtsold     17653     <rtsock_input_ifannounce> interface vxlan1 removed
Oct 14 18:32:37     kernel         vxlan0: link state changed to UP
Oct 14 18:32:37     kernel         vxlan3: Ethernet address: 58:9c:fc:10:ff:f5
Oct 14 18:32:37     kernel         vxlan3: link state changed to UP
Oct 14 18:32:37     kernel         vxlan1: link state changed to DOWN
Oct 14 18:32:37     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan0
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan3
Oct 14 18:32:37     kernel         vxlan1: link state changed to UP
Oct 14 18:32:37     kernel         vxlan2: Ethernet address: 58:9c:fc:10:ff:99
Oct 14 18:32:37     kernel         vxlan2: link state changed to UP
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     kernel         done.
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     kernel         done.
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan2 

#6 Updated by Renato Botelho 28 days ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from Steve Wheeler to Renato Botelho

I'll work on this one

#7 Updated by Jim Pingle 23 days ago

  • Status changed from In Progress to Pull Request Review

#8 Updated by Renato Botelho 23 days ago

  • Status changed from Pull Request Review to Feedback
  • Assignee changed from Renato Botelho to Viktor Gurov

PR has been merged. Thanks!

#9 Updated by Viktor Gurov 10 days ago

#10 Updated by Jim Pingle 9 days ago

  • Status changed from Feedback to Pull Request Review

#11 Updated by Renato Botelho 8 days ago

  • Status changed from Pull Request Review to Feedback

PR has been merged. Thanks!

#12 Updated by Viktor Gurov 3 days ago

  • Status changed from Feedback to Resolved

works as expected on 2.5.0.a.20201127.1850:

Nov 28 19:26:35 pf41 check_reload_status[373]: Linkup starting vxlan0
Nov 28 19:26:35 pf41 kernel: vxlan0: Ethernet address: 58:9c:fc:10:cd:26
Nov 28 19:26:35 pf41 kernel: vxlan0: link state changed to UP

Also available in: Atom PDF