Project

General

Profile

Actions
Copy link

Bug #10960

closed

Bring up VXLANs correctly at boot

Added by Steve Wheeler over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Viktor Gurov
Category:
Interfaces
Target version:
-
Start date:
10/06/2020
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
Affected Architecture:
All

Description

With the patches from 10898 and 10899 VXLAN interfaces can be added and are created at boot. VXLANs on WAN are configured correctly but throw an error:

Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'

This doesn't seem to affect the interface.

VXLANs on other parent interfaces try to come up before the parent resulting in:

Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan3' description 'VXLAN4'' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist'
Oct 6 00:58:31     php     432     rc.bootup: The command '/sbin/ifconfig 'vxlan3' -staticarp ' returned exit code '1', the output was 'ifconfig: interface vxlan3 does not exist'
Oct 6 00:58:31     php     432     rc.bootup: The command '/usr/sbin/arp -d -i 'vxlan3' -a > /dev/null 2>&1 ' returned exit code '1', the output was ''

And incomplete configuration. Missing the description and MTU values.

Files

Screenshot from 2020-12-06 23-16-04.png (89.1 KB) Screenshot from 2020-12-06 23-16-04.png Steve Wheeler, 12/06/2020 05:16 PM
Actions
Copy link
 #1

Updated by Steve Wheeler over 3 years ago

This addresses the 2nd issue here:
https://github.com/pfsense/pfsense/pull/4474

Actions
Copy link
 #2

Updated by Renato Botelho over 3 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

PR has been merged. Thanks!

Actions
Copy link
 #3

Updated by Steve Wheeler over 3 years ago

  • Status changed from Feedback to Confirmed

Tested:

2.5.0-DEVELOPMENT (amd64)
built on Sun Oct 11 01:01:59 EDT 2020
FreeBSD 12.2-STABLE

The interface configuration ordering is now correct, it no longer tries to configyre interfaces that do not yet exist.
However vxlans on WAN (only) still throw an error at boot:

Oct 11 12:14:48     kernel         vtnet0: link state changed to UP
Oct 11 12:14:50     sshd     8764     Server listening on :: port 22.
Oct 11 12:14:50     sshd     8764     Server listening on 0.0.0.0 port 22.
Oct 11 12:14:52     check_reload_status     1127     rc.newwanip starting vtnet0
Oct 11 12:14:52     php     1143     rc.bootup: calling interface_dhcpv6_configure.
Oct 11 12:14:52     php     1143     rc.bootup: Accept router advertisements on interface vtnet0
Oct 11 12:14:52     php     1143     rc.bootup: Starting rtsold process
Oct 11 12:14:53     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 11 12:14:53     kernel         vxlan0: link state changed to UP
Oct 11 12:14:53     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 11 12:14:53     kernel         vxlan1: link state changed to UP
Oct 11 12:14:53     check_reload_status     1127     Linkup starting vxlan0
Oct 11 12:14:53     check_reload_status     1127     Linkup starting vxlan1
Oct 11 12:14:54     php     1143     rc.bootup: The command '/sbin/ifconfig 'vxlan0' vxlanid '12345' vxlanremote '172.21.16.187' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'
Oct 11 12:14:54     php     1143     rc.bootup: The command '/sbin/ifconfig 'vxlan1' vxlanid '54321' vxlanremote '172.21.16.222' vxlanlocal '172.21.16.167' vxlanlearn ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'
Oct 11 12:14:54     check_reload_status     1127     Linkup starting vtnet1
Oct 11 12:14:54     kernel         vtnet1: link state changed to UP
Oct 11 12:14:54     check_reload_status     1127     Linkup starting vxlan3
Oct 11 12:14:54     kernel         vxlan3: Ethernet address: 58:9c:fc:10:ff:f5
Oct 11 12:14:54     kernel         vxlan3: link state changed to UP

Though the resulting interfaces are given a vxlanid correctly:

vxlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN0
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:cb:30
    inet6 fe80::5a9c:fcff:fe10:cb30%vxlan0 prefixlen 64 scopeid 0x7
    groups: vxlan
    vxlan vni 12345 local 172.21.16.167:4789 remote 172.21.16.187:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vxlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN3
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:9f
    inet6 fe80::5a9c:fcff:fe10:ff9f%vxlan1 prefixlen 64 scopeid 0x8
    groups: vxlan
    vxlan vni 54321 local 172.21.16.167:4789 remote 172.21.16.222:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vxlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    description: VXLAN4
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:f5
    inet6 fe80::5a9c:fcff:fe10:fff5%vxlan3 prefixlen 64 scopeid 0x9
    groups: vxlan
    vxlan vni 54322 local 192.168.167.1:4789 remote 192.168.180.56:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Actions
Copy link
 #5

Updated by Steve Wheeler over 3 years ago

With that change the vxlans on WAN are created then destroyed and re-created at boot. Somehow the vxlans on WAN are being created before that code is run. Which I imagine is also triggering that error when you don't destroy them.

Oct 14 18:32:32     kernel         vtnet0: link state changed to UP
Oct 14 18:32:33     sshd     8015     Server listening on :: port 22.
Oct 14 18:32:33     sshd     8015     Server listening on 0.0.0.0 port 22.
Oct 14 18:32:35     check_reload_status     416     rc.newwanip starting vtnet0
Oct 14 18:32:35     php     432     rc.bootup: calling interface_dhcpv6_configure.
Oct 14 18:32:35     php     432     rc.bootup: Accept router advertisements on interface vtnet0
Oct 14 18:32:35     php     432     rc.bootup: Starting rtsold process
Oct 14 18:32:36     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 14 18:32:36     kernel         vxlan0: link state changed to UP
Oct 14 18:32:36     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 14 18:32:36     kernel         vxlan1: link state changed to UP
Oct 14 18:32:36     check_reload_status     416     Linkup starting vxlan0
Oct 14 18:32:36     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     check_reload_status     416     Linkup starting vtnet1
Oct 14 18:32:37     kernel         vtnet1: link state changed to UP
Oct 14 18:32:37     kernel         vxlan0: link state changed to DOWN
Oct 14 18:32:37     rtsold     17653     <rtsock_input_ifannounce> interface vxlan0 removed
Oct 14 18:32:37     kernel         vxlan0: Ethernet address: 58:9c:fc:10:cb:30
Oct 14 18:32:37     rtsold     17653     <rtsock_input_ifannounce> interface vxlan1 removed
Oct 14 18:32:37     kernel         vxlan0: link state changed to UP
Oct 14 18:32:37     kernel         vxlan3: Ethernet address: 58:9c:fc:10:ff:f5
Oct 14 18:32:37     kernel         vxlan3: link state changed to UP
Oct 14 18:32:37     kernel         vxlan1: link state changed to DOWN
Oct 14 18:32:37     kernel         vxlan1: Ethernet address: 58:9c:fc:10:ff:9f
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan0
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan3
Oct 14 18:32:37     kernel         vxlan1: link state changed to UP
Oct 14 18:32:37     kernel         vxlan2: Ethernet address: 58:9c:fc:10:ff:99
Oct 14 18:32:37     kernel         vxlan2: link state changed to UP
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     kernel         done.
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan1
Oct 14 18:32:37     kernel         done.
Oct 14 18:32:37     check_reload_status     416     Linkup starting vxlan2
Actions
Copy link
 #6

Updated by Renato Botelho over 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from Steve Wheeler to Renato Botelho

I'll work on this one

Actions
Copy link
 #7

Updated by Jim Pingle over 3 years ago

  • Status changed from In Progress to Pull Request Review
Actions
Copy link
 #8

Updated by Renato Botelho over 3 years ago

  • Status changed from Pull Request Review to Feedback
  • Assignee changed from Renato Botelho to Viktor Gurov

PR has been merged. Thanks!

Actions
Copy link
 #9

Updated by Viktor Gurov over 3 years ago

Actions
Copy link
 #10

Updated by Jim Pingle over 3 years ago

  • Status changed from Feedback to Pull Request Review
Actions
Copy link
 #11

Updated by Renato Botelho over 3 years ago

  • Status changed from Pull Request Review to Feedback

PR has been merged. Thanks!

Actions
Copy link
 #12

Updated by Viktor Gurov over 3 years ago

  • Status changed from Feedback to Resolved

works as expected on 2.5.0.a.20201127.1850:

Nov 28 19:26:35 pf41 check_reload_status[373]: Linkup starting vxlan0
Nov 28 19:26:35 pf41 kernel: vxlan0: Ethernet address: 58:9c:fc:10:cd:26
Nov 28 19:26:35 pf41 kernel: vxlan0: link state changed to UP

Actions
Copy link
 #13

Updated by Steve Wheeler over 3 years ago

With the current patch set you cannot create new VXLAN interfaces. The following error is generated:

Warning: escapeshellarg() expects parameter 1 to be string, array given in /etc/inc/interfaces.inc on line 1546 
Warning: pfSense_interface_flags() expects parameter 1 to be string, array given in /etc/inc/interfaces.inc on line 40 
Warning: stristr() expects parameter 1 to be string, array given in /usr/local/www/interfaces_vxlan_edit.php on line 117

Tested:

2.5.0-DEVELOPMENT (amd64)
built on Sun Dec 06 03:01:20 EST 2020
FreeBSD 12.2-STABLE

Looks like this: https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/11#note_26713

Actions
Copy link
 #14

Updated by Steve Wheeler over 3 years ago

  • % Done changed from 100 to 50
Actions
Copy link
 #15

Updated by Steve Wheeler over 3 years ago

  • Status changed from Needs Patch to Confirmed
Actions
Copy link
 #17

Updated by Jim Pingle over 3 years ago

  • Status changed from Confirmed to Pull Request Review
Actions
Copy link
 #18

Updated by Renato Botelho over 3 years ago

  • Status changed from Pull Request Review to Feedback

PR has been merged. Thanks!

Actions
Copy link
 #19

Updated by Steve Wheeler over 3 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 50 to 100

This looks good now.

Can create new interfaces. Can create interfaces out of sequence: where vxlan0 and vxlan2 exist a new vxlan is created as vxlan1.

Interfaces come up correctly at boot. vxlans on WAN are still brought up and down twice but that is true of all tunnel type interfaces.

Actions
Copy link
 #20

Updated by Jim Pingle about 3 years ago

  • Target version deleted (2.5.0)
  • Affected Version deleted (2.5.0)
Actions
Copy link

Also available in: Atom PDF