pfSense

Traffic with a vlan tag is not seen on the vlan interface but is seen on the master interface.

traffic with a vlan tag is not recognized.

I'm using a jetway NF99FL-525 atom based chassis with a 3 port intel expansion board.

em0 is setup as the wan connection, having two vlan's
em0_vlan4 = IPtv (vlan is bridged to em4 which is the local IPtv interface, DHCP for local IP address so IGMP proxy can be used to have local lan join the IPtv multicast streams)
em0_vlan6 = Internet (actually this is PPPoE over vlan 6)

In my current 2.0.1 firewall, based on the same hardware, this works flawlessly.
on pfsense 2.1 I can see outgoing traffic (PPPoE discovery initiation frames- PADI) I also see the incoming PPPoE discovery offer. (PPPoE - PADO)
Traces of this traffic are made with port mirroring on a switch (MRV OS906) and with tcpdump -ei em0

When running tcpdump -ei em0_vlan6 I only see the initiation frames and not the offer from the provider.
This also happens with the DHCP request on the IPtv vlan.

To check again and to be sure incoming traffic on the vlan is not handled, I gave the vlan an IP address and started a ping.
result is that I can see the ARP requst gouing out of the firewall with the right vlan tag.
I can also see the incoming ARP reply with the tag on the em0 interface, but not on the em0_vlan6 interface.

I changed the PCP bits but this did not help.

There is no mixing of lan and vlan. the physical interface em0 is not used for traffic. It's not even configured as a firewall interface.
on em0 there are two vlans, em0_vlan4 (obtaining an ip address by dhdp and bridged to em4) and em0_vlan6 obtaining its ip address by PPPoE.

There is no misconfiguration of vlans.

[2.1-BETA0][admin@firewall]/root(9): ifconfig -a
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=5219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
ether 00:30:18:a2:bd:13
inet6 fe80::230:18ff:fea2:bd13%em0 prefixlen 64 scopeid 0x6
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em0_vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:30:18:a2:bd:13
inet6 fe80::76f0:6dff:fe80:9448%em0_vlan4 prefixlen 64 scopeid 0x13
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 4 vlanpcp: 0 parent interface: em0
em0_vlan6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:30:18:a2:bd:13
inet6 fe80::76f0:6dff:fe80:9448%em0_vlan6 prefixlen 64 scopeid 0x14
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 6 vlanpcp: 0 parent interface: em0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:fe:4a:c8:9c:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em0_vlan4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 19 priority 128 path cost 20000
member: em4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000000

If there was a vlan misconfiguration one would expect there was no traffic on the vlan at all.
Now I can see outgoing traffic on the vlan.
Incoming traffic is seen on the master interface em0 but not on the vlan interface.

Traffic seen on the em0 interface, with vlan tag
[2.1-BETA0][admin@firewall]/root(12): tcpdump -ei em0
tcpdump: WARNING: em0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
18:35:21.906368 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x40196E0500FFFFFF] [Service-Name]
18:35:27.908087 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x00E6542000FFFFFF] [Service-Name]
18:35:29.907267 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x00E6542000FFFFFF] [Service-Name]
^C3 packets captured
3 packets received by filter
0 packets dropped by kernel

traffic seen on the em0_vlan6 vlan interface, as tcpdump captures the virtual interface, traffic is seen without the VID
[2.1-BETA0][admin@firewall]/root(14): tcpdump -ei em0_vlan6
tcpdump: WARNING: em0_vlan6: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0_vlan6, link-type EN10MB (Ethernet), capture size 96 bytes
18:39:01.960611 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 36: PPPoE PADI [Host-Uniq 0x007C2A2000FFFFFF] [Service-Name]
18:39:05.963381 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 36: PPPoE PADI [Host-Uniq 0x00E43B0500FFFFFF] [Service-Name]
18:39:07.962558 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 36: PPPoE PADI [Host-Uniq 0x00E43B0500FFFFFF] [Service-Name]
18:39:11.962525 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 36: PPPoE PADI [Host-Uniq 0x00E43B0500FFFFFF] [Service-Name]
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

the answer from the provider is seen on the master interface, but NOT on the vlan interface so firewall keeps sending requests
[2.1-BETA0][admin@firewall]/root(15): tcpdump -ei em0
tcpdump: WARNING: em0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
18:43:55.033543 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x80053C0500FFFFFF] [Service-Name]
18:43:59.033491 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x80053C0500FFFFFF] [Service-Name]
18:43:59.047620 00:90:1a:a4:60:4d (oui Unknown) > 00:30:18:a2:bd:13 (oui Unknown), ethertype 802.1Q (0x8100), length 75: vlan 6, p 7, ethertype PPPoE D, PPPoE PADO [AC-Name "dr7.d12"] [Host-Uniq 0x80053C0500FFFFFF] [Service-Name] [AC-Cookie 0x8CE02000C47364613408C38906D53904] [EOL]
18:44:06.039529 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x0046B72000FFFFFF] [Service-Name]
18:44:08.039526 00:30:18:a2:bd:13 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 40: vlan 6, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0x0046B72000FFFFFF] [Service-Name]

I also attached two capture files made by a port mirror on a MRV switch.
On the 2.1 capture you can see the switch repeating the PADI request as it does net see the answer from the provider on the vlan interface.

Because I wanted to rule out any hardware issues I tried another system (a jetway NF92 board) also with a intel 3port eth extension board.
The on-board interface has a Broadcom chipset.

On this system I used the re0 with vlan's as the WAN interface.
Again, this system shows exactly the same symptoms. Incoming traffic is seen on the main interface but not on the vlan interface.

Here I also created a FW rule allowing any traffic to be able to test with ICMP (ping) also.

As a final test I did the same using exactly the same configuration with 2.0.1 software and everything works as a charm.

Andre