pfSense

Short version - I have a PPPoE link from my ISP, which is delivered tagged as VLAN 10 by the local telco.

Outbound IPv6 works fine (like browsing redmine) but replies to any inbound connections fail to be routed out the correct interface.

Details
I have IPv6 address o 2400:6900:ffff:fe00:225:b3ff:fe0a:b422/64 on my work desktop ttt07 which will be used for this test.

pfSense box is a Atom running 64 bit, currently 2.1.1-PRERELEASE (amd64)
built on Sun Mar 23 12:34:36 EDT 2014 FreeBSD 8.3-RELEASE-p14

LAN 2400:6900:ffff:1::1:1/64
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fe80::230:1bff:fe82:bcac%pppoe0 prefixlen 64 scopeid 0xa
inet6 fe80::b1f5:da07:7127:5a0d%pppoe0 prefixlen 64 scopeid 0xa
inet 175.111.102.128 --> 175.111.100.37 netmask 0xffffffff
inet6 2400:6900:ffff:1:230:1bff:fe82:bcac prefixlen 64 autoconf
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
re0_vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3<RXCSUM,TXCSUM>
ether 00:30:1b:82:bc:ac
inet6 fe80::230:1bff:fe82:bcac%re0_vlan10 prefixlen 64 scopeid 0x6
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (1000baseT <full-duplex,master>)
status: active
vlan: 10 vlanpcp: 0 parent interface: re0

My VDSL link is terminated by a crappy dynalink modem in bridging mode. The local Telco provides a link tagged into vlan 10 and the ISP has a PPPoE session over that link.

Supporting packet traces
I'm just demonstrating with icmp6 ping, but http and ssh etc do exactly the same thing:

On my remote test machine
[cfalconer@ttt07 ~]$ ping6 criggie.org.nz -nn
PING criggie.org.nz(2400:6900:ffff:1::1:2) 56 data bytes
(nothing)
^C

While that is running, on the pfsense box
[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(7): tcpdump -i pppoe0 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 96 bytes
10:39:15.988572 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 54, length 64
10:39:16.988364 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 55, length 64
10:39:17.988303 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 56, length 64
10:39:18.988303 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 57, length 64
^C
4 packets captured
171 packets received by filter
0 packets dropped by kernel

on the internal pfsense LAN interface - we see ping request and an answering ping reply.

[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(9): tcpdump -i re0 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 96 bytes
10:40:05.988470 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 104, length 64
10:40:05.988587 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 104, length 64
10:40:06.988240 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 105, length 64
10:40:06.988355 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 105, length 64
10:40:07.988490 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 106, length 64
10:40:07.988606 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 106, length 64
10:40:08.990528 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 107, length 64
10:40:08.990635 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 107, length 64
^C
14 packets captured
2410 packets received by filter
0 packets dropped by kernel
[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(10):

The internal host is a linux box, and its definitely seeing the IPv6 ping and replying correctly.
criggie@caffeine:~$ sudo tcpdump -i eth0 icmp6 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:41:15.987947 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 174, length 64
10:41:15.987981 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 174, length 64
10:41:16.987999 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 175, length 64
10:41:16.988032 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 175, length 64
10:41:17.987923 IP6 ttt07 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 176, length 64
10:41:17.987955 IP6 2400:6900:ffff:1::1:2 > ttt07: ICMP6, echo reply, seq 176, length 64
^C
6 packets captured
10 packets received by filter
0 packets dropped by kernel

So where is the reply going? Its not blocked by any firewall rules. If I look on the "physical" interface that carries the PADI/PADO packets of PPPoE then I see neighbour solicitations, and these stop dead when I stop the test ping.
[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(10): tcpdump -i re0_vlan10 icmp6
tcpdump: WARNING: re0_vlan10: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0_vlan10, link-type EN10MB (Ethernet), capture size 96 bytes
10:44:27.988464 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:44:28.988045 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:44:29.988102 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:44:30.988491 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:44:31.988227 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
^C
5 packets captured
728 packets received by filter
0 packets dropped by kernel

fe80::230:1bff:fe82:bcac is an IPv6 address on the pfsense box.

Here's the IPv6 routing table as shown by netstat -arn

Internet6:
Destination Gateway Flags Netif Expire
default fe80::c664:13ff:fe9e:bf80%pppoe0 UGS pppoe0
::1 ::1 UH lo0
2400:6900:ffff:1::/64 link#1 U re0
2400:6900:ffff:1::1:1 link#1 UHS lo0
2400:6900:ffff:1::90:0/120 link#9 U re0_vlan
2400:6900:ffff:1::90:1 link#9 UHS lo0
2400:6900:ffff:1:230:1bff:fe82:bcac link#10 UHS lo0
fe80::%re0/64 link#1 U re0
fe80::230:1bff:fe82:bcac%re0 link#1 UHS lo0
fe80::%lo0/64 link#4 U lo0
fe80::1%lo0 link#4 UHS lo0
fe80::%re0_vlan10/64 link#6 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan10 link#6 UHS lo0
fe80::%re0_vlan30/64 link#7 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan30 link#7 UHS lo0
fe80::%re0_vlan40/64 link#8 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan40 link#8 UHS lo0
fe80::%re0_vlan90/64 link#9 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan90 link#9 UHS lo0
fe80::%pppoe0/64 link#10 U pppoe0
fe80::230:1bff:fe82:bcac%pppoe0 link#10 UHS lo0
fe80::b1f5:da07:7127:5a0d%pppoe0 link#10 UHS lo0
ff01::%re0/32 fe80::230:1bff:fe82:bcac%re0 U re0
ff01::%lo0/32 ::1 U lo0
ff01::%re0_vlan10/32 fe80::230:1bff:fe82:bcac%re0_vlan10 U re0_vlan
ff01::%re0_vlan30/32 fe80::230:1bff:fe82:bcac%re0_vlan30 U re0_vlan
ff01::%re0_vlan40/32 fe80::230:1bff:fe82:bcac%re0_vlan40 U re0_vlan
ff01::%re0_vlan90/32 fe80::230:1bff:fe82:bcac%re0_vlan90 U re0_vlan
ff01::%pppoe0/32 fe80::230:1bff:fe82:bcac%pppoe0 U pppoe0
ff02::%re0/32 fe80::230:1bff:fe82:bcac%re0 U re0
ff02::%lo0/32 ::1 U lo0
ff02::%re0_vlan10/32 fe80::230:1bff:fe82:bcac%re0_vlan10 U re0_vlan
ff02::%re0_vlan30/32 fe80::230:1bff:fe82:bcac%re0_vlan30 U re0_vlan
ff02::%re0_vlan40/32 fe80::230:1bff:fe82:bcac%re0_vlan40 U re0_vlan
ff02::%re0_vlan90/32 fe80::230:1bff:fe82:bcac%re0_vlan90 U re0_vlan
ff02::%pppoe0/32 fe80::230:1bff:fe82:bcac%pppoe0 U pppoe0

WAN interface (pppoe0)
Status up
PPPoE up
Uptime 00:48:23
MAC address 00:00:00:00:00:00 - Xerox
IPv4 address 175.111.102.128
Subnet mask IPv4 255.255.255.255
Gateway IPv4 175.111.100.37
IPv6 Link Local fe80::230:1bff:fe82:bcac%re0_vlan10 <--- this looks wrong
Gateway IPv6 fe80::c664:13ff:fe9e:bf80
ISP DNS servers 127.0.0.1
131.203.248.1
131.203.1.5
175.111.100.32
In/out packets 271596/156593 (320.78 MB/17.54 MB)
In/out packets (pass) 271596/156593 (320.78 MB/17.54 MB)
In/out packets (block) 2166/0 (362 KB/0 bytes)
In/out errors 0/0
Collisions 0

Here's a ping of the default IPv6 gateway, from the pfsense box specifying that pppoe0 is the interface to use - this works fine.

[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(19): ping6 fe80::c664:13ff:fe9e:bf80%pppoe0
PING6(56=40+8+8 bytes) fe80::b1f5:da07:7127:5a0d%pppoe0 --> fe80::c664:13ff:fe9e:bf80%pppoe0
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=0 hlim=64 time=5.787 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=1 hlim=64 time=5.208 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=2 hlim=64 time=6.414 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=3 hlim=64 time=6.088 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=4 hlim=64 time=6.420 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=5 hlim=64 time=5.881 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=6 hlim=64 time=6.280 ms
^C
--- fe80::c664:13ff:fe9e:bf80%pppoe0 ping6 statistics ---
7 packets transmitted, 7 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 5.208/6.011/6.420/0.400 ms

Same ping, but specifying that re0_vlan10 is the interface to use - this fails:
[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(18): ping6 fe80::c664:13ff:fe9e:bf80%re0_vlan10
PING6(56=40+8+8 bytes) fe80::230:1bff:fe82:bcac%re0_vlan10 --> fe80::c664:13ff:fe9e:bf80%re0_vlan10
^C
--- fe80::c664:13ff:fe9e:bf80%re0_vlan10 ping6 statistics ---
36 packets transmitted, 0 packets received, 100.0% packet loss

AND this generates the same neighbour solicitation on re0_vlan10
[2.1.1-PRERELEASE][root@pfsense.criggie.org.nz]/root(2): tcpdump -i re0_vlan10 icmp6
tcpdump: WARNING: re0_vlan10: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0_vlan10, link-type EN10MB (Ethernet), capture size 96 bytes
10:51:21.564012 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:51:22.564177 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:51:23.564118 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
10:51:24.564197 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32


So my hypothesis is that IPv6 traffic from the LAN to the Internet side generates a state-table entry (connection tracking) that makes the reply route correctly.

When an IPv6 connection arrives on pppoe0 the reply traffic somehow doesn't go out the correct interface.

I cannot get my ISP to untag their traffic - all VDSLs in the country are done this way. To eliminate this as a possible cause I have tried adding a second ethernet interface to the firewall and presenting vlan 10 "untagged" by a managed switch. This removed the vlan10 tag, but the net results were identical - outbound replies to inbound traffic went out the "underlying" interface, rather than through the pppoe tunnel.
It doesn't feel like a VLAN issue.

To reiterate - normal IPv6 access to the net works fine, this is only access inwards from the internet at large.
I run a IPv6 enabled webserver at criggie.org.nz which definitely works via IPv6 from internal LAN clients.

Any questions please ask.