Project

General

Profile

Actions

Bug #6981

closed

IPv6, rc.newwanipv6, flooding log and resets connection periodically

Added by Marcel Mayer over 7 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Interfaces
Target version:
-
Start date:
12/03/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.3.2
Affected Architecture:
amd64

Description

As you can see here (logfiles attached in threads!)

(English)https://forum.pfsense.org/index.php?topic=119439.0
(German) https://forum.pfsense.org/index.php?topic=119409.0

there seems to be a bug in the implementation of handling IPv6 over PPPoE or in dhcpcd6. The logfile will be flooded and the connection will be reset every few seconds. In generally IPv6 works. Clients are getting the correct prefix and RA also works but the periodically reset makes the line unusable, cause connections get lost.

My provider is DTAG which has a quite basic IPv6 deployment setup.

Ticket will be monitored by me and questions answered as soon as possible.


Files

system_general.log (7.32 KB) system_general.log Marcel Mayer, 12/11/2016 08:44 AM
system_general2.log (19.5 KB) system_general2.log Marcel Mayer, 12/12/2016 06:44 AM
system_general3.log (5.42 KB) system_general3.log Marcel Mayer, 12/12/2016 12:53 PM
dhcp6c_no_duid.log (20.7 KB) dhcp6c_no_duid.log Arno Gramatke, 12/22/2016 08:45 AM
dhcp6c_with_duid.log (3.28 KB) dhcp6c_with_duid.log Arno Gramatke, 12/22/2016 08:46 AM
Capture.JPG (21.8 KB) Capture.JPG Martin Wasley, 12/26/2016 02:32 AM
Actions #1

Updated by Martin Wasley over 7 years ago

Marcel Mayer wrote:

As you can see here (logfiles attached in threads!)

(English)https://forum.pfsense.org/index.php?topic=119439.0
(German) https://forum.pfsense.org/index.php?topic=119409.0

there seems to be a bug in the implementation of handling IPv6 over PPPoE or in dhcpcd6. The logfile will be flooded and the connection will be reset every few seconds. In generally IPv6 works. Clients are getting the correct prefix and RA also works but the periodically reset makes the line unusable, cause connections get lost.

My provider is DTAG which has a quite basic IPv6 deployment setup.

Ticket will be monitored by me and questions answered as soon as possible.

Is your address/pd allocation changing everytime this happens?

There has been a lot of work done around dhcp6c in later versions, you might want to try updating to 2.33. Also, I have a version of dhcp6c I have compiled myself which gives more log information than the distributed version, I could let you have that and see if it reveals anything, you would need to use 2.4 for that though.

Actions #2

Updated by Marcel Mayer over 7 years ago

The addresses are not changing. They stay.
What do you preffer or suggest? Updating would be ok for me. Is it possible via Webinterface? Or do I have to flash a new image?

Actions #3

Updated by Phillip Davis over 7 years ago

Go to System->Updates->Update Settings, change Branch to "Development Snapshots" and save.
Now it will show an upgrade to 2.3.3

Actions #4

Updated by Martin Wasley over 7 years ago

Can I make a suggestion. Before you do any major revision updates save a copy of your config file in case you wish to revert or test other versions, a little painful experience taught me this lesson. Once you have migrated to 2.3.3 see how things are with that version, then if you wish you can then try 2.4, again remember to back up the config before migrating. When you get to 2.4 let me know and I can send you my test version of the dhcp6c suite.

Note... If you go to 2.4 do some research before you do to make sure that any packages you use are compatible with 2.4, pfBlocker for example can cause problems at present as it's not yet been made totally 2.4 compatible.

Actions #5

Updated by Rick Strangman over 7 years ago

Does this issue seem similar to bug #6000? If so I can probably help.
Rick

Actions #6

Updated by Martin Wasley over 7 years ago

#6000 is about virtual IP's or am I missing something... quite possible at my age. :)

Actions #7

Updated by Marcel Mayer over 7 years ago

Thank you Rick Strangman for the reply. I don't think, the issus are similar.
The Update will be scheduled for next weekend. So I report in here my results.

Regards^^

Actions #8

Updated by Martin Wasley over 7 years ago

Marcel Kallinger

It would be interesting to see what your dhcp6 is doing at the same time, could you post a snippit of both logs. I see the same entries as you, but only once every thirty minutes as dhcp6c renews on timeout; the ISP sets that period. Now, my dhcp6c client gives me the timeout in the logs as shown here, my entries show newest at top:

Dec 5 10:18:43 dhcp6c 91606 update a prefix 2a02:xxx:xxx:xxxx::/56 pltime=3600, vltime=3600
Dec 5 10:18:43 dhcp6c 91606 dhcp6c Received INFO
Dec 5 10:18:43 dhcp6c 91606 send renew to ff02::1:2%re0

and from the general log:

Dec 5 10:18:47 xinetd 24097 Reconfigured: new=0 old=1 dropped=0 (services)
Dec 5 10:18:47 xinetd 24097 readjusting service 6969-udp
Dec 5 10:18:47 xinetd 24097 Swapping defaults
Dec 5 10:18:47 xinetd 24097 Starting reconfiguration
Dec 5 10:18:46 check_reload_status Reloading filter
Dec 5 10:18:46 php-fpm 55520 /rc.newwanipv6: Removing static route for monitor fe80::xxx:xxxx:xxxx:xxxx and adding a new route through fe80::xxx:xxxx:xxxx:xxxx%re0
Dec 5 10:18:46 php-fpm 55520 /rc.newwanipv6: ROUTING: setting IPv6 default route to fe80::xxx:xxxx:xxxx:xxxx%re0
Dec 5 10:18:43 php-fpm 55520 /rc.newwanipv6: rc.newwanipv6: on (IP address: fe80::xxx:xxxx:xxxx:xxxx%re0) (interface: wan) (real interface: re0).
Dec 5 10:18:43 php-fpm 55520 /rc.newwanipv6: rc.newwanipv6: Info: starting on re0.

Naturally, although the allocation time is sixty minutes, the renew happens at half that period.

Actions #9

Updated by Rick Strangman over 7 years ago

Sorry, seems like bug #6000 has been deleted and i was not refering to feature #6000

Actions #10

Updated by Marcel Mayer over 7 years ago

I started to do the test today and realised, that IPv6 is working for the moment without the described issue.
Used this how-to in past: https://moerbst.wordpress.com/2016/07/31/ipv6mit-pfsense-an-dsl-der-telekom/
So the ticket can be closed (for the moment).

Are there informations or logfiles you still want to have?

Actions #11

Updated by Marcel Mayer over 7 years ago

What I found and confuses me are this lines in general log:

/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid re2_vlan1 re2_vlan2 re2_vlan4' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.3.4 Copyright 2004-2016 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpd.conf Database file: /var/db/dhcpd.leases PID file: /var/run/dhcpd.pid Wrote 68 leases to leases file. Listening on BPF/re2_vlan4/00:0d:b9:33:95:42/192.168.4.0/24 Sending on BPF/re2_vlan4/00:0d:b9:33:95:42/192.168.4.0/24 Listening on BPF/re2_vlan2/00:0d:b9:33:95:42/192.168.2.0/24 Sending on BPF/re2_vlan2/00:0d:b9:33:95:42/192.168.2.0/24 Listening on BPF/re2_vlan1/00:0d:b9:33:95:42/192.168.1.0/24 Sending on BPF/re2_vlan1/00:0d:b9:33:95:42/192.168.1.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that ther

More loge content in attached file

Actions #12

Updated by Marcel Mayer over 7 years ago

Found the solution for that.
The Leasetable hold two entries (no idea why). After deleting them, everything now works like a charm.

Thank you guys, for the support, merry christmas and a happy new year^^

Actions #13

Updated by Marcel Mayer over 7 years ago

Some confusing log entrys are still there. See attached file.

Actions #14

Updated by Martin Wasley over 7 years ago

What do you have selected in DNS Resolver/General - Network Interfaces and Outgoing interfaces?

Actions #15

Updated by Marcel Mayer over 7 years ago

DNS Resolver/General - Network Interfaces and Outgoing interfaces = both are set to "All"

Removed dhcpd from monitoring by watchdog. Now the log looks better (see attached).

Actions #17

Updated by Martin Wasley over 7 years ago

That now looks pretty normal.

Actions #18

Updated by → luckman212 over 7 years ago

I think System Watchdog needs some dusting off. I stopped using it a while ago as it seems to cause more problems than it solves at this point.

Actions #19

Updated by Kill Bill over 7 years ago

The only thing that the watchdog does is setting up a cronjob which in turn checks every minute whether configured services are running. If not, it starts them. Cannot imagine what dusting off could be done there. Having stuff like https://mmonit.com/monit/ available would be much better, if someone wants to package it.

Actions #20

Updated by Marcel Mayer over 7 years ago

@Kill Bill: Deactivating watchdog solves my issue. It realy seems to make some trouble running it at moment. Especailly with DHCP.

Actions #21

Updated by Arno Gramatke over 7 years ago

I still have this issue. I am not sure whether this has to do with the watchdog at all. When I enable IPv6 from the web interface for the WAN interface and kill the dhcp6c from the command line and restart it manually, I get the log output as shown in the two attached files (one shows what is happening after I deleted the IPv6 DUID, the other one shows what is happening with an existing DUID).

I would expect the dhcp6c do its thing and then go to sleep until half of the lease time is over. And then renew the IPv6 lease.

This is how I started dhcp6c:

  1. /usr/local/sbin/dhcp6c -D -f -c /var/etc/dhcp6c_wan.conf -p /var/run/dhcp6c_pppoe0.pid pppoe0
Actions #22

Updated by Arno Gramatke over 7 years ago

And I think it's important that this seems to be a problem with dhcp6c and NOT dhcp6d.

Actions #23

Updated by Martin Wasley over 7 years ago

Don't delete the DUID mid session, its pointless, dhcp6c will generate a new one, which means that your ISP then sees another DUID which can confuse the BNG. Does your ISP supply a router? If so can you do a wireshark packet capture to grab the ipv6 dhcp packets and post them here? Can you also post your v6 configuration settings.

The server was replying with no addresses which is interesting, dhcp6c then just recycles as the timeout is 0. Do you also know exactly what options your ISP requires?

Actions #24

Updated by Arno Gramatke over 7 years ago

The ISP (Deutsche Telekom) doesn't supply a router, so I can't do any Wireshark capture there. The German Telekom uses PPPoE for IPv4 and DHCP6 for IPv6. The DHCP6 information should be requested through the IPv4 connection. A /64 prefix should be handed out to be used on the WAN interface and another /56 prefix is handed out for the LAN side.

Please note that the Telekom is currently switching customers from their old backbone network to a new one called BNG (yes, very generic, but that's what they call it ...). The IPv6 behaviour should be the same for both backbones and it worked for me without issues while my line was connected to the old backbone. After my line was transferred to the BNG IPv6 showed the behaviour I am reporting.

It seems that there is something strange going on with the BNG's IPv6 implementation as several router manufacturers had to update their firmware for IPv6 to work with the BNG. The most notable example being bintec who had to update a firmware for a business router they build for the German Telekom. From a forum entry at the Telekom's customer forum I could gather that during prefix delegation the router should request a renew instead of requesting a new prefix. This would be something that wouldn't work at the moment. Does this make sense?

The configuration settings of the WAN interface are as follows:

=== General Configuration ===
Enable Interface: checked
Description: WAN
IPv4 Configuration Type: PPPoE
IPv6 Configuration Type: DHCP6
MAC Address: <empty>
MTU: <empty>
MSS: <empty>

=== DHCP6 Client Configuration ===
Options:
Advanced Configuration: unchecked
Configuration Override: unchecked
Use IPv4 connectivity as parent interface: checked
Request only an IPv6 prefix: Unchecked
DHCPv6 Prefix Delegation size: 56
Send IPv6 prefix hint: checked
debug: unchecked

=== PPPoE Configuration ===
Username: <username>
Password: <password>
Service name: <empty>
Dial on demand: unchecked
Idle timeout: <empty>
Periodic reset: Disabled
Advanced and MLPP: No changes made here

=== Reserved Networks ===
Block private networks and loopback addresses: checked
Block bogon networks: checked

Actions #25

Updated by Marcel Mayer over 7 years ago

In my opinion we are not talking about a bug any more.
The problem seems to be a missconfiguration ...

For DTAG "Request only an IPv6 prefix: Unchecked" needs to be checked.

Actions #26

Updated by Arno Gramatke over 7 years ago

When I check "Request only an IPv6 prefix" the WAN interface uses the first /64 prefix (prefix ID 0) out of the /56 prefix that is delegated. But the WAN interface should use the /64 prefix that is being given to the router for the transport segment. If you check this "Request only an IPv6 prefix" you can only start at prefix ID 1 for the LAN interfaces (instead of 0).

BTW: The tutorial that you mentioned earlier doesn't state this either.

It does work, though. But I think this is not the way it is supposed to work. As far as I know DTAG hands out a /64 prefix for the transport segment (i.e. to use this on the WAN side) and another /56 prefix that can be used on the LAN side as several /64 prefixes.

On this page a user did some wireshark captures: https://blog.webernetz.net/2015/05/19/telekom-dual-stack-verbindungsaufbau/

Maybe some of the information in there is helpful? In the middle of the post are some pictures of the wireshark output.

Actions #27

Updated by Marcel Mayer over 7 years ago

The Address given to the WAN interface is more or less irrelevant, cause it's not realy necessary for your firewall rules.
Devices behind the firewall will be adressed directly. So I think, there will be no problem and starting with ID 1 will be ok.

Actions #28

Updated by Martin Wasley over 7 years ago

I'm confused as to why they would issue a /64 PD on the wan.

dhcp6c does ask for renew, as shown in the attached image.

Actions #29

Updated by Arno Gramatke over 7 years ago

It does work for me now - Marcel's hint to check "Request only an IPv6 prefix" was indeed correct. After two more reboots it finally worked.

The IPv6 address of the WAN interface is handed out via Router Advertisement after the interface identifiers have been exchanged in the PPP IPV6CP stage. After that the /56 prefix is delegated via DHCP6. So to check the "Request only an IPv6 prefix" is correct.

From my point of view this ticket can be closed.

Actions #30

Updated by Marcel Mayer over 7 years ago

Happy to helped. Last good action for 2016.
Happy new year^^

Actions #31

Updated by Jim Pingle over 4 years ago

  • Category set to Interfaces
  • Status changed from New to Closed
Actions

Also available in: Atom PDF