https://redmine.pfsense.org/https://redmine.pfsense.org/favicon.ico?16780521162016-05-03T01:52:07ZpfSense bugtrackerpfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=269002016-05-03T01:52:07ZChris Buechlercbuechler@gmail.com
<ul><li><strong>Project</strong> changed from <i>pfSense</i> to <i>pfSense Packages</i></li><li><strong>Category</strong> set to <i>Quagga OSPF</i></li><li><strong>Target version</strong> deleted (<del><i>2.3.1</i></del>)</li></ul> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=270322016-05-09T01:19:13Zjeroen van breedamhepnix@gmail.com
<ul></ul><p>for me, downgrading to older version seems to solve all issues. no confirmation if this is the case for the OP of the forum post.</p>
<p>working version: <a class="external" href="http://pkg.freebsd.org/freebsd:10:x86:64/release_3/All/quagga-0.99.24.1_2.txz">http://pkg.freebsd.org/freebsd:10:x86:64/release_3/All/quagga-0.99.24.1_2.txz</a></p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=271732016-05-19T03:23:44Zjeroen van breedamhepnix@gmail.com
<ul></ul><p>OP hasn't found the time to respond to the post.<br />a different forum member has confirmed reverting to version above solves it.</p>
<p>Could anyone revert the update to before 1.x ?</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=272702016-05-25T01:02:58Zjeroen van breedamhepnix@gmail.com
<ul></ul><p>A different forum member has came across this issue & has confirmed that reverting to 0.99.24.1 fixes the problem.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=273612016-06-02T13:56:41Zjeroen van breedamhepnix@gmail.com
<ul></ul><p>Any of the coredevs been able to replicate this?</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=273942016-06-03T09:10:53ZReqlez Guy
<ul></ul><p>jeroen van breedam wrote:</p>
<blockquote>
<p>Any of the coredevs been able to replicate this?</p>
</blockquote>
<p>I have ( but I'm not a Dev) And everybody else who actually uses OSPF and tried to failover links has.</p>
<p>This is a core functionality bug and it needs more attention.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=273952016-06-03T10:47:43ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>I see the routes sometimes (but not always) marked as Kernel routes in the Zebra routing table, but I have not seen this be an actual problem for routing. So it's possible the K routes are not actually the source of your problems.</p>
<p>I have a test setup here with a central router, and two clients that each have two WANs, so four OpenVPN instances total on the "server" (two each on clients), and each of them have OSPF. If I kill a preferred WAN on a client, the K route goes inactive, quagga selects the O route and traffic flows again. When it recovers, it continues to work as well, but with the expected traffic hiccup as OSPF switches things around.</p>
<p>It's possible there is some misconfiguration happening that is handled differently in quagga 1.0.x vs 0.99.x, but as far as I can see, it works when configured properly. Continue the discussion on the forum thread but please post the contents of the zebra.conf and quagga.conf files (masking/removing passwords), and preferably post the entire output of the status tab in an attachment on the forum thread.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=273962016-06-03T12:26:32Zjeroen van breedamhepnix@gmail.com
<ul></ul><p><a class="user active" href="https://redmine.pfsense.org/users/10">Jim Pingle</a></p>
<p>i've updated reply <a class="issue tracker-4 status-5 priority-4 priority-default closed" title="Todo: [ Fit123 ] Captive Portal (Closed)" href="https://redmine.pfsense.org/issues/7">#7</a> to include the config of client side & server side</p>
<p>the status of before/after was there already. <br />I'm currently not in a position to dump the entire contents of status before/after because employees&bosses don't like it much when things stop working =)</p>
<p>Atleast 4 people have reported similar issue's. It's odd that it doesn't occur in the test-setup - we must be missing something here ...</p>
<p>If you need more data i'll try to provide it.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=275332016-06-09T16:15:27Zjeroen van breedamhepnix@gmail.com
<ul></ul><p>seems like someone has found a way to reproduce consistenly. (this is currently not verified by others)</p>
<p><a class="external" href="https://forum.pfsense.org/index.php?topic=111108.msg630396#msg630396">https://forum.pfsense.org/index.php?topic=111108.msg630396#msg630396</a></p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=278282016-06-25T13:41:31ZReqlez Guy
<ul></ul><p>Okay ... I have to set-up already that if i upgrade the package back to the new one, the issue will happen. Jim ... can I just privately send you the config files of the routers somehow and you can just take a look ? or what info do you need that i can safely post on this bug tracker ?</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=285632016-08-10T08:46:26ZJuri Dmitrijev
<ul></ul><p>Any update on the topic?</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=285642016-08-10T08:59:56ZJim Pingle
<ul></ul><p>Someone who can reproduce it reliably needs to get the details of how to reproduce it reported to the Quagga project directly.</p>
<p>We can't reproduce it reliably here, and it does not appear to be a bug in any of our code, but in the current version of Quagga on FreeBSD.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=290402016-10-10T22:58:51ZReqlez Guy
<ul></ul><p>Jim Pingle wrote:</p>
<blockquote>
<p>Someone who can reproduce it reliably needs to get the details of how to reproduce it reported to the Quagga project directly.</p>
<p>We can't reproduce it reliably here, and it does not appear to be a bug in any of our code, but in the current version of Quagga on FreeBSD.</p>
</blockquote>
<p>I emailed the quagga users list and got a response <a class="external" href="https://lists.quagga.net/pipermail/quagga-users/2016-October/014474.html">https://lists.quagga.net/pipermail/quagga-users/2016-October/014474.html</a></p>
<p>Above is the thread regarding this. Also ... I know <a class="user active" href="https://redmine.pfsense.org/users/10">Jim Pingle</a> have provided a "no routing packages restart" patch for 2.3.1 ... but ... every time we update pfsense this is not going to work ... is it possible for this "no routing packages restart" to be made into an option under advenced settings in pfsense ? I have an issue with unstable links that bring down the network even if those links are lower priority because it seems that every time zebra gets rebooted the routes are wiped out and there is a period of a few seconds where there is no traffic while zebra restarts and learns the routes again, very annoying.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=290742016-10-14T12:56:50ZNate Baker
<ul></ul><p>Jim Pingle wrote:</p>
<blockquote>
<p>Someone who can reproduce it reliably needs to get the details of how to reproduce it reported to the Quagga project directly.</p>
<p>We can't reproduce it reliably here, and it does not appear to be a bug in any of our code, but in the current version of Quagga on FreeBSD.</p>
</blockquote>
<p>We are having this issue as well. It looks like to reproduce it, the quagga services (probably just zebra) need to be restarted. When that happens the kernel routes show up in the zebra routes, and from that point on things don't work properly. So it seems like there are two problems:</p>
<p>1) Every time a change is made to OSPF the services are restarted with the new config. This can be disruptive, and the Quagga team says it shouldn't be necessary. Also it triggers the problem with Quagga.<br />2) When Quagga is restarted, the kernel routes (which it put there before it was restarted) are pulled into Zebra, and will always take precedence until the firewall is restarted.</p>
<p>If we restart the firewall and never touch the Quagga settings things work fine. So to fix number 1, is it possible to write the configuration files and change the Quagga configuration by connecting to the Quagga VTYs, instead of restarting it? It seems like number 2 needs to be fixed by Quagga.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=290752016-10-14T14:04:05ZReqlez Guy
<ul></ul><p>Nate Baker wrote:</p>
<blockquote>
<p>Jim Pingle wrote:</p>
<blockquote>
<p>Someone who can reproduce it reliably needs to get the details of how to reproduce it reported to the Quagga project directly.</p>
<p>We can't reproduce it reliably here, and it does not appear to be a bug in any of our code, but in the current version of Quagga on FreeBSD.</p>
</blockquote>
<p>We are having this issue as well. It looks like to reproduce it, the quagga services (probably just zebra) need to be restarted. When that happens the kernel routes show up in the zebra routes, and from that point on things don't work properly. So it seems like there are two problems:</p>
<p>1) Every time a change is made to OSPF the services are restarted with the new config. This can be disruptive, and the Quagga team says it shouldn't be necessary. Also it triggers the problem with Quagga.<br />2) When Quagga is restarted, the kernel routes (which it put there before it was restarted) are pulled into Zebra, and will always take precedence until the firewall is restarted.</p>
<p>If we restart the firewall and never touch the Quagga settings things work fine. So to fix number 1, is it possible to write the configuration files and change the Quagga configuration by connecting to the Quagga VTYs, instead of restarting it? It seems like number 2 needs to be fixed by Quagga.</p>
</blockquote>
<p>I'm working with Martin from Quagga and collecting debug logs this weekend. He thinks that "Quagga is being restarted in some hard way that won't allow it to clean up routes" but he also says he doesnt understand why Quagga needs to be restarted in pfsense in the first place when links change.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=291432016-10-28T00:52:43ZReqlez Guy
<ul></ul><p>So far the only thing I got from Martin was that -9 is not a nice way to stop quagga and could cause the issues... Also I saw 1.1 release of quagga had this commit ... not sure if this is related:</p>
<p>commit 7e73eb740f3c52a5b7c0ae9c2cd33b486d885552<br />Author: Timo Teräs <<a class="email" href="mailto:timo.teras@iki.fi">timo.teras@iki.fi</a>><br />Date: Sat Apr 9 17:22:32 2016 +0300</p>
<pre><code>zebra: handle multihop nexthop changes properly</code></pre>
<pre><code>The rib entries are normally added and deleted when they are<br /> changed. However, they are modified in placae when the nexthop<br /> reachability changes. This fixes to:<br /> - properly detect nexthop changes from nexthop_active_update()<br /> calls from rib_process()<br /> - rib_update_kernel() to not reset FIB flags when a RIB entry<br /> is being modifed (old and new RIB are same)<br /> - improves the "show ip route &lt;prefix&gt;" output to display<br /> both ACTIVE and FIB flags for each nexthop</code></pre>
<pre><code>Fixes: 325823a5 "zebra: support FIB override routes" <br /> Signed-off-by: Timo Teräs &lt;<a class="email" href="mailto:timo.teras@iki.fi">timo.teras@iki.fi</a>&gt;<br /> Reported-By: Igor Ryzhov &lt;<a class="email" href="mailto:iryzhov@nfware.com">iryzhov@nfware.com</a>&gt;<br /> Tested-by: NetDEF CI System &lt;<a class="email" href="mailto:cisystem@netdef.org">cisystem@netdef.org</a>&gt;</code></pre> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=305232017-01-09T22:06:50Zwinmasta winmastawinmasta@yandex.ru
<ul></ul><p>Affected me too. I tried settings with OpenVPN server + OpenVPN client.</p>
<p>Both:<br />Pfsense 2.3.2-RELEASE-p1<br />Quagga_OSPF 0.6.16 (quagga-1.0.20160315)</p>
<p><strong>Server:</strong></p>
<p>Quagga ospfd.conf</p>
<pre>
# This file was created by the pfSense package manager. Do not edit!
password ***
log syslog
interface ovpns1
ip ospf cost 10
interface ovpns2
ip ospf cost 20
router ospf
ospf router-id 192.168.3.7
log-adjacency-changes detail
redistribute connected
network 10.0.8.0/24 area 0.0.0.1
</pre><br />Quagga zebra.conf
<pre>
# This file was created by the pfSense package manager. Do not edit!
password ***
log syslog
</pre>
<p>sudo cat /var/etc/openvpn/server1.conf</p>
<pre>
dev ovpns1
verb 1
dev-type tun
tun-ipv6
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher BF-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 1.2.3.4
tls-server
server 10.0.8.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server1
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'server' 1"
lport 1194
management /var/etc/openvpn/server1.sock unix
push "route 192.168.3.0 255.255.255.0"
client-to-client
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /etc/dh-parameters.1024
tls-auth /var/etc/openvpn/server1.tls-auth 0
comp-lzo adaptive
passtos
persist-remote-ip
float
topology subnet
route 192.168.1.0 255.255.255.0 10.0.8.1
route 192.168.0.0 255.255.255.0 10.0.8.1
route 192.168.5.0 255.255.255.0 10.0.8.1
route 192.168.8.0 255.255.255.0 10.0.8.1
route 192.168.9.0 255.255.255.0 10.0.8.1
route 192.168.10.0 255.255.255.0 10.0.8.1
</pre>
<p>sudo cat /var/etc/openvpn/server2.conf</p>
<pre>
dev ovpns2
verb 1
dev-type tun
tun-ipv6
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher BF-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 5.6.7.8
tls-server
server 10.1.8.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'server' 1"
lport 1194
management /var/etc/openvpn/server2.sock unix
push "route 192.168.3.0 255.255.255.0"
client-to-client
ca /var/etc/openvpn/server2.ca
cert /var/etc/openvpn/server2.cert
key /var/etc/openvpn/server2.key
dh /etc/dh-parameters.1024
tls-auth /var/etc/openvpn/server2.tls-auth 0
passtos
persist-remote-ip
float
topology subnet
</pre>
<p><strong>Client:</strong></p>
<p>Quagga ospfd.conf<br /><pre>
# This file was created by the pfSense package manager. Do not edit!
password ***
log syslog
interface ovpnc1
ip ospf cost 10
interface ovpnc2
ip ospf cost 20
router ospf
ospf router-id 192.168.8.1
log-adjacency-changes detail
redistribute connected
timers throttle spf 200 2 20
network 10.0.8.0/24 area 0.0.0.1
</pre></p>
<p>Quagga zebra.conf</p>
<pre>
# This file was created by the pfSense package manager. Do not edit!
password ***
log syslog
</pre>
<p>sudo cat /var/etc/openvpn/client1.conf</p>
<pre>
dev ovpnc1
verb 1
dev-type tun
tun-ipv6
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher BF-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 3.21.7.21
tls-client
client
lport 0
management /var/etc/openvpn/client1.sock unix
remote 1.2.3.4 1194
route 192.168.3.0 255.255.255.0
ca /var/etc/openvpn/client1.ca
cert /var/etc/openvpn/client1.cert
key /var/etc/openvpn/client1.key
tls-auth /var/etc/openvpn/client1.tls-auth 1
comp-lzo adaptive
passtos
resolv-retry infinite
</pre>
<p>sudo cat /var/etc/openvpn/client2.conf</p>
<pre>
dev ovpnc2
verb 1
dev-type tun
tun-ipv6
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher BF-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 3.21.7.21
tls-client
client
lport 0
management /var/etc/openvpn/client2.sock unix
remote 5.6.7.8 1194
route 192.168.3.0 255.255.255.0
ca /var/etc/openvpn/client2.ca
cert /var/etc/openvpn/client2.cert
key /var/etc/openvpn/client2.key
tls-auth /var/etc/openvpn/client2.tls-auth 1
passtos
resolv-retry infinite
</pre> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=308682017-01-25T04:31:14ZKill Bill
<ul></ul><p><a class="external" href="https://github.com/pfsense/FreeBSD-ports/pull/265">https://github.com/pfsense/FreeBSD-ports/pull/265</a> - that's not a real solution obviously, so kindly leave this bug open even if merged.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=318372017-02-28T08:06:42ZHanno Stock
<ul></ul><p>Looks like Zebra sets RTF_PROTO1 flag on the routes it installs in the routing table.</p>
<p>So I assume in order to get the old behavior it would be needed to flush the routes marked with RTF_PROTO1 in the restart script?</p>
<p>I still think configuration changes are better handled by connecting to the daemons - however in case of a last resort kind of restart this could help.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=351012017-11-14T15:50:53ZJim Pingle
<ul></ul><p>If this still happens with Quagga, give FRR a try instead.</p> pfSense Packages - Bug #6305: Quagga problems updating routes / mistakenly showing "kernel"-routes while they are nothttps://redmine.pfsense.org/issues/6305?journal_id=411292019-08-13T09:03:46ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li></ul>