https://redmine.pfsense.org/
https://redmine.pfsense.org/favicon.ico?1678052116
2019-04-03T07:22:09Z
pfSense bugtracker
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=40335
2019-04-03T07:22:09Z
Jim Pingle
<ul><li><strong>Category</strong> set to <i>Multi-WAN</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li><li><strong>Target version</strong> set to <i>2.5.0</i></li><li><strong>Affected Version</strong> set to <i>2.4.x</i></li><li><strong>Affected Architecture</strong> <i>All</i> added</li><li><strong>Affected Architecture</strong> deleted (<del><i></i></del>)</li></ul>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=41000
2019-07-22T19:04:12Z
Chris Linstruth
<ul></ul><p>Adding these log snippets. They are groups of dpinger gateway logs followed by the system logs for the corresponding periods.<br />I marked the actual pfSense gateway group changes with an asterisk. Note how, in this case, the gateway remained down until the administrator logged in and manually did something to make the gateway group be reevaluated. Editing and saving a gateway I believe despite having recovered and stabilized hours ago.</p>
<pre>
Jun 27 14:49:07 firewall-name dpinger: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 203.0.113.1 bind_addr 203.0.113.200 identifier "OPT1_DHCP "
Jul 1 01:59:15 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 1328830us stddev 3086974us loss 0%
Jul 1 02:01:11 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 43794us stddev 199505us loss 17%
Jul 1 01:59:15 firewall-name rc.gateway_alarm[70843]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:1328.830ms RTTsd:3086.974ms Loss:0%)
Jul 1 01:59:15 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 01:59:15 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 01:59:15 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 01:59:15 firewall-name check_reload_status: Reloading filter
*Jul 1 01:59:16 firewall-name php-fpm: /rc.openvpn: MONITOR: WANGW is down, omitting from routing group Primary 198.51.100.109|198.51.100.110|WANGW|1328.911ms|3086.939ms|0.0%|down*
Jul 1 01:59:16 firewall-name php-fpm: /rc.openvpn: Gateway, switch to: OPT1_DHCP
Jul 1 01:59:16 firewall-name php-fpm: /rc.openvpn: Default gateway setting Interface OPT1_DHCP Gateway as default.
Jul 1 01:59:16 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 01:59:16 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:01:11 firewall-name rc.gateway_alarm[69387]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:43.794ms RTTsd:199.505ms Loss:17%)
Jul 1 02:01:11 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:01:11 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:01:11 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:01:11 firewall-name check_reload_status: Reloading filter
Jul 1 02:01:12 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:01:12 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:01:14 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 46483us stddev 205767us loss 22%
Jul 1 02:02:36 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 3168568us stddev 1678463us loss 90%
Jul 1 02:03:30 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 1884347us stddev 2092328us loss 5%
Jul 1 02:04:25 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 2895us stddev 520us loss 0%
Jul 1 02:01:15 firewall-name rc.gateway_alarm[1952]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:46.483ms RTTsd:205.767ms Loss:22%)
Jul 1 02:01:15 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:01:15 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:01:15 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:01:15 firewall-name check_reload_status: Reloading filter
Jul 1 02:01:16 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:01:16 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:02:36 firewall-name rc.gateway_alarm[66874]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:3168.568ms RTTsd:1678.463ms Loss:90%)
Jul 1 02:02:36 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:02:36 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:02:36 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:02:36 firewall-name check_reload_status: Reloading filter
Jul 1 02:02:37 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:02:37 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:03:30 firewall-name rc.gateway_alarm[33286]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:1884.347ms RTTsd:2092.328ms Loss:5%)
Jul 1 02:03:30 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:03:30 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:03:30 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:03:30 firewall-name check_reload_status: Reloading filter
Jul 1 02:03:31 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:03:31 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:04:25 firewall-name rc.gateway_alarm[2150]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:2.895ms RTTsd:.520ms Loss:0%)
Jul 1 02:04:25 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:04:25 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:04:25 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:04:25 firewall-name check_reload_status: Reloading filter
*Jul 1 02:04:26 firewall-name php-fpm: /rc.openvpn: 352MONITOR: WANGW is available now, adding to routing group Primary 198.51.100.109|198.51.100.110|WANGW|2.93ms|0.589ms|0.0%|none*
Jul 1 02:04:26 firewall-name php-fpm: /rc.openvpn: Gateway, switch to: WANGW
Jul 1 02:04:26 firewall-name php-fpm: /rc.openvpn: Default gateway setting WAN Gateway as default.
Jul 1 02:04:26 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:04:26 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:05:19 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 547071us stddev 1647180us loss 0%
Jul 1 02:06:24 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 11531us stddev 55680us loss 21%
Jul 1 02:07:44 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 3116322us stddev 1833237us loss 90%
Jul 1 02:08:38 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 821647us stddev 1789602us loss 5%
Jul 1 02:08:54 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 160206us stddev 666241us loss 0%
Jul 1 02:05:19 firewall-name rc.gateway_alarm[68226]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:547.071ms RTTsd:1647.180ms Loss:0%)
Jul 1 02:05:19 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:05:19 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:05:19 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:05:19 firewall-name check_reload_status: Reloading filter
*Jul 1 02:05:20 firewall-name php-fpm: /rc.openvpn: MONITOR: WANGW is down, omitting from routing group Primary 198.51.100.109|198.51.100.110|WANGW|547.077ms|1647.178ms|0.0%|down*
Jul 1 02:05:20 firewall-name php-fpm: /rc.openvpn: Gateway, switch to: OPT1_DHCP
Jul 1 02:05:20 firewall-name php-fpm: /rc.openvpn: Default gateway setting Interface OPT1_DHCP Gateway as default.
Jul 1 02:05:20 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:05:20 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:06:24 firewall-name rc.gateway_alarm[31317]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:11.531ms RTTsd:55.680ms Loss:21%)
Jul 1 02:06:24 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:06:24 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:06:24 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:06:24 firewall-name check_reload_status: Reloading filter
Jul 1 02:06:25 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:06:25 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:07:44 firewall-name rc.gateway_alarm[97435]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:3116.322ms RTTsd:1833.237ms Loss:90%)
Jul 1 02:07:44 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:07:44 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:07:44 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:07:44 firewall-name check_reload_status: Reloading filter
Jul 1 02:07:45 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:07:45 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:08:38 firewall-name rc.gateway_alarm[63258]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:821.647ms RTTsd:1789.602ms Loss:5%)
Jul 1 02:08:38 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:08:38 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:08:38 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:08:38 firewall-name check_reload_status: Reloading filter
Jul 1 02:08:39 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:08:39 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:08:54 firewall-name rc.gateway_alarm[94197]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:160.206ms RTTsd:666.241ms Loss:0%)
Jul 1 02:08:54 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:08:54 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:08:54 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:08:54 firewall-name check_reload_status: Reloading filter
*Jul 1 02:08:55 firewall-name php-fpm: /rc.openvpn: 352MONITOR: WANGW is available now, adding to routing group Primary 198.51.100.109|198.51.100.110|WANGW|94.413ms|446.542ms|0.0%|none*
Jul 1 02:08:56 firewall-name php-fpm: /rc.openvpn: Gateway, switch to: WANGW
Jul 1 02:08:56 firewall-name php-fpm: /rc.openvpn: Default gateway setting WAN Gateway as default.
Jul 1 02:08:56 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:08:56 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:12:18 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 501264us stddev 1540741us loss 0%
Jul 1 02:12:37 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 499349us stddev 1534363us loss 0%
Jul 1 02:12:18 firewall-name rc.gateway_alarm[67803]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:501.264ms RTTsd:1540.741ms Loss:0%)
Jul 1 02:12:18 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:12:18 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:12:18 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:12:18 firewall-name check_reload_status: Reloading filter
*Jul 1 02:12:19 firewall-name php-fpm[351]: /rc.openvpn: MONITOR: WANGW has high latency, omitting from routing group Primary 198.51.100.109|198.51.100.110|WANGW|497.108ms|1534.978ms|0.0%|delay*
Jul 1 02:12:19 firewall-name php-fpm[351]: /rc.openvpn: Gateway, switch to: OPT1_DHCP
Jul 1 02:12:19 firewall-name php-fpm[351]: /rc.openvpn: Default gateway setting Interface OPT1_DHCP Gateway as default.
Jul 1 02:12:19 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:12:19 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:12:37 firewall-name rc.gateway_alarm[99477]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:499.349ms RTTsd:1534.363ms Loss:0%)
Jul 1 02:12:37 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:12:37 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:12:37 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:12:37 firewall-name check_reload_status: Reloading filter
Jul 1 02:12:38 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:12:38 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:12:42 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 502376us stddev 1533719us loss 0%
Jul 1 02:12:54 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 8749us stddev 39389us loss 10%
Jul 1 02:12:42 firewall-name rc.gateway_alarm[32265]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:502.376ms RTTsd:1533.719ms Loss:0%)
Jul 1 02:12:42 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:12:42 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:12:42 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:12:42 firewall-name check_reload_status: Reloading filter
Jul 1 02:12:44 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:12:44 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:12:54 firewall-name rc.gateway_alarm[64010]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:8.749ms RTTsd:39.389ms Loss:10%)
Jul 1 02:12:54 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:12:54 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:12:54 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:12:54 firewall-name check_reload_status: Reloading filter
Jul 1 02:12:55 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:12:55 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:13:00 firewall-name rc.gateway_alarm[94382]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:9.583ms RTTsd:41.807ms Loss:21%)
Jul 1 02:13:00 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 9583us stddev 41807us loss 21%
Jul 1 02:14:36 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 2507259us stddev 1635716us loss 90%
Jul 1 02:15:30 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 441793us stddev 966826us loss 5%
Jul 1 02:15:34 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 380640us stddev 848533us loss 0%
Jul 1 02:13:00 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:13:00 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:13:00 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:13:00 firewall-name check_reload_status: Reloading filter
Jul 1 02:13:01 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:13:01 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:14:36 firewall-name rc.gateway_alarm[93227]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:2507.259ms RTTsd:1635.716ms Loss:90%)
Jul 1 02:14:36 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:14:36 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:14:36 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:14:36 firewall-name check_reload_status: Reloading filter
Jul 1 02:14:37 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:14:37 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:15:30 firewall-name rc.gateway_alarm[61727]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:441.793ms RTTsd:966.826ms Loss:5%)
Jul 1 02:15:30 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:15:30 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:15:30 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:15:30 firewall-name check_reload_status: Reloading filter
Jul 1 02:15:31 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:15:31 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:15:34 firewall-name rc.gateway_alarm[92708]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:380.640ms RTTsd:848.533ms Loss:0%)
Jul 1 02:15:34 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:15:34 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:15:34 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:15:34 firewall-name check_reload_status: Reloading filter
Jul 1 02:15:35 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:15:35 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:16:56 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 729582us stddev 1662029us loss 0%
Jul 1 02:18:05 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 448842us stddev 1388060us loss 6%
Jul 1 02:16:56 firewall-name rc.gateway_alarm[55853]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:729.582ms RTTsd:1662.029ms Loss:0%)
Jul 1 02:16:56 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:16:56 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:16:56 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:16:56 firewall-name check_reload_status: Reloading filter
Jul 1 02:16:57 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:16:57 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:18:05 firewall-name rc.gateway_alarm[23620]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:448.842ms RTTsd:1388.060ms Loss:6%)
Jul 1 02:18:05 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:18:05 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:18:05 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:18:05 firewall-name check_reload_status: Reloading filter
Jul 1 02:18:06 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 515366us stddev 1517537us loss 7%
Jul 1 02:18:22 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 391842us stddev 1286779us loss 0%
Jul 1 02:18:06 firewall-name rc.gateway_alarm[26820]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:515.366ms RTTsd:1517.537ms Loss:7%)
Jul 1 02:18:06 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:18:06 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:18:06 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:18:06 firewall-name check_reload_status: Reloading filter
Jul 1 02:18:06 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:18:06 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:18:08 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:18:08 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:18:22 firewall-name rc.gateway_alarm[83385]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:391.842ms RTTsd:1286.779ms Loss:0%)
Jul 1 02:18:22 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:18:22 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:18:22 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:18:22 firewall-name check_reload_status: Reloading filter
Jul 1 02:18:23 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:18:23 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:19:02 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 833425us stddev 1822007us loss 0%
Jul 1 02:19:17 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 445433us stddev 1416569us loss 0%
Jul 1 02:19:02 firewall-name rc.gateway_alarm[47953]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:833.425ms RTTsd:1822.007ms Loss:0%)
Jul 1 02:19:02 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:19:02 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:19:02 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:19:02 firewall-name check_reload_status: Reloading filter
Jul 1 02:19:03 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:19:03 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:19:17 firewall-name rc.gateway_alarm[79365]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:445.433ms RTTsd:1416.569ms Loss:0%)
Jul 1 02:19:17 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:19:17 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:19:17 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:19:17 firewall-name check_reload_status: Reloading filter
Jul 1 02:19:18 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:19:18 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:21:29 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 13472us stddev 62844us loss 22%
Jul 1 02:22:32 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 2175721us stddev 1371078us loss 92%
Jul 1 02:23:27 firewall-name dpinger: WANGW 198.51.100.109: Alarm latency 528310us stddev 908015us loss 6%
Jul 1 02:23:41 firewall-name dpinger: WANGW 198.51.100.109: Clear latency 288361us stddev 628682us loss 0%
Jul 1 02:21:29 firewall-name rc.gateway_alarm[76265]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:13.472ms RTTsd:62.844ms Loss:22%)
Jul 1 02:21:29 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:21:29 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:21:29 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:21:29 firewall-name check_reload_status: Reloading filter
Jul 1 02:21:30 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:21:30 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:22:32 firewall-name rc.gateway_alarm[44123]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:2175.721ms RTTsd:1371.078ms Loss:92%)
Jul 1 02:22:32 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:22:32 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:22:32 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:22:32 firewall-name check_reload_status: Reloading filter
Jul 1 02:22:33 firewall-name php-fpm[351]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:22:33 firewall-name php-fpm[351]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:23:27 firewall-name rc.gateway_alarm[13090]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:1 RTT:528.310ms RTTsd:908.015ms Loss:6%)
Jul 1 02:23:27 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:23:27 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:23:27 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:23:27 firewall-name check_reload_status: Reloading filter
Jul 1 02:23:29 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:23:29 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 02:23:41 firewall-name rc.gateway_alarm[63462]: >>> Gateway alarm: WANGW (Addr:198.51.100.109 Alarm:0 RTT:288.361ms RTTsd:628.682ms Loss:0%)
Jul 1 02:23:41 firewall-name check_reload_status: updating dyndns WANGW
Jul 1 02:23:41 firewall-name check_reload_status: Restarting ipsec tunnels
Jul 1 02:23:41 firewall-name check_reload_status: Restarting OpenVPN tunnels/interfaces
Jul 1 02:23:41 firewall-name check_reload_status: Reloading filter
Jul 1 02:23:42 firewall-name php-fpm: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 1 02:23:42 firewall-name php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
Jul 1 12:47:39 firewall-name dpinger: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 198.51.100.109 bind_addr 198.51.100.110 identifier "WANGW "
Jul 1 12:47:39 firewall-name dpinger: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 203.0.113.1 bind_addr 203.0.113.200 identifier "OPT1_DHCP "
Jul 1 12:46:59 firewall-name php-fpm: /index.php: Successful login for user 'admin' from: 10.1.1.2 (Local Database)
*Jul 1 12:47:05 firewall-name php-fpm[351]: /system_gateways.php: 351MONITOR: WANGW is available now, adding to routing group Primary 198.51.100.109|198.51.100.110|WANGW|19.264ms|86.126ms|0.0%|none*
Jul 1 12:47:30 firewall-name check_reload_status: Syncing firewall
</pre>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=46468
2020-05-27T14:44:23Z
Jim Pingle
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Pull Request Review</i></li></ul><p>Proposed fix PR: <a class="external" href="https://github.com/pfsense/pfsense/pull/4336">https://github.com/pfsense/pfsense/pull/4336</a></p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48141
2020-09-28T02:00:08Z
Jörn Greszki
<ul></ul><p>Dear gents</p>
<p>is the behavior I describe</p>
<p><a class="external" href="https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken">https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken</a></p>
<p>related to your findings?</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48660
2020-10-14T11:01:30Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>Pull Request Review</i> to <i>Feedback</i></li><li><strong>Assignee</strong> set to <i>Renato Botelho</i></li></ul><p>PR has been merged. Thanks!</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48663
2020-10-14T11:05:11Z
Vladimir Voskoboynikov
vladv@vladv.org
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="Prevent possible race condition, fixes #9450 Instead of calling get_dpinger_status() again in th..." href="https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/5affb137561c74bb5559f0706c86c28a85b14557">5affb137561c74bb5559f0706c86c28a85b14557</a>.</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48691
2020-10-15T14:27:02Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li></ul><p>Some problems reported at <a class="external" href="https://forum.netgate.com/topic/157633/wan-gw-monitor/2">https://forum.netgate.com/topic/157633/wan-gw-monitor/2</a> after this was merged. I'll work on a fix</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48693
2020-10-15T14:45:07Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>Applied in changeset <a class="changeset" title="Fix #9450 Change default value for clear_alarm to true so cache file is used only once. We can'..." href="https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/0de6758e2893e4390acfa0b55e31b1dece231618">0de6758e2893e4390acfa0b55e31b1dece231618</a>.</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48702
2020-10-16T05:54:01Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>As mentioned on forums, it works</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48756
2020-10-16T09:20:53Z
Anonymous
<ul></ul><p>It now appears that gateways that are forced down in the gateway configs do not appear to be offlined.</p>
<p>Previously, such a gateway would show status as "Forced" or "Administrative" Down in the gateway status.</p>
<p>If gateway is configured with both "Disable Monitoring" & "Force State: Mark Gateway as Down" checkboxes selected, it will show as "Online (unmonitored)" on Gateway status page. If only "Force State" is checked, will show as "Pending" in gateway status. In both cases the gateway groups status will show this gateway as "Gathering data".</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=48757
2020-10-16T09:34:11Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>In Progress</i></li></ul>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=49008
2020-10-26T14:53:47Z
Renato Botelho
renato@netgate.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>I've reverted the cache changes and it should be fine now</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=49118
2020-11-02T04:00:19Z
Jörn Greszki
<ul></ul><p>I get for unknown reasons sometimes partial or full loss, but this is not the issue.</p>
<p>Nov 2 10:37:56 dpinger 16236 WAN_PHY1_IGB0GW 8.8.4.4: Alarm latency 0us stddev 0us loss 100%</p>
<p>Problem is that this status remains until any change to the gateway group is made - then it works immediately.</p>
<p>Tested with 2.5.0.a.20201101.1850</p>
<p>Full story: <a class="external" href="https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken">https://forum.netgate.com/topic/156890/dpinger-broken-or-dashboard-broken-or-my-brain-is-broken</a></p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=49436
2020-11-12T13:54:00Z
Dee D
<ul></ul><p>I’m not getting desired behavior but I don’t know if it’s caused by changes to this</p>
<p>output of /tmp/rules.debug after reset</p>
<p>GWCABLE_DHCP = " route-to ( hn0 24.52.xxx.97 ) " <br />GWDSL_PPPOE = " route-to ( pppoe0 206.248.xxx.132 ) " <br />GWCable_failover = " route-to { ( hn0 24.52.xxx.97 ) ( hn0 24.52.xxx.97 ) } " <br />GWDSL_Failover = " route-to { ( pppoe0 206.248.xxx.132 ) ( pppoe0 206.248.xxx.132 ) } " <br />GWLoad_Balance = " "</p>
<p>Cable_Failover is CABLE_DHCP tier 1, DSL_PPOE tier 2<br />DSL_Failover is DSL_PPPOE tier 1, CABLE_DHCP tier 2<br />Load_balance is both tier 1</p>
<p>Is the output above correct? It didn’t used to look like that (but then it didn’t work then either)</p>
<p>DSL gets taken off for packet loss:</p>
<p>8:40:30 MONITOR: DSL_PPPOE has packet loss, omitting from routing group Cable_failover<br />8.8.4.4|206.248.xxx.113|DSL_PPPOE|16.47ms|0.303ms|22%|down|highloss<br />8:48:12 MONITOR: DSL_PPPOE is available now, adding to routing group Cable_failover<br />8.8.4.4|206.248.xxx.113|DSL_PPPOE|16.301ms|0.061ms|0.0%|online|none</p>
<p>output from /tmp/rules/debug following the event:</p>
<p>GWCABLE_DHCP = " route-to ( hn0 24.52.xxx.97) " <br />GWDSL_PPPOE = " " <br />GWCable_failover = " route-to { ( hn0 24.52.xxx.97 ) ( hn0 24.52.xxx.97 ) } " <br />GWDSL_Failover = " route-to { ( hn0 24.52.xxx.97 ) ( hn0 24.52.xxx.97 ) } " <br />GWLoad_Balance = " "</p>
<p>DSL doesn’t work anymore until manual intervention</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=50532
2021-01-25T11:59:43Z
Max Leighton
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>I can not reproduce this in 2.5 under the same conditions that cause it in 2.4.5p1. Dee D's response sounds like the behavior we see in 2.4.5p1 when this is triggered, but it's not happening in the latest build. I will mark this resolved because I can consistently reproduce this in 2.4.5p1 but no longer in 2.5</p>
pfSense - Bug #9450: Multiwan gateway group fail-over not working as expected (possible race condition)
https://redmine.pfsense.org/issues/9450?journal_id=52687
2021-03-26T01:35:07Z
Dee D
<ul></ul><p>Yet every time my dsl goes offline e.g. for packet loss and comes back according to the notification, when I cat /tmp/rules.debug the wrong (secondary) gateway is still in the failover group. There were a lot of potential explanations for what was happening during 2.5 dev but none of the resolved bugs fixed it?</p>