pfSense

After searching through logs and pfctl / netstat output I have found that after pppoe address renewal static route to monitor ip is no more in the routing table. In system log I can find this error:

Sep 3 11:20:47 php: /system_gateways.php: The command '/sbin/route delete -host '209.85.135.99'' returned exit code '1', the output was 'route: writing to routing socket: No such process delete host 209.85.135.99: not in table'

this is routing table after pppoe address renewal:

89.201.224.1 link#12 UH 0 0 pppoe0
89.201.228.101 link#12 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 35 lo0
127.0.0.2 127.0.0.1 UHS 0 0 lo0
192.168.167.8/29 link#8 U 0 0 re0_vl
192.168.167.9 link#8 UHS 0 0 lo0
192.168.168.0/30 link#7 U 0 26093432 re0_vl
192.168.168.1 link#7 UHS 0 0 lo0
192.168.168.16/29 link#2 U 0 0 rl0
192.168.168.17 link#2 UHS 0 0 lo0
213.147.96.3 192.168.168.18 UGHS 4 128796 rl0
213.147.96.4 192.168.167.10 UGHS 4 128796 re0_vl

This is routing table after manualy editing and saving any of the gateways:

89.201.224.1 link#12 UH 0 0 pppoe0
89.201.228.101 link#12 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 35 lo0
127.0.0.2 127.0.0.1 UHS 0 0 lo0
192.168.167.8/29 link#8 U 0 0 re0_vl
192.168.167.9 link#8 UHS 0 0 lo0
192.168.168.0/30 link#7 U 0 26124466 re0_vl
192.168.168.1 link#7 UHS 0 0 lo0
192.168.168.16/29 link#2 U 0 0 rl0
192.168.168.17 link#2 UHS 0 0 lo0
209.85.135.99 89.201.224.1 UGHS 0 55 pppoe0
213.147.96.3 192.168.168.18 UGHS 4 55 rl0
213.147.96.4 192.168.167.10 UGHS 4 55 re0_vl

I think I need to refine this issue, because problems after upgrade and pppoe address renewal are simular but not the same. After upgrade, pppoe gateway are not shown in the gateway list at all, while in pppoe address renewal, it is shown, but marked as offline. Workaround from first post, is the same for both cases.

Can you give more details about your configuration.
As verbose as possible will be better.
For example.
All configuration, ifconfig/netstat -rn/ps -ax output, /tmp/apinger* /var/etc/apinger*, /tmp/rules.debug, screenshots.

This is system output while it is not routinh trough pppoe

1. uname -an
   FreeBSD 8.1-RELEASE FreeBSD 8.1-RELEASE #1: Thu Sep 2 00:14:57 EDT 2010 i386

########################################

1. ifconfig
   re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0 prefixlen 64 scopeid 0x1
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 00:0e:2e:8e:4c:d3
   inet 192.168.168.17 netmask 0xfffffff8 broadcast 192.168.168.23
   inet6 fe80::20e:2eff:fe8e:4cd3%rl0 prefixlen 64 scopeid 0x2
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (100baseTX <full-duplex>)
   status: active
   pfsync0: flags=0<> metric 0 mtu 1460
   syncpeer: 224.0.0.240 maxupd: 128
   enc0: flags=0<> metric 0 mtu 1536
   lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=3<RXCSUM,TXCSUM>
   inet 127.0.0.1 netmask 0xff000000
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   pflog0: flags=100<PROMISC> metric 0 mtu 33200
   re0_vlan50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0_vlan50 prefixlen 64 scopeid 0x7
   inet 192.168.168.1 netmask 0xfffffffc broadcast 192.168.168.3
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 50 parent interface: re0
   re0_vlan56: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0_vlan56 prefixlen 64 scopeid 0x8
   inet 192.168.167.9 netmask 0xfffffff8 broadcast 192.168.167.15
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 56 parent interface: re0
   re0_vlan57: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0_vlan57 prefixlen 64 scopeid 0x9
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 57 parent interface: re0
   re0_vlan58: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0_vlan58 prefixlen 64 scopeid 0xa
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 58 parent interface: re0
   re0_vlan40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether 00:1c:c0:d8:dd:45
   inet6 fe80::21c:c0ff:fed8:dd45%re0_vlan40 prefixlen 64 scopeid 0xb
   inet 10.30.0.18 netmask 0xfffffff8 broadcast 10.30.0.23
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 40 parent interface: re0
   pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1462
   inet6 fe80::21c:c0ff:fed8:dd45%pppoe0 prefixlen 64 scopeid 0xc
   inet 89.201.232.62 --> 89.201.224.1 netmask 0xffffffff
   nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

################################################

1. netstat -rn
   Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.167.10 UGS 5 309231 re0_vl
10.26.0.0/24 10.30.0.17 UGS 1 4138447 re0_vl
10.26.4.0/22 10.30.0.17 UGS 0 7149494 re0_vl
10.26.8.0/22 10.30.0.17 UGS 0 6864077 re0_vl
10.26.12.0/22 10.30.0.17 UGS 0 4867166 re0_vl
10.26.16.0/22 10.30.0.17 UGS 0 3803211 re0_vl
10.26.20.0/22 10.30.0.17 UGS 0 22155731 re0_vl
10.26.24.0/22 10.30.0.17 UGS 0 10511034 re0_vl
10.26.28.0/22 10.30.0.17 UGS 0 1269479 re0_vl
10.26.32.0/22 10.30.0.17 UGS 0 0 re0_vl
10.30.0.16/29 link#11 U 2 302234 re0_vl
10.30.0.18 link#11 UHS 0 0 lo0
89.201.224.1 link#12 UH 0 0 pppoe0
89.201.232.62 link#12 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 35 lo0
127.0.0.2 127.0.0.1 UHS 0 0 lo0
192.168.167.8/29 link#8 U 0 0 re0_vl
192.168.167.9 link#8 UHS 0 0 lo0
192.168.168.0/30 link#7 U 0 43691931 re0_vl
192.168.168.1 link#7 UHS 0 0 lo0
192.168.168.16/29 link#2 U 0 0 rl0
192.168.168.17 link#2 UHS 0 0 lo0
213.147.96.3 192.168.168.18 UGHS 4 56516 rl0
213.147.96.4 192.168.167.10 UGHS 4 56516 re0_vl

###############################################

1. ps aux
   USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
   root 11 195.4 0.0 0 16 ?? RL Thu10PM 9798:36.07 [idle]
   root 12 4.8 0.0 0 128 ?? WL Thu10PM 195:02.28 [intr]
   root 52784 0.9 4.1 44404 20376 ?? S Fri12PM 0:08.58 /usr/local/bin/php
   root 26421 0.1 3.5 43380 17156 ?? S Thu10PM 0:05.87 /usr/local/bin/php
   root 0 0.0 0.0 0 48 ?? DLs Thu10PM 0:00.75 [kernel]
   root 1 0.0 0.1 1888 432 ?? ILs Thu10PM 0:00.03 /sbin/init --
   root 2 0.0 0.0 0 8 ?? DL Thu10PM 0:16.90 [g_event]
   root 3 0.0 0.0 0 8 ?? DL Thu10PM 0:11.16 [g_up]
   root 4 0.0 0.0 0 8 ?? DL Thu10PM 0:06.86 [g_down]
   root 5 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [crypto]
   root 6 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [crypto returns]
   root 7 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [sctp_iterator]
   root 8 0.0 0.0 0 8 ?? DL Thu10PM 0:28.09 [pfpurge]
   root 9 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [xpt_thrd]
   root 10 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [audit]
   root 13 0.0 0.0 0 16 ?? DL Thu10PM 1:27.18 [ng_queue]
   root 14 0.0 0.0 0 8 ?? DL Thu10PM 14:58.77 [yarrow]
   root 15 0.0 0.0 0 160 ?? DL Thu10PM 0:07.30 [usb]
   root 16 0.0 0.0 0 8 ?? DL Thu10PM 0:00.35 [pagedaemon]
   root 17 0.0 0.0 0 8 ?? DL Thu10PM 0:00.00 [vmdaemon]
   root 18 0.0 0.0 0 8 ?? DL Thu10PM 0:00.51 [idlepoll]
   root 19 0.0 0.0 0 8 ?? DL Thu10PM 0:00.01 [pagezero]
   root 20 0.0 0.0 0 8 ?? DL Thu10PM 0:01.59 [bufdaemon]
   root 21 0.0 0.0 0 8 ?? DL Thu10PM 0:15.22 [syncer]
   root 22 0.0 0.0 0 8 ?? DL Thu10PM 0:01.77 [vnlru]
   root 23 0.0 0.0 0 8 ?? DL Thu10PM 0:02.10 [softdepflush]
   root 24 0.0 0.0 0 8 ?? DL Thu10PM 0:02.36 [flowcleaner]
   root 35 0.0 0.0 0 8 ?? DL Thu10PM 0:00.36 [md0]
   root 363 0.0 0.1 1888 532 ?? Is Thu10PM 0:00.03 /sbin/devd
   root 729 0.0 0.2 3316 1036 ?? Is Thu10PM 0:00.33 minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
   root 2298 0.0 0.2 3316 1036 ?? Is Thu10PM 0:00.02 minicron 3600 /var/run/expire_accounts.pid /etc/rc.exipireaccounts
   root 2893 0.0 0.2 3316 1036 ?? Is Thu10PM 0:00.00 minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
   root 3166 0.0 0.3 6092 1460 ?? SNs Thu10PM 0:08.47 /usr/local/sbin/check_reload_status
   root 7924 0.0 0.7 7992 3516 ?? Ss 10:47AM 0:00.11 sshd: root@pts/0 (sshd)
   root 8415 0.0 0.2 3316 976 ?? Is 10:47AM 0:00.01 /usr/local/sbin/sshlockout_pf
   root 9561 0.0 0.6 5272 3100 ?? Is Thu10PM 0:00.00 /usr/sbin/sshd
   root 10808 0.0 0.3 3448 1464 ?? Ss Thu10PM 4:35.76 /usr/sbin/syslogd c -f /var/etc/syslog.conf
   _ntp 11743 0.0 0.3 3316 1344 ?? I Thu10PM 0:00.71 ntpd: ntp engine (ntpd)
   root 11804 0.0 0.3 3316 1340 ?? Ss Thu10PM 0:00.40 ntpd: [priv] (ntpd)
   root 12670 0.0 0.3 3436 1432 ?? Is Thu10PM 0:00.03 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
   root 16775 0.0 0.9 9488 4556 ?? Ss 12:00AM 0:01.24 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt4.conf -p /var/run/pppoe_opt4.pid -s ppp pppoeclie
   root 25308 0.0 1.2 8604 5800 ?? S Thu10PM 0:14.34 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
   root 25585 0.0 2.1 42356 10228 ?? Is Thu10PM 0:00.13 /usr/local/bin/php
   root 26110 0.0 2.1 42356 10264 ?? Is Thu10PM 0:00.14 /usr/local/bin/php
   root 26422 0.0 3.7 44404 18344 ?? I Thu10PM 0:04.65 /usr/local/bin/php
   root 28661 0.0 4.1 44404 20276 ?? S Fri12PM 0:18.21 /usr/local/bin/php
   nobody 31046 0.0 0.5 4528 2420 ?? I Thu10PM 0:00.00 /usr/local/sbin/dnsmasq --local-ttl 1 --all-servers --dns-forward-max=5000 --cache-size=10000
   root 37107 0.0 0.3 3316 1344 ?? Ss 7:07PM 0:28.09 /usr/local/sbin/apinger -c /var/etc/apinger.conf
   root 37356 0.0 0.4 4480 1804 ?? I 7:07PM 0:00.92 /usr/local/bin/rrdtool -
   root 45311 0.0 0.3 3404 1380 ?? Is Thu10PM 0:00.88 /usr/sbin/cron -s
   root 50600 0.0 0.1 1564 592 ?? SN 10:50AM 0:00.00 sleep 60
   root 11009 0.0 0.2 3376 1180 v0 Is+ Thu10PM 0:00.00 /usr/libexec/getty Pc ttyv0
   root 11581 0.0 1.4 10008 7124 v0 S Thu10PM 1:38.76 /usr/sbin/tcpdump s 256 -v -l -n -e -ttt -i pflog0
   root 11586 0.0 0.2 3316 924 v0 S Thu10PM 2:07.18 logger t pf -p local0.info
   root 39962 0.0 0.3 3656 1500 v0 SN Thu10PM 1:38.10 /bin/sh /var/db/rrd/updaterrd.sh
   root 8711 0.0 0.3 3656 1536 0 Is 10:47AM 0:00.01 -sh (sh)
   root 9447 0.0 0.3 3656 1540 0 I 10:47AM 0:00.01 /bin/sh /etc/rc.initial
   root 11980 0.0 0.5 4696 2384 0 S 10:47AM 0:00.03 /bin/tcsh

#############################################

1. cat /tmp/apinger.status
   213.147.96.3|192.168.168.17|gw_met2|56610|56602|1283770306|14.477ms|0.0%|none
   10.30.0.17|10.30.0.18|gw_lan|56610|56382|1283770306|28.486ms|2.0%|none
   213.147.96.4|192.168.167.9|gw_wan|56610|56253|1283770306|14.550ms|6.0%|none
   209.85.135.99|89.201.226.164|GW_OPT4|56610|17484|1283731200|55.276ms|100.0%|down

############################################

1. cat /var/log/apinger.log
   Sep 2 22:48:07 guard apinger: Starting Alarm Pinger, apinger(16429)
   Sep 2 22:48:17 guard apinger: ALARM: gw_met2(213.147.96.3) * down
   Sep 2 22:48:40 guard apinger: alarm canceled: gw_met2(213.147.96.3) down
   Sep 2 22:52:40 guard apinger: Exiting on signal 15.
   Sep 2 22:52:40 guard apinger: Starting Alarm Pinger, apinger(11020)
   Sep 2 22:52:56 guard apinger: Exiting on signal 15.
   Sep 2 22:52:56 guard apinger: Starting Alarm Pinger, apinger(20471)
   Sep 3 00:00:10 guard apinger: ALARM: GW_OPT4(209.85.135.99) down
   Sep 3 07:59:02 guard apinger: ALARM: gw_wan(213.147.96.4) loss
   Sep 3 07:59:53 guard apinger: alarm canceled: gw_wan(213.147.96.4) loss
   Sep 3 08:01:46 guard apinger: ALARM: gw_wan(213.147.96.4) loss
   Sep 3 08:02:41 guard apinger: alarm canceled: gw_wan(213.147.96.4) loss
   Sep 3 08:06:52 guard apinger: ALARM: gw_wan(213.147.96.4) loss
   Sep 3 08:07:46 guard apinger: alarm canceled: gw_wan(213.147.96.4) loss
   Sep 3 08:15:30 guard apinger: ALARM: gw_wan(213.147.96.4) loss
   Sep 3 08:17:26 guard apinger: alarm canceled: gw_wan(213.147.96.4) loss
   Sep 3 11:20:47 guard apinger: Exiting on signal 15.
   Sep 3 11:20:47 guard apinger: Starting Alarm Pinger, apinger(41621)
   Sep 3 11:20:53 guard apinger: Exiting on signal 15.
   Sep 3 11:20:53 guard apinger: Starting Alarm Pinger, apinger(50764)
   Sep 3 23:58:35 guard apinger: ALARM: GW_OPT4(209.85.135.99) down
   Sep 4 18:05:31 guard apinger: ALARM: gw_met2(213.147.96.3) down
   Sep 4 18:05:31 guard apinger: ALARM: gw_wan(213.147.96.4) down
   Sep 4 18:06:01 guard apinger: alarm canceled: gw_met2(213.147.96.3) down
   Sep 4 18:06:01 guard apinger: alarm canceled: gw_wan(213.147.96.4) down
   Sep 4 23:24:03 guard apinger: Exiting on signal 15.
   Sep 4 23:24:03 guard apinger: Starting Alarm Pinger, apinger(58033)
   Sep 4 23:58:38 guard apinger: ALARM: GW_OPT4(209.85.135.99) down
   Sep 5 19:04:48 guard apinger: Exiting on signal 15.
   Sep 5 19:04:48 guard apinger: Starting Alarm Pinger, apinger(42495)
   Sep 5 19:07:24 guard apinger: Exiting on signal 15.
   Sep 5 19:07:24 guard apinger: Starting Alarm Pinger, apinger(37107)
   Sep 6 00:00:11 guard apinger: ALARM: GW_OPT4(209.85.135.99) down *

############################################

cat /var/log/system.log

Sep 6 00:00:05 guard ppp: [opt4] Bundle: Status update: up 1 link, total bandwidth 64000 bps
Sep 6 00:00:05 guard ppp: [opt4] IPCP: Open event
Sep 6 00:00:05 guard ppp: [opt4] IPCP: state change Initial --> Starting
Sep 6 00:00:05 guard ppp: [opt4] IPCP: LayerStart
Sep 6 00:00:05 guard ppp: [opt4] IPCP: Up event
Sep 6 00:00:05 guard ppp: [opt4] IPCP: state change Starting --> Req-Sent
Sep 6 00:00:05 guard ppp: [opt4] IPCP: SendConfigReq #1
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 0.0.0.0
Sep 6 00:00:05 guard ppp: [opt4] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Sep 6 00:00:05 guard ppp: [opt4] IPCP: rec'd Configure Request #0 (Req-Sent)
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 89.201.224.1
Sep 6 00:00:05 guard ppp: [opt4] 89.201.224.1 is OK
Sep 6 00:00:05 guard ppp: [opt4] IPCP: SendConfigAck #0
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 89.201.224.1
Sep 6 00:00:05 guard ppp: [opt4] IPCP: state change Req-Sent --> Ack-Sent
Sep 6 00:00:05 guard ppp: [opt4] IPCP: rec'd Configure Reject #1 (Ack-Sent)
Sep 6 00:00:05 guard ppp: [opt4] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Sep 6 00:00:05 guard ppp: [opt4] IPCP: SendConfigReq #2
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 0.0.0.0
Sep 6 00:00:05 guard ppp: [opt4] IPCP: rec'd Configure Nak #2 (Ack-Sent)
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 89.201.232.62
Sep 6 00:00:05 guard ppp: [opt4] 89.201.232.62 is OK
Sep 6 00:00:05 guard ppp: [opt4] IPCP: SendConfigReq #3
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 89.201.232.62
Sep 6 00:00:05 guard ppp: [opt4] IPCP: rec'd Configure Ack #3 (Ack-Sent)
Sep 6 00:00:05 guard ppp: [opt4] IPADDR 89.201.232.62
Sep 6 00:00:05 guard ppp: [opt4] IPCP: state change Ack-Sent --> Opened
Sep 6 00:00:05 guard ppp: [opt4] IPCP: LayerUp
Sep 6 00:00:05 guard ppp: [opt4] 89.201.232.62 -> 89.201.224.1
Sep 6 00:00:05 guard ppp: [opt4] IFACE: Up event
Sep 6 00:00:07 guard check_reload_status: rc.newwanip starting
Sep 6 00:00:08 guard php: : rc.newwanip: Informational is starting pppoe0.
Sep 6 00:00:08 guard php: : rc.newwanip: on (IP address: 89.201.232.62) (interface: opt4) (real interface: pppoe0).
Sep 6 00:00:11 guard apinger: ALARM: GW_OPT4(209.85.135.99) * down *
Sep 6 00:00:14 guard check_reload_status: updating dyndns
Sep 6 00:00:16 guard check_reload_status: reloading filter
Sep 6 00:00:17 guard php: : MONITOR: GW_OPT4 has high latency, removing from routing group
Sep 6 00:00:17 guard last message repeated 3 times
Sep 6 00:00:19 guard check_reload_status: Rewriting resolv.conf
Sep 6 00:00:27 guard check_reload_status: reloading filter
Sep 6 00:00:28 guard php: : MONITOR: GW_OPT4 has high latency, removing from routing group
Sep 6 00:00:28 guard last message repeated 3 times

########################################

1. cat /tmp/rules.debug
   #System aliases

loopback = "{ lo0 }"
WAN = "{ re0_vlan56 }"
LAN = "{ re0_vlan40 }"
GLAN = "{ re0_vlan50 }"
MET2 = "{ rl0 }"
OPTIM1 = "{ pppoe0 }"

#SSH Lockout Table
table <sshlockout> persist
#Snort2C table
table <snort2c>

table <virusprot>

1. User Aliases
   table <it_pcs> { 10.26.4.129 10.26.4.132 10.26.4.133 }
   it_pcs = "<it_pcs>"
   std_ports_in = "{ 21 80 110 143 443 993 995 7443 1935 8080 8888 }"
   std_ports_out = "{ 22 23 25 53 123 500 1000 1194 1723 1972 3389 4500 5000 5900 10000 10010 11160 3322 20400 2082 11496 1352 3900 65505 65510 }"

1. Gateways
   GWgw_met2 = " route-to ( rl0 192.168.168.18 ) "
   GWgw_lan = " route-to ( re0_vlan40 10.30.0.17 ) "
   GWgw_wan = " route-to ( re0_vlan56 192.168.167.10 ) "
   GWGW_OPT4 = " route-to ( pppoe0 89.201.224.1 ) "
   GWggw_lan_in = " route-to { ( re0_vlan56 192.168.167.10 ) } "
   GWggw_lan_out = " route-to { ( re0_vlan56 192.168.167.10 ) } "
   GWggw_glan_in = " route-to { ( re0_vlan56 192.168.167.10 ) } "
   GWggw_glan_out = " route-to { ( re0_vlan56 192.168.167.10 ) } "

set loginterface re0_vlan56
set loginterface re0_vlan40
set loginterface re0_vlan50
set loginterface rl0
set loginterface pppoe0
set optimization aggressive
set limit states 47000

set skip on pfsync0

nat-anchor "natearly/*"
nat-anchor "natrules/*"

1. Outbound NAT rules

1. Subnets to NAT
   table <tonatsubnets> { 10.26.0.0/24 10.26.12.0/22 10.26.16.0/22 10.26.20.0/22 10.26.24.0/22 10.26.28.0/22 10.26.32.0/22 10.26.4.0/22 10.26.8.0/22 10.30.0.16/29 192.168.168.0/30 }
   nat on $WAN from <tonatsubnets> port 500 to any port 500 -> 192.168.167.9/32 port 500
   nat on $WAN from <tonatsubnets> to any -> 192.168.167.9/32 port 1024:65535

nat on $MET2 from <tonatsubnets> port 500 to any port 500 -> 192.168.168.17/32 port 500
nat on $MET2 from <tonatsubnets> to any -> 192.168.168.17/32 port 1024:65535

nat on $OPTIM1 from <tonatsubnets> port 500 to any port 500 -> 89.201.232.62/32 port 500
nat on $OPTIM1 from <tonatsubnets> to any -> 89.201.232.62/32 port 1024:65535

1. Load balancing anchor
   rdr-anchor "relayd/*"
2. TFTP proxy
   rdr-anchor "tftp-proxy/*"
   table <direct_networks> { 192.168.167.8/29 10.30.0.16/29 192.168.168.0/30 192.168.168.16/29 89.201.232.62/32 }
3. NAT Inbound Redirects
   rdr on re0_vlan56 proto tcp from any to 192.168.167.9 port 3323 -> 192.168.168.2 port 3322
   rdr on re0_vlan56 proto tcp from any to 192.168.167.9 port 8888 -> 192.168.168.2 port 80
   rdr on re0_vlan56 proto tcp from any to 192.168.167.9 port 3389 -> 10.26.0.8
4. UPnPd rdr anchor
   rdr-anchor "miniupnpd"

1. default deny rules
   #---------------------------------------------------------------------------
   block in log all label "Default deny rule"
   block out log all label "Default deny rule"

1. We use the mighty pf, we cannot be fooled.
   block quick proto { tcp, udp } from any port = 0 to any
   block quick proto { tcp, udp } from any to any port = 0

1. Block all IPv6
   block in quick inet6 all
   block out quick inet6 all

1. snort2c
   block quick from <snort2c> to any label "Block snort2c hosts"
   block quick from any to <snort2c> label "Block snort2c hosts"

1. package manager early specific hook
   anchor "packageearly"

1. carp
   anchor "carp"

1. SSH lockout
   block in log quick proto tcp from <sshlockout> to any port 3322 label "sshlockout"
   block in quick from <virusprot> to any label "virusprot overload table"
   antispoof for re0_vlan56
   antispoof for re0_vlan40
   antispoof for re0_vlan50
   antispoof for rl0
   table <bogons> persist file "/etc/bogons"
2. block bogon networks
3. http://www.cymru.com/Documents/bogon-bn-nonagg.txt
   anchor "opt4bogons"
   block in log quick on $OPTIM1 from <bogons> to any label "block bogon networks from OPTIM1"
   antispoof for pppoe0
4. block anything from private networks on interfaces with the option set
   antispoof for $OPTIM1
   block in log quick on $OPTIM1 from 10.0.0.0/8 to any label "block private networks from wan block 10/8"
   block in log quick on $OPTIM1 from 127.0.0.0/8 to any label "block private networks from wan block 127/8"
   block in log quick on $OPTIM1 from 172.16.0.0/12 to any label "block private networks from wan block 172.16/12"
   block in log quick on $OPTIM1 from 192.168.0.0/16 to any label "block private networks from wan block 192.168/16"
   anchor "spoofing"

1. loopback
   anchor "loopback"
   pass in on $loopback all label "pass loopback"
   pass out on $loopback all label "pass loopback"

1. let out anything from the firewall host itself and decrypted IPsec traffic
   pass out all keep state allow-opts label "let out anything from firewall host itself"
   pass out route-to ( re0_vlan56 192.168.167.10 ) from 192.168.167.9 to !192.168.167.8/29 keep state allow-opts label "let out anything from firewall host itself"
   pass out route-to ( rl0 192.168.168.18 ) from 192.168.168.17 to !192.168.168.16/29 keep state allow-opts label "let out anything from firewall host itself"
   pass out route-to ( pppoe0 89.201.224.1 ) from 89.201.232.62 to !89.201.232.62/32 keep state allow-opts label "let out anything from firewall host itself"
2. make sure the user cannot lock himself out of the webConfigurator or SSH
   anchor "anti-lockout"
   pass in quick on re0_vlan40 from any to (re0_vlan40) keep state label "anti-lockout rule"

1. User-defined rules follow
   pass on { re0_vlan56 re0_vlan40 rl0 pppoe0 } proto tcp from any to any port 10010 flags S/SA keep state label "USER_RULE"
   pass on { re0_vlan56 re0_vlan40 rl0 pppoe0 } proto tcp from any to any port 3322 flags S/SA keep state label "USER_RULE"
   pass in quick on $WAN reply-to ( re0_vlan56 192.168.167.10 ) proto tcp from any to 192.168.168.2 port 3322 label "USER_RULE: NAT "
   pass in quick on $WAN reply-to ( re0_vlan56 192.168.167.10 ) proto tcp from any to 192.168.168.2 port 80 label "USER_RULE: NAT "
   pass in quick on $WAN reply-to ( re0_vlan56 192.168.167.10 ) proto tcp from any to 10.26.0.8 port 3389 label "USER_RULE: NAT "
   pass in quick on $MET2 reply-to ( rl0 192.168.168.18 ) proto { tcp udp } from 88.198.227.140/24 to 10.26.0.0/16 keep state label "USER_RULE"
   pass in quick on $GLAN proto { tcp udp } from 192.168.168.1/30 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $GLAN $GWggw_glan_in proto { tcp udp } from 192.168.168.1/30 to any port $std_ports_in keep state label "USER_RULE"
   pass in quick on $GLAN proto { tcp udp } from 192.168.168.1/30 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $GLAN $GWggw_glan_out proto { tcp udp } from 192.168.168.1/30 to any port $std_ports_out keep state label "USER_RULE"
   pass in quick on $GLAN proto esp from 192.168.168.1/30 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $GLAN $GWggw_glan_out proto esp from 192.168.168.1/30 to any keep state label "USER_RULE"
   pass in quick on $GLAN proto gre from 192.168.168.1/30 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $GLAN $GWggw_glan_out proto gre from 192.168.168.1/30 to any keep state label "USER_RULE"
   pass in quick on $LAN inet proto icmp from 10.26.0.0/16 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $LAN $GWggw_lan_out inet proto icmp from 10.26.0.0/16 to any keep state label "USER_RULE"
   pass in quick on $LAN $GWgw_met2 proto { tcp udp } from 10.26.0.0/16 to 88.198.227.0/24 keep state label "USER_RULE: lacroma internet radio"
   pass in quick on $LAN proto { tcp udp } from 10.26.0.0/16 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $LAN $GWggw_lan_in proto { tcp udp } from 10.26.0.0/16 to any port $std_ports_in keep state label "USER_RULE"
   pass in quick on $LAN proto { tcp udp } from 10.26.0.0/16 to <vpns> keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
   pass in quick on $LAN $GWggw_lan_out proto { tcp udp } from 10.26.0.0/16 to any port $std_ports_out keep state label "USER_RULE"

###############################################

1. cat /var/etc/apinger*

1. pfSense apinger configuration file. Automatically Generated!

1. User and group the pinger should run as
   user "root"
   group "wheel"

1. Mailer to use (default: "/usr/lib/sendmail -t")
   #mailer "/var/qmail/bin/qmail-inject"

1. Location of the pid-file (default: "/var/run/apinger.pid")
   pid_file "/var/run/apinger.pid"

1. Format of timestamp (%s macro) (default: "%b %d %H:%M:%S")
   #timestamp_format "%Y%m%d%H%M%S"

status { ## File where the status information whould be written to
file "/tmp/apinger.status" ## Interval between file updates ## when 0 or not set, file is written only when SIGUSR1 is received
interval 10s
}

1. RRDTool status gathering configuration
2. Interval between RRD updates
   rrd interval 60s;

1. These parameters can be overriden in a specific alarm configuration
   alarm default {
   command on "/usr/bin/touch /tmp/filter_dirty"
   command off "/usr/bin/touch /tmp/filter_dirty"
   combine 10s
   }

1. "Down" alarm definition.
2. This alarm will be fired when target doesn't respond for 30 seconds.
   alarm down "down" {
   time 10s
   }

1. "Delay" alarm definition.
2. This alarm will be fired when responses are delayed more than 200ms
3. it will be canceled, when the delay drops below 100ms
   alarm delay "delay" {
   delay_low 200ms
   delay_high 500ms
   }

1. "Loss" alarm definition.
2. This alarm will be fired when packet loss goes over 20%
3. it will be canceled, when the loss drops below 10%
   alarm loss "loss" {
   percent_low 10
   percent_high 20
   }

target default { ## How often the probe should be sent
interval 1s

1. How many replies should be used to compute average delay
2. for controlling "delay" alarms
   avg_delay_samples 10

1. How many probes should be used to compute average loss
   avg_loss_samples 50

1. The delay (in samples) after which loss is computed
2. without this delays larger than interval would be treated as loss
   avg_loss_delay_samples 20

1. Names of the alarms that may be generated for the target
   alarms "down","delay","loss"

1. Location of the RRD
   #rrd file "/var/db/rrd/apinger-%t.rrd"
   }

1. Targets to probe
2. Each one defined with:
3. target <address> { <parameter>... }
4. The parameters are those described above in the "target default" section
5. plus the "description" parameter.
6. the <address> should be IPv4 or IPv6 address (not hostname!)
   target "213.147.96.3" {
   description "gw_met2"
   srcip "192.168.168.17"
   alarms override "loss","delay","down";
   rrd file "/var/db/rrd/gw_met2-quality.rrd"
   }

target "10.30.0.17" {
description "gw_lan"
srcip "10.30.0.18"
alarms override "loss","delay","down";
rrd file "/var/db/rrd/gw_lan-quality.rrd"
}

target "213.147.96.4" {
description "gw_wan"
srcip "192.168.167.9"
alarms override "loss","delay","down";
rrd file "/var/db/rrd/gw_wan-quality.rrd"
}

target "209.85.135.99" {
description "GW_OPT4"
srcip "89.201.226.164"
alarms override "loss","delay","down";
rrd file "/var/db/rrd/GW_OPT4-quality.rrd"
}

Just to mention again: this is not hardware problem or link problem. I have 3 firewalls with different hardware, different Internet providers and same situation.
I hope we can catch this bug because it is in core functionality. There are 2 or 3 references on PF forum of this bug.

Does your pppoe address change during renewal or it stays the same?

yes...it changes

I will check this new change on /etc/inc/gwlb.inc and report back

Ermal...nothing have changed.
After restart there is no reference of pppoe gateway in system logs, like it does not exists.
On status_gateways.php GW_OPT4 is not on the list.
But in system_gateways.php pppoe gateway GW_OPT4 is on the list.
Everything will be ok if I edit any of the gateways and reload filters.

After testing I have find out that:
- after disconnecting pppoe interface, static route to monitor is is correctly deleted
- after connecting again, static route to monitor is is not added
I think something is wrong in /usr/local/sbin/ppp-linkup ?

1. cat /usr/local/sbin/ppp-linkup
   #!/bin/sh

1. unset CGI environment variables so as not to confuse PHP
   unset CONTENT_TYPE GATEWAY_INTERFACE REMOTE_USER REMOTE_ADDR AUTH_TYPE
   unset HTTP_USER_AGENT CONTENT_LENGTH SCRIPT_FILENAME HTTP_HOST
   unset SERVER_SOFTWARE HTTP_REFERER SERVER_PROTOCOL REQUEST_METHOD
   unset SERVER_PORT SCRIPT_NAME SERVER_NAME

1. write nameservers to file
   if [ $6 = "dns1" ]; then
   echo $7 > /var/etc/nameserver_$1
   /sbin/route add $7 $4
   fi

if [ $8 = "dns2" ]; then
echo $9 >> /var/etc/nameserver_$1
/sbin/route add $9 $4
fi

1. let the configuration system know that the ip has changed.
   /bin/echo $1 > /tmp/rc.newwanip # this file is newer written and this is opt if ... not wan
   /bin/echo $4 > /tmp/$1_router
   /usr/bin/touch /tmp/$1up
   exit 0
   ´´´
   In this file there is no code to add route to monitor ip

This last commit didn't help also c3b1ba3fcd6284d2a36b23c5938a3fee4f520cdf
This pppoe interface is on vlan OPT4 interface.
When I do manual disconnect than connect, static route to monitor ip is gone. Is it because of delay while pppoe link is established?
But when I edit any of the gateways, static route to monitor ip is there again.
What is happening in "edit gw" procedure and not happening when pppoe interface automaticly renews ip?
I will try to find traces, but still learning pfsense code structure.
Thanks for your help.

Its the same thing.
setup_gateway_monitoring() is getting called.

Probably you need to test with a newer snapshots since now the reload will be quite fast as soon as the links comes up.
Not delayed some seconds as before.

I have removed all routing groups to make it simple.
Than I edit and saved settings of pppoe interface and route to monitor ip is back.
Than I disconnected and connected pppoe interface, and static route is gone.

using latest snapshot 06.09

I can't see setup_gateways_monitor() function being called from system_gateways_edit.php, page where static routes are added correctly.
If it is not included there from gwlb.inc, than it probably should be common function system wide.
Still digging and hoping for some help

setup_gateways_monitor is called from /etc/rc.bootup /etc/newwanip /etc/inc/upgrade_config.inc
All 3 prrocedures does not add static route to monitor ip of dynamic gateway.
Only thing that works for me is system_gateways_edit.php, page that don't seems to call that function.

Just another question.
The static route you speak about is your static route, created through the gui or the one created by pfSense itself for its monitoring ips?

static route created by pfsense to monitor ip of pppoe interface

NOTE: function setup_gateways_monitor in gwdb.inc - it seems it is not configured to add route to monitor IP of dynamic gw:

if ($gateway['gateway'] == "dynamic") {
                    $gateway['monitor'] = "127.0.0.{$i}";
                    $i++;
                }

.......

if($gateway['monitor'] == $gateway['gateway']) {
                /* if the gateway is the same as the monitor we do not add a
    * route as this will break the routing table */
                continue;
            } else {
                if ($gateway['gateway'] != "dynamic" && is_ipaddr($gateway['gateway'])) {
                    mwexec("/sbin/route delete -host " . escapeshellarg($gateway['monitor']));
                    mwexec("/sbin/route add -host " . escapeshellarg($gateway['monitor']) .
                        " " . escapeshellarg($gateway['gateway']));
                    log_error("Removing static route for monitor {$gateway['monitor']} and adding a new route through {$gateway['gateway']}");
                }
            }

This should be ok now.
Since i committed another fix.

NOTE: I am testing it on different firewall with almost identical configuration. I can not crash first one.

Using 13.09 snapshot without latest Ermal's commit (not in repositry, but i will apply it later)
Upgrade started with folowing in status_gateways.php:

GW_WAN 85.114.48.109 85.114.48.109 Online
GW_WTMP 85.114.55.133 85.114.55.133 Online
DSL 95.178.192.1 Unknown Interface opt3 Dynamic Gateway

No static route to 209.85.135.99 which is monitor ip for DSL OPT3
After I reapply configuration of DSL interface, pfsense adds automaticly gateway named GW_OPT3 with its first hop as monitor ip. This seems to be naming inconsistemcy.

After I edit any of the gateways from system_gateways_edit.php, everything is ok. I have status:
GW_WAN 85.114.48.109 85.114.48.109 Online
GW_WTMP 85.114.55.133 85.114.55.133 Online
GW_OPT3 95.178.192.1 95.178.192.1 Online Interfaceopt3dynamic gateway

After manual pppoe reconnect I have following situation:
GW_WAN 85.114.48.109 85.114.48.109 Online
GW_WTMP 85.114.55.133 85.114.55.133 Online
GW_OPT3 95.178.192.1 95.178.192.1 Offline Interfaceopt3dynamic gateway

APPLYING LATEST PATCH: c65e1e0da7df7b367ff97e89dad16f602571cecb THAN REBOOT
Unfortunalely still no change. It shows status offline and doues not route through OPT3 interface (routes through default gw)
I haven't tried gw groups.

Packet capture on that interface shows:
19:51:16.858392 IP 95.178.201.200 > 95.178.192.1: ICMP echo request, id 51496, seq 26114, length 44
19:51:17.859840 IP 95.178.201.200 > 95.178.192.1: ICMP echo request, id 51496, seq 26370, length 44
19:51:18.861324 IP 95.178.201.200 > 95.178.192.1: ICMP echo request, id 51496, seq 26626, length 44

INTREFACE IP IS: 95.178.207.135

I can see for sure that after manual reconnect of pppoe interface, apinger is using old IP as source for ping packets. Thats why system marks gateway as OFFLINE and not using it.

I wrote a bug #889, but it was closed as a duplicate of this bug.

But this problem is relevant for latest snapshot (Tue Sep 14 20:22:41 EDT 2010)

all gateway groups have ceased to work.
All traffic goes through a router by default.

When booting, the console can see the error message in the file /etc/inc/gwlb.inc in 8th row.

Latest commits should fix issues reported.

testing latest snapshot 15.09 with latest commits by Ermal not included in latest snapshot.
Can't test post upgrade situation, but
- on reboot status_gateways.php shows:
GW_WAN 85.114.48.109 - Unknown
GW_WTMP 85.114.55.133 - Unknown
GW_OPT3 dynamic - Unknown Interfaceopt3dynamic
No monitor IP in the row.

- edit gateways in system_gateways_edit.php
got php warning: fsockopen(): unable to connect to unix:///var/run/check_reload_status:-1 (Connection refused) in /etc/inc/util.inc on line 143
All gateways are online

- manual reconnect of pppoe interface and STATUS OF DYNAMIC GW STAYS UP
This is the most important thing for my fw. Still, initialy on reboot there is some error of filling/reading array of monitor IPs and telling apinger to return correct status of gateways.

Ermal, on reboot, apinger is not started. There is apinger proces - ps auxw.
It apears atfer I edit some gw

root 48126 0.0 0.1 3324 1360 ?? Ss 12:23AM 0:00.01 /usr/local/sbin/apinger -c /var/etc/apinger.conf

It works even with alternate monitor ip on pppoe (static route is correctly added).
Thing that I noticed on manual reconnect: pppoe disconnects ok, but when I press connect button, it hangs (tested several times). When I reload status_interfaces.php it shows status connected.
Php warning is not connected with this issue, because it sometimes appears on other pages.
(fsockopen(): unable to connect to unix:///var/run/check_reload_status:-1 (Connection refused) in /etc/inc/util.inc on line 143) - probably should be some other issue - event error

As of 18.09 snapshot I can confirm that this issue is resolved. Dynamic gateways are ok on upgrade / reboot / reconnect

I recreated gateway groups and can confirm that there is still issue with status of gateways in group. Status of each gateway is "Gathering data". I saw simular reports in forum.
In system logs there is message:
php: : Gateways status could not be determined, considering all as up/active.

Another thing I can notice is that dynamic gateway is automaticly displayed as default. My wan static gateway is default too, so now I have 2 defult gateways displayed in system_gateways.php. Still, routing table shows ony one default route - through wan interface.

Ivan, can you test a current snapshot (without your modified status_gateway_groups.php) for this problem again? There have been a lot of changes recently.

Thanks.

I tested it on 2 firewalls and everything seems to be ok. This issue actualy included few other connected issues. My opinion is that you can close this. Thanks for your help