Bug #2548
closed
[ICMP] Packets of specific protocol cannot traverse firewall, pfsense cannot ping or traceroute
Added by Kaishi Axon over 12 years ago.
Updated over 12 years ago.
Affected Architecture:
amd64
Description
PREFACE: I spent 5 days examining this issue, along with 2 network engineers. The issue described does not appear to be config-related.
VERSION: pfsense 2.1 builds from July 04 - July 08, perhaps newer as well)
SYMPTOMS:
- ICMP packets are completely unable to traverse the firewall. Their origin and destination has no effect: LAN-packets cannot reach WAN-hosts, and WAN-packets cannot reach LAN-hosts. The pfsense system (pfsense for short in the future) cannot ping any host on either the LAN or WAN interfaces.
- Firewall logs do not show ICMP connections being blocked.
- Hosts that attempt to use ICMP receive "ICMP Destination Host Unreachable" as the result.
- "Gateway" status page shows the connection to the first hop disconnected either immediately, or upon examining/changing the WAN-interface properties.
CONFOUNDING VARIABLES:
- All non-ICMP traffic (i.e. HTTP, HTTPS, FTP) appears to pass the firewall as usual, per the firewall rules.
- No rule to block ICMP exists, and creating a ICMP-pass rule has no effect.
TEST:The following features were removed or disabled to guarantee a pure test:
- ipv6 tunnel
- traffic shaper
- all firewall rules (excluding default LAN->Any rules)
- squid package.
The following WAN Hosts (all confirmed pingable) were tested:
- Google.com | 8.8.8.8
- Yahoo.com
- Microsoft.com
Testing procedure used:
- Traceroute attempted on a LAN-side workstation: failure.
- Ping attempted on a LAN-side workstation: failure.
- Traceroute attempted from pfsense via WebUI, ICMP checkbox disabled: success.
- Traceroute attempted from pfsense via WebUI, ICMP checkbox enabled: failure.
- Ping attempted from pfsense via WebUI, WAN Host (first ISP Hop): failure.
- Ping attempted from pfsense via WebUI, LAN Host: failure.
- Ping attempted from pfsense via SSH, WAN Host (first ISP Hop): failure.
- Ping attempted from pfsense via SSH, LAN Host: failure.
- Retested the above while running TCPDUMP on the LAN interface: packet comes in, and receives "ICMP Destination Host Unreachable"
- Reteated the above while running TCPDUMP on the WAN interface: packets do not leave the WAN interface, no ICMP traffic under any of the above tests.
HARDWARE:
- 2x Intel Intel 82574L Gigabit Ethernet
- WAN-side MTU set to default (1500), which is correct for ISP
Forgot to mention another symptom:
LAN workstations are able to ping or traceroute WAN-Hosts, if and only if the traffic is established within a few minutes (max 15 per empirical tests) of pfsense booting. If a ping loop is left running on a LAN workstation to a WAN-Host, then the pings will continue to come back indefinitely. If the loop is stopped and restarted, it will fail.
Also, LAN workstations are able to ping pfsense LAN-interface, no problems there.
- Status changed from New to Feedback
This does not appear to be a general issue as I have several VMs around on current snapshots and they are pinging through just fine. As such, we'd need a lot more info about your specific setup.
- Output of: ifconfig -a
- List of packages installed
- Firewall ruleset (/tmp/rules.debug)
- Your config.xml would be helpful to see as well
If you do not want to post those publicly, you may e-mail them to me (jimp [a] pfsense [d] org)
It sounds more like firewall rules or something like snort are blocking it, and if you have an existing state before whatever it is that blocks it comes up, it keeps going, otherwise it's being blocked.
Jim P: I'm happy to provide as much information as I can. I only posted this report after I saw another user complain of the same issues in ##pfsense on freenode. I can provide my config.xml, and I have some logs I've captured, but I have already reverted the system in question to 2.0.1 for the sake of functionality.
I'll email you the logged information I can provide. I don't have access to the config.xml at this moment, but I'll email that you to as soon as I do.
I emailed you the config.xml as well as the logs I captured over this weekend. They should contain all the info you need, but I'll add a few more bits of info to reduce the amount of examination needed. Also, I must stress that the same config has been used under 2.0-beta1, through 2.0.1-Release without incident. It was not until 2.1-BETA0 that I encountered this issue. Rolling back to 2.0.1 but using the same config fixes the issue entirely.
Interface Configuration:
LAN IP: Static: 10.0.0.254/24
LAN MTU: 9000
WAN IP: DHCP from ISP
WAN MTU: default (1500)
pfsense sits directly between the ISP and the primary switch for the network. No other gateways, DHCP servers, DNS servers, etc. exist on the same physical network.
Packages Installed:
- squid (removed as a test, but the config was captured while it was still installed)
- widescreen
Roles Enabled:
- DHCP Server
- DNS Forwarder
- IGMP Proxy
- OpenNTPD
- SSH
One other tidbit that might help: when running ping from pfsense via SSH, the failure message was "Operation Not Permitted" even when running as root. This made no sense to me at all.
Jim-P: do you or any other developers need any additional information about this one? I'm really interested in getting it fixed and I'm willing to provide as much information as needed. I'm also willing to try any test cases you might want, and if I can get it to occur again (by upgrading back to 2.1, for example) then I'd be willing to capture and host a VM / VHD in that state. Just let me know whatever I can do to help.
(once I have a fully functional install of 2.1 I plan to help with bug testing and any other basic development tasks I can handle. I'm almost ready to bring up my new VM host environment for testing)
I still have not seen anyone else that can replicate this, but I have been extremely busy so I haven't been able to closely monitor the forum/irc.
Only thing that stands out is to maybe try it without jumbo frames and see if that makes a difference (Put the MTU back to 1500). There really isn't much of a reason to have the MTU as 9000 there, you're not passing traffic between two local interfaces that have jumbo frames on, and it's not likely to be helping all that much with retrieving data from the squid cache.
Seeing the ruleset from /tmp/rules.debug when it's working and when it's broken, to compare, may help also.
The ICMP rules are about the only rules I see in the config that have ipproto set to 'inet'. Most others do not have it set or it's set to inet6, so that's one potential place to look.
Jim P: I've upgraded to 2.1 again, and I'm not having that issue anymore, so I think we can mark this one as either a fluke or as resolved. It could just have been a less-than-graceful conversion of rules from the 2.0 format to the 2.1 format. I dunno. I'm not concerned by it anymore.
- Status changed from Feedback to Closed
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by