Bug #13652
closedInconsistent behavior filtering ICMP traffic
0%
Description
I have the following FLOATING rules to filter out unwanted ICMP traffic on the network (these are repeated for all interfaces in the firewall but only the WAN, em1.66, is shown here):
block drop quick on em1.66 inet proto icmp all icmp-type timereq label "USER_RULE: Discard any and all ICMP Timestamp requests" ridentifier 1667827889
block drop quick on em1.66 inet proto icmp all icmp-type maskreq label "USER_RULE: Discard any and all ICMP Address mask requests" ridentifier 1667828285
block drop out log on em1.66 inet proto icmp all icmp-type timerep label "USER_RULE: Discard any and all ICMP Outgoing Timestamp replies" ridentifier 1667764490
block drop out log on em1.66 inet proto icmp all icmp-type maskrep label "USER_RULE: Discard any and all ICMP Outgoing Address mask re..." ridentifier 1667828326
Under unknown conditions, ICMP TimeStamp requests are allowed on the network and pfSense replies. This behavior does not occur for Address Mask requests.
I can reproduce this issue using the Qualys scanners over a set of consecutive IP addresses. I have placed a network TAP (Dualcomm ETAP-PI) in front of the ISP router and the relevant capture is attached. The pfSense host is labelled pfSenseWAN and handles two virtual IPs labeled pfSenseVirt and pfSenseVirtHost.
All Address mask requests from the Qulays scanner are dropped:
157 204.410768675 QualysScanner pfSenseWANVirt ICMP 60 Address mask request id=0x118f, seq=26729/26984, ttl=245
158 204.414693012 QualysScanner pfSenseWAN ICMP 60 Address mask request id=0x1192, seq=30825/27000, ttl=245
159 204.524882130 QualysScanner pfSenseVirtHost ICMP 60 Address mask request id=0x1195, seq=18537/26952, ttl=245
These are repeated twice in the attached capture (between sequence number 160 to 179)
However, TimeStamp requests are honored by the pfSense firewall itself:
138 202.411235592 QualysScanner pfSenseWANVirt ICMP 60 Timestamp request id=0x118f, seq=26729/26984, ttl=245
139 202.411318979 pfSenseWANVirt QualysScanner ICMP 60 Timestamp reply id=0x118f, seq=26729/26984, ttl=64
140 202.414854325 QualysScanner pfSenseWAN ICMP 60 Timestamp request id=0x1192, seq=30825/27000, ttl=245
141 202.414940916 pfSenseWAN QualysScanner ICMP 60 Timestamp reply id=0x1192, seq=30825/27000, ttl=64
ICMP traffic that is NATed to an internal host is properly discarded (the second virtual IP is NATed to this host via simple NAT rules in this setup):
146 202.524821388 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
[...]
162 210.524959887 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
[...]
176 218.525489225 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
Clearly, this behavior is inconsistent.
Even stranger, the inbound rules work as expected when the Qualys scanner or any other utility, such as hping3 or nmap, is used to target each host one at a time.
Also. when I disable the timereq/maskreq rules, I do not see the outgoing replies blocked in the pfSense Firewall log. My understanding is that this is the expected behavior from the above rules.
NOTE: I moved these rules to FLOATING because the behavior was the same when they were in the WAN ruleset.
Files
Related issues
Updated by Infra Weavers almost 2 years ago
Clearly, this behavior is inconsistent.
Even stranger, the inbound rules work as expected when the Qualys scanner or any other utility, such as hping3 or nmap, is used to target each host one at a time.
We have, what we believe, to be the same exact behaviour; except that our WAN rule is:
pass in quick on $WAN reply-to ( lagg0 <GATEWAY> ) inet proto icmp from any to <WANSUBNET> icmp-type echoreq tracker 1633694002 keep state label "USER_RULE"
And if we scan 2 IP's using Qualys, we have traffic captures showing a response from the firewall. If we scan a single IP or try and force a reply using hping, we don't get a response.
Updated by Steve Wheeler almost 2 years ago
Have you been able to replicate this without using the Qualys Scanner?
Rules all work as expected for various ICMP types for me using hping as you found. I tested 22.05 and 23.01-Beta. I'm unable to replicate it.
Maybe you can supply an actual pcap file we can examine?
Updated by Serge Caron almost 2 years ago
Hello Steve,
I have not been able to replicate this with any other tool.
You have a PCAP file attached to this ticket. I do not have authorization to submit an actual PCAP file for the IPs used in this capture. I you want an actual PCAP file, I will have to arrange something different. The results will be the same as the contents of the attached file.
However, there is a larger issue: regardless of the inbound issue, the OUTGOING ICMP packets are not logged.
Regards,
Serge Caron
Updated by Infra Weavers almost 2 years ago
We have also been unable to reproduce this without the Qualys scanner; literally every other tool we have used has resulted in the traffic being blocked (i.e. hping, nping etc).
We can share an actual .pcap taken from PFSense (granted this will be our non-prod so it will be 2.4.5-RELEASE-p1 rather than 2.6.0, however it seems the bug is present on all versions both). I'll kick the process off for the captures now.
Updated by Infra Weavers almost 2 years ago
Please find attached the packet capture reduced down to just ICMP traffic. The associated firewall rule is:
pass in quick on $WAN2_DSL reply-to ( igb2 <gateway> ) inet proto icmp from any to <WANsubnet> icmp-type echoreq tracker 1464807726 keep state label "USER_RULE"
I have filtered the pcap down to just the ICMP packets (so you can see the ping responses, timestamp responses, but no address mask responses).
The pcap was taken without any other filtering so if other traffic being sent is relevant, I can forward to you the whole pcap (1.8MB)
Updated by Marcos M almost 2 years ago
- Status changed from New to Not a Bug
- All ICMP QIDs selected (including
ICMP Mask Reply
,ICMP Timestamp Request
,ICMP Replies Received
) - Scan range of
198.51.100.4-198.51.100.6
- No TCP/UDP scanning
The firewall rule used was:
pass in quick on $ISP1 reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp from any to 198.51.100.0/24 icmp-type { echoreq,timex,trace,unreach } ! tagged "blocklist" ridentifier 1529162985 keep state label "USER_RULE: ping to firewall" label "id:1529162985"
pcap:
No. Time Source Destination Protocol Length Info 411 48.200547 64.39.98.10 198.51.100.5 ICMP 60 Echo (ping) request id=0x90ab, seq=37035/43920, ttl=54 (reply in 412) 412 48.200642 198.51.100.5 64.39.98.10 ICMP 50 Echo (ping) reply id=0x90ab, seq=37035/43920, ttl=64 (request in 411) 413 48.369620 64.39.98.10 198.51.100.5 ICMP 60 Timestamp request id=0x3be3, seq=2931/29451, ttl=245 414 48.391532 64.39.98.10 198.51.100.5 ICMP 90 Echo (ping) request id=0x3bef, seq=15343/61243, ttl=245 (reply in 415) 415 48.391581 198.51.100.5 64.39.98.10 ICMP 90 Echo (ping) reply id=0x3bef, seq=15343/61243, ttl=64 (request in 414) 416 50.371693 64.39.98.10 198.51.100.5 ICMP 60 Address mask request id=0x3be3, seq=2931/29451, ttl=245 417 53.396186 64.39.98.10 198.51.100.5 ICMP 95 Echo (ping) request id=0x3bef, seq=28346/47726, ttl=245 (reply in 418) 418 53.396224 198.51.100.5 64.39.98.10 ICMP 95 Echo (ping) reply id=0x3bef, seq=28346/47726, ttl=64 (request in 417) 419 56.376805 64.39.98.10 198.51.100.5 ICMP 60 Timestamp request id=0x3be3, seq=2931/29451, ttl=245 420 58.379240 64.39.98.10 198.51.100.5 ICMP 60 Address mask request id=0x3be3, seq=2931/29451, ttl=245 573 64.384944 64.39.98.10 198.51.100.5 ICMP 60 Timestamp request id=0x3be3, seq=2931/29451, ttl=245 632 66.386656 64.39.98.10 198.51.100.5 ICMP 60 Address mask request id=0x3be3, seq=2931/29451, ttl=245
Either this does not affect 23.01/2.7, or there's something specific to the environment in the reported cases. I'm going to mark this as Not a Bug
until it can be reproduced.
Updated by Serge Caron almost 2 years ago
Hello Marcos,
I don't know how you specified the hosts range in the Qualys scanner.
In the log you provided, we can see only timestamp requests for a single host. When that test runs for multiple hosts at the same time, we can see timestamp requests for each host in the sequence, such as:
138 202.411235592 QualysScanner pfSenseWANVirt ICMP 60 Timestamp request id=0x118f, seq=26729/26984, ttl=245
139 202.411318979 pfSenseWANVirt QualysScanner ICMP 60 Timestamp reply id=0x118f, seq=26729/26984, ttl=64
140 202.414854325 QualysScanner pfSenseWAN ICMP 60 Timestamp request id=0x1192, seq=30825/27000, ttl=245
141 202.414940916 pfSenseWAN QualysScanner ICMP 60 Timestamp reply id=0x1192, seq=30825/27000, ttl=64
142 202.459692983 QualysScanner pfSenseWAN ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 143)
143 202.459898978 pfSenseWAN QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=64 (request in 142)
144 202.465811401 QualysScanner pfSenseWANVirt ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 145)
145 202.466066747 pfSenseWANVirt QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=64 (request in 144)
146 202.524821388 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
147 202.574570121 QualysScanner pfSenseVirtHost ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 148)
148 202.575220810 pfSenseVirtHost QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=127 (request in 147)
149 203.978816545 QualysScanner pfSenseWAN ICMP 60 Echo (ping) request id=0x0b0b, seq=2827/2827, ttl=1 (reply in 150)
150 203.978934486 pfSenseWAN QualysScanner ICMP 60 Echo (ping) reply id=0x0b0b, seq=2827/2827, ttl=64 (request in 149)
151 203.979088724 pfSenseWAN QualysScanner ICMP 82 Time-to-live exceeded (Time to live exceeded in transit)
where pfSenseWANVirt, pfSenseWAN, and pfSenseVirtHost are consecutive hosts in the same subnet. Look at the sequence in packets 138 to 148.
That I can reproduce with 2.6 using the Qualys scanner.
Regards,
Serge Caron
Updated by Marcos M almost 2 years ago
It's not listed there because the VIP address doesn't actually reach pfSense in my test, only the primary interface address. All reports so far show replies from the WAN address as well, hence I'd expect the same behavior here if it was an issue on 23.01/2.7.
Updated by Infra Weavers almost 2 years ago
It's not listed there because the VIP address doesn't actually reach pfSense in my test, only the primary interface address. All reports so far show replies from the WAN address as well, hence I'd expect the same behavior here if it was an issue on 23.01/2.7.
Ahh! So it's critical that both IP's are reachable and on the firewalls. We tested a plethora of different configurations, unless two sequential IP's on the same PFSense host are being tested by Qualys in the same test, the behaviour doesn't happen.
If you look at the traffic capture that we provided above, both the primary IP (.155) and the IP alias (.156) respond with netmask responses.
Updated by Serge Caron almost 2 years ago
To add to these observations, the issue does NOT occur for Address Mask requests even when sequential IPs are used.
There are various conditions under which you want to use virtual IPs as a mean to protect internal equipments but ICMP is always difficult to handle : it should be forwarded to the internal equipment to satisfy normal trafic flow unless it is used with some malicious intent.
In our environment, we don't want timestamp, address mask, or info requests/replies to cross any subnet, hence the floating rules ... which don't seem to work as I understand them.
Regards,
Serge Caron
Updated by Marcos M almost 2 years ago
I didn't expect there to be a difference between a single address and multiple address, but I've now tested with multiple addresses on an old 22.09 build - results below. ICMP echo request/replies are not included in this pcap:
No. Time Source Destination Protocol Length Info 600 98.114132 64.39.98.235 192.0.2.5 ICMP 60 Timestamp request id=0x71e2, seq=55697/37337, ttl=248 602 98.440478 64.39.98.235 192.0.2.4 ICMP 60 Timestamp request id=0x71e8, seq=2449/37129, ttl=248 616 100.116273 64.39.98.235 192.0.2.5 ICMP 60 Address mask request id=0x71e2, seq=55697/37337, ttl=248 617 100.442648 64.39.98.235 192.0.2.4 ICMP 60 Address mask request id=0x71e8, seq=2449/37129, ttl=248 669 108.123466 64.39.98.235 192.0.2.5 ICMP 60 Timestamp request id=0x71e2, seq=55697/37337, ttl=248 671 108.450430 64.39.98.235 192.0.2.4 ICMP 60 Timestamp request id=0x71e8, seq=2449/37129, ttl=248 675 110.125602 64.39.98.235 192.0.2.5 ICMP 60 Address mask request id=0x71e2, seq=55697/37337, ttl=248 678 110.452432 64.39.98.235 192.0.2.4 ICMP 60 Address mask request id=0x71e8, seq=2449/37129, ttl=248 724 118.133618 64.39.98.235 192.0.2.5 ICMP 60 Timestamp request id=0x71e2, seq=55697/37337, ttl=248 728 118.460800 64.39.98.235 192.0.2.4 ICMP 60 Timestamp request id=0x71e8, seq=2449/37129, ttl=248 757 120.135662 64.39.98.235 192.0.2.5 ICMP 60 Address mask request id=0x71e2, seq=55697/37337, ttl=248 759 120.462774 64.39.98.235 192.0.2.4 ICMP 60 Address mask request id=0x71e8, seq=2449/37129, ttl=248
Updated by Infra Weavers almost 2 years ago
- File config-TESTpfSense.home.arpa-20230124161406.xml config-TESTpfSense.home.arpa-20230124161406.xml added
- File Scan_Profile.html Scan_Profile.html added
- File Scan_2023-01-24_16-19-36.pcap Scan_2023-01-24_16-19-36.pcap added
Hiya Marcos,
We've just reproduced this on a totally stock PFsense 2.6.0 install. The only things we did was to configure an IP alias to add a second address and add the firewall rule. I'm attaching the backup of the configuration (config-TESTpfSense.home.arpa-20230124161406.xml).
We scanned using 81.143.222.156-81.143.222.157 as the range in Qualys. I've attached the bottom part of the report so you can see the scan options we used (Scan_profile.html). As this is a test box I have also included the entire packet capture (Scan_2023-01-24_16-19-36.pcap) which shows (packets no 1179, 1180, 1189,1190) timestamp requests and timestamp responses. You can also see addressmask requests however they are not responded to.
We're going to upgrade to a 2.7.0 snapshot now and see if the problem is still reproducable.
Thanks,
Rob
Updated by Infra Weavers almost 2 years ago
- File config-testpfsense.home.arpa-20230125115540.xml config-testpfsense.home.arpa-20230125115540.xml added
- File PF_2.7.0_2023-01-25_11-56-46.pcap PF_2.7.0_2023-01-25_11-56-46.pcap added
Hello,
We have just tested pfSense-CE-memstick-2.7.0-DEVELOPMENT-amd64-20230125-0600.img.gz and we are seeing the ICMP timestamp responses on there as well.
We have attached the backup of the 2.7.0 configuration (config-testpfsense.home.arpa-20230125115540.xml) so you can see the rule that's added etc, it's basically identical to the 2.6.0 configuration uploaded yesterday.
I have also attached the pcap of all the traffic so it can be verified that there are ICMP timestamp responses.
The target of the scan was: 81.143.222.156-81.143.222.157
The option profile within Qualys was as follows:
Title: DEFAULT_VM_OPTIONS_PROFILE_2022 Options: Standard TCP port list and Additional TCP Port List : 8080, 8081, 8082, 8083, 8084, 8085, 8090, 50000, 50001, 50003, 33848, Standard UDP port list and Additional UDP Port List : 8080, 8081, 8082, 8083, 8084, 8085, 8090, 50000, 50001, 50002, 33848, parallel ML scaling disabled for appliances, Load balancer detection ON, Intrusive Checks: Excluded, Windows Authentication ON, Unix Authentication ON, SNMP Authentication ON, VMware Authentication ON, HTTP Authentication ON, Ignore firewall-generated RST packets, Ignore firewall-generated SYN-ACK packets, Custom Host Discovery TCP Port list and Additional Host Discovery TCP Port list 21-23, 25, 53, 80, 88, 110-111, 135, 139, 443, 445, 1433, 1521, 1525, 1526, 1527, 1529, 1571, ICMP Host Discovery, Overall Performance: Normal, Allow Parallel Scanning: Disabled, Hosts to Scan in Parallel - External Scanners: 15, Hosts to Scan in Parallel - Scanner Appliances: 30, Total Processes to Run in Parallel: 10, HTTP Processes to Run in Parallel: 10, Packet (Burst) Delay: Medium, Intensity: Normal
I am currently getting our Qualys guys to configure a profile that is ICMP checks only, so we can see if the issue only occurs with mixed traffic (which is my hunch).
I will report back if we get any more information.
Thanks
Updated by Infra Weavers almost 2 years ago
- File Option Profile Information.html Option Profile Information.html added
- File packetcapture_25_01_2023_15_43.pcap packetcapture_25_01_2023_15_43.pcap added
Hiya,
So we think we have got this down the smallest scan we can (takes about 90 seconds). There unfortunately isn't a way of exporting or saving the policies out of qualysguard.qg2.apps.qualys.com; so I've hit "view" and saved the HTML of the profile. It's not the cleanest/easiest to read however all the settings are there. Myself and someone else set up two different scans in Qualys, mine didn't show the problem, his did. Turned out that I had ticked "Include dead hosts in scans" and "Enable host alive testing". That combination made the scan not report it back.
We have also attached the traffic capture using that profile (which is much smaller than the others as you would expect) which was run 139.87.112.105 (Scanner 12.12.48-1, Vulnerability Signatures 2.5.684-2)
Just to clarify, we are running from the Qualys Cloud Platform, not from any on-prem agent or anything.
Thanks,
Updated by Marcos M almost 2 years ago
- Status changed from Not a Bug to Confirmed
- A
Search List
with the82003 ICMP Timestamp Request
QID. - An
Options Profile
withHost Discovery: ICMP
turned on; all other tests/scans/options disabled (in particular,Include dead hosts in scans
andEnable host alive testing
). - A consecutive target range (confirmed it fails with a single host as the target).
On pfSense, allow ICMP echo requests (needed for the scanner to detect the host and subsequently run the timestamp scan), and block all other ICMP:
pass in quick on { vmx0.99 } reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp from any to 198.51.100.0/24 icmp-type echoreq ridentifier 1675104047 keep state label "USER_RULE: testa" label "id:1675104047" block in log quick on { vmx0.99 } reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp from any to 198.51.100.0/24 ridentifier 1675103204 label "USER_RULE: testb" label "id:1675103204"
Packet capture of scan:
No. Time Delta Source Destination Protocol Length Info 987 97.946388 0.008637 64.39.98.3 198.51.100.5 ICMP 60 Echo (ping) request id=0x0000, seq=0/0, ttl=246 (reply in 988) 988 97.946412 0.000024 198.51.100.5 64.39.98.3 ICMP 42 Echo (ping) reply id=0x0000, seq=0/0, ttl=64 (request in 987) 1055 99.849817 0.030909 64.39.98.3 198.51.100.5 ICMP 60 Echo (ping) request id=0x4bc5, seq=19755/11085, ttl=246 (reply in 1056) 1056 99.849844 0.000027 198.51.100.5 64.39.98.3 ICMP 42 Echo (ping) reply id=0x4bc5, seq=19755/11085, ttl=64 (request in 1055) 1103 101.850952 0.102859 64.39.98.3 198.51.100.5 ICMP 60 Timestamp request id=0x4bc5, seq=19755/11085, ttl=246 1104 101.850965 0.000013 198.51.100.5 64.39.98.3 ICMP 54 Timestamp reply id=0x4bc5, seq=19755/11085, ttl=64 1169 103.853321 0.045354 64.39.98.3 198.51.100.5 ICMP 60 Address mask request id=0x4bc5, seq=19755/11085, ttl=246 1253 109.860308 0.176726 64.39.98.3 198.51.100.5 ICMP 60 Address mask request id=0x4bc5, seq=19755/11085, ttl=246 1300 115.866207 0.023488 64.39.98.3 198.51.100.5 ICMP 60 Address mask request id=0x4bc5, seq=19755/11085, ttl=246
Updated by Marcos M almost 2 years ago
- Status changed from Confirmed to Closed
I've created a separate report with specific details and easily reproducible steps; I'm going to close this one out and link it as a related issue.
Updated by Marcos M almost 2 years ago
- Related to Bug #13918: ICMP timestamp requests are passed by states created from ICMP echo requests if they use the same ID added