Project

General

Profile

Actions

Bug #13652

closed

Inconsistent behavior filtering ICMP traffic

Added by Serge Caron 3 months ago. Updated 1 day ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Rules / NAT
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Default
Affected Version:
2.6.x
Affected Architecture:

Description

I have the following FLOATING rules to filter out unwanted ICMP traffic on the network (these are repeated for all interfaces in the firewall but only the WAN, em1.66, is shown here):

block drop quick on em1.66 inet proto icmp all icmp-type timereq label "USER_RULE: Discard any and all ICMP Timestamp requests" ridentifier 1667827889
block drop quick on em1.66 inet proto icmp all icmp-type maskreq label "USER_RULE: Discard any and all ICMP Address mask requests" ridentifier 1667828285

block drop out log on em1.66 inet proto icmp all icmp-type timerep label "USER_RULE: Discard any and all ICMP Outgoing Timestamp replies" ridentifier 1667764490
block drop out log on em1.66 inet proto icmp all icmp-type maskrep label "USER_RULE: Discard any and all ICMP Outgoing Address mask re..." ridentifier 1667828326

Under unknown conditions, ICMP TimeStamp requests are allowed on the network and pfSense replies. This behavior does not occur for Address Mask requests.

I can reproduce this issue using the Qualys scanners over a set of consecutive IP addresses. I have placed a network TAP (Dualcomm ETAP-PI) in front of the ISP router and the relevant capture is attached. The pfSense host is labelled pfSenseWAN and handles two virtual IPs labeled pfSenseVirt and pfSenseVirtHost.

All Address mask requests from the Qulays scanner are dropped:

157 204.410768675  QualysScanner         pfSenseWANVirt        ICMP     60     Address mask request id=0x118f, seq=26729/26984, ttl=245
158 204.414693012 QualysScanner pfSenseWAN ICMP 60 Address mask request id=0x1192, seq=30825/27000, ttl=245
159 204.524882130 QualysScanner pfSenseVirtHost ICMP 60 Address mask request id=0x1195, seq=18537/26952, ttl=245

These are repeated twice in the attached capture (between sequence number 160 to 179)

However, TimeStamp requests are honored by the pfSense firewall itself:

138 202.411235592  QualysScanner         pfSenseWANVirt        ICMP     60     Timestamp request    id=0x118f, seq=26729/26984, ttl=245
139 202.411318979 pfSenseWANVirt QualysScanner ICMP 60 Timestamp reply id=0x118f, seq=26729/26984, ttl=64
140 202.414854325 QualysScanner pfSenseWAN ICMP 60 Timestamp request id=0x1192, seq=30825/27000, ttl=245
141 202.414940916 pfSenseWAN QualysScanner ICMP 60 Timestamp reply id=0x1192, seq=30825/27000, ttl=64

ICMP traffic that is NATed to an internal host is properly discarded (the second virtual IP is NATed to this host via simple NAT rules in this setup):

146 202.524821388  QualysScanner         pfSenseVirtHost       ICMP     60     Timestamp request    id=0x1195, seq=18537/26952, ttl=245
[...]
162 210.524959887 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
[...]
176 218.525489225 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245

Clearly, this behavior is inconsistent.

Even stranger, the inbound rules work as expected when the Qualys scanner or any other utility, such as hping3 or nmap, is used to target each host one at a time.

Also. when I disable the timereq/maskreq rules, I do not see the outgoing replies blocked in the pfSense Firewall log. My understanding is that this is the expected behavior from the above rules.

NOTE: I moved these rules to FLOATING because the behavior was the same when they were in the WAN ruleset.


Files

ICMP (No details).txt (13.4 KB) ICMP (No details).txt WireShark capture Serge Caron, 11/11/2022 10:40 AM
packet_capture_just_icmp_packets.pcap (4.54 KB) packet_capture_just_icmp_packets.pcap Infra Weavers, 01/20/2023 05:51 AM
config-TESTpfSense.home.arpa-20230124161406.xml (16.9 KB) config-TESTpfSense.home.arpa-20230124161406.xml Infra Weavers, 01/24/2023 10:14 AM
Scan_Profile.html (13.1 KB) Scan_Profile.html Infra Weavers, 01/24/2023 10:18 AM
Scan_2023-01-24_16-19-36.pcap (2.02 MB) Scan_2023-01-24_16-19-36.pcap Infra Weavers, 01/24/2023 10:19 AM
config-testpfsense.home.arpa-20230125115540.xml (12.7 KB) config-testpfsense.home.arpa-20230125115540.xml Infra Weavers, 01/25/2023 06:03 AM
PF_2.7.0_2023-01-25_11-56-46.pcap (2.08 MB) PF_2.7.0_2023-01-25_11-56-46.pcap Infra Weavers, 01/25/2023 06:03 AM
Option Profile Information.html (20.7 KB) Option Profile Information.html Qualys Profile Infra Weavers, 01/25/2023 09:34 AM
packetcapture_25_01_2023_15_43.pcap (358 KB) packetcapture_25_01_2023_15_43.pcap Infra Weavers, 01/25/2023 09:44 AM

Related issues

Related to Bug #13918: ICMP timestamp requests are passed by states created from ICMP echo requests if they use the same IDNewReid Linnemann

Actions
Actions #1

Updated by Infra Weavers 19 days ago

Clearly, this behavior is inconsistent.

Even stranger, the inbound rules work as expected when the Qualys scanner or any other utility, such as hping3 or nmap, is used to target each host one at a time.

We have, what we believe, to be the same exact behaviour; except that our WAN rule is:

pass in quick on $WAN reply-to ( lagg0 <GATEWAY> ) inet proto icmp from any to <WANSUBNET> icmp-type echoreq tracker 1633694002 keep state label "USER_RULE"

And if we scan 2 IP's using Qualys, we have traffic captures showing a response from the firewall. If we scan a single IP or try and force a reply using hping, we don't get a response.

Actions #2

Updated by Steve Wheeler 12 days ago

Have you been able to replicate this without using the Qualys Scanner?

Rules all work as expected for various ICMP types for me using hping as you found. I tested 22.05 and 23.01-Beta. I'm unable to replicate it.

Maybe you can supply an actual pcap file we can examine?

Actions #3

Updated by Serge Caron 12 days ago

Hello Steve,

I have not been able to replicate this with any other tool.

You have a PCAP file attached to this ticket. I do not have authorization to submit an actual PCAP file for the IPs used in this capture. I you want an actual PCAP file, I will have to arrange something different. The results will be the same as the contents of the attached file.

However, there is a larger issue: regardless of the inbound issue, the OUTGOING ICMP packets are not logged.

Regards,

Serge Caron

Actions #4

Updated by Infra Weavers 12 days ago

We have also been unable to reproduce this without the Qualys scanner; literally every other tool we have used has resulted in the traffic being blocked (i.e. hping, nping etc).

We can share an actual .pcap taken from PFSense (granted this will be our non-prod so it will be 2.4.5-RELEASE-p1 rather than 2.6.0, however it seems the bug is present on all versions both). I'll kick the process off for the captures now.

Actions #5

Updated by Infra Weavers 12 days ago

Please find attached the packet capture reduced down to just ICMP traffic. The associated firewall rule is:

pass  in  quick  on $WAN2_DSL reply-to ( igb2 <gateway> ) inet proto icmp  from any to <WANsubnet> icmp-type echoreq tracker 1464807726 keep state  label "USER_RULE" 

I have filtered the pcap down to just the ICMP packets (so you can see the ping responses, timestamp responses, but no address mask responses).

The pcap was taken without any other filtering so if other traffic being sent is relevant, I can forward to you the whole pcap (1.8MB)

Actions #6

Updated by Marcos M 10 days ago

  • Status changed from New to Not a Bug
I could not replicate this either on 23.01 using Qualys with the following scan options:
  • All ICMP QIDs selected (including ICMP Mask Reply, ICMP Timestamp Request, ICMP Replies Received)
  • Scan range of 198.51.100.4-198.51.100.6
  • No TCP/UDP scanning

The firewall rule used was:

pass  in  quick  on $ISP1 reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp  from any to 198.51.100.0/24 icmp-type { echoreq,timex,trace,unreach }  !  tagged "blocklist" ridentifier 1529162985 keep state label "USER_RULE: ping to firewall" label "id:1529162985" 

pcap:

No.    Time    Source    Destination    Protocol    Length    Info
411    48.200547    64.39.98.10    198.51.100.5    ICMP    60    Echo (ping) request  id=0x90ab, seq=37035/43920, ttl=54 (reply in 412)
412    48.200642    198.51.100.5    64.39.98.10    ICMP    50    Echo (ping) reply    id=0x90ab, seq=37035/43920, ttl=64 (request in 411)
413    48.369620    64.39.98.10    198.51.100.5    ICMP    60    Timestamp request    id=0x3be3, seq=2931/29451, ttl=245
414    48.391532    64.39.98.10    198.51.100.5    ICMP    90    Echo (ping) request  id=0x3bef, seq=15343/61243, ttl=245 (reply in 415)
415    48.391581    198.51.100.5    64.39.98.10    ICMP    90    Echo (ping) reply    id=0x3bef, seq=15343/61243, ttl=64 (request in 414)
416    50.371693    64.39.98.10    198.51.100.5    ICMP    60    Address mask request id=0x3be3, seq=2931/29451, ttl=245
417    53.396186    64.39.98.10    198.51.100.5    ICMP    95    Echo (ping) request  id=0x3bef, seq=28346/47726, ttl=245 (reply in 418)
418    53.396224    198.51.100.5    64.39.98.10    ICMP    95    Echo (ping) reply    id=0x3bef, seq=28346/47726, ttl=64 (request in 417)
419    56.376805    64.39.98.10    198.51.100.5    ICMP    60    Timestamp request    id=0x3be3, seq=2931/29451, ttl=245
420    58.379240    64.39.98.10    198.51.100.5    ICMP    60    Address mask request id=0x3be3, seq=2931/29451, ttl=245
573    64.384944    64.39.98.10    198.51.100.5    ICMP    60    Timestamp request    id=0x3be3, seq=2931/29451, ttl=245
632    66.386656    64.39.98.10    198.51.100.5    ICMP    60    Address mask request id=0x3be3, seq=2931/29451, ttl=245

Either this does not affect 23.01/2.7, or there's something specific to the environment in the reported cases. I'm going to mark this as Not a Bug until it can be reproduced.

Actions #7

Updated by Serge Caron 10 days ago

Hello Marcos,

I don't know how you specified the hosts range in the Qualys scanner.

In the log you provided, we can see only timestamp requests for a single host. When that test runs for multiple hosts at the same time, we can see timestamp requests for each host in the sequence, such as:
138 202.411235592 QualysScanner pfSenseWANVirt ICMP 60 Timestamp request id=0x118f, seq=26729/26984, ttl=245
139 202.411318979 pfSenseWANVirt QualysScanner ICMP 60 Timestamp reply id=0x118f, seq=26729/26984, ttl=64
140 202.414854325 QualysScanner pfSenseWAN ICMP 60 Timestamp request id=0x1192, seq=30825/27000, ttl=245
141 202.414940916 pfSenseWAN QualysScanner ICMP 60 Timestamp reply id=0x1192, seq=30825/27000, ttl=64
142 202.459692983 QualysScanner pfSenseWAN ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 143)
143 202.459898978 pfSenseWAN QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=64 (request in 142)
144 202.465811401 QualysScanner pfSenseWANVirt ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 145)
145 202.466066747 pfSenseWANVirt QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=64 (request in 144)
146 202.524821388 QualysScanner pfSenseVirtHost ICMP 60 Timestamp request id=0x1195, seq=18537/26952, ttl=245
147 202.574570121 QualysScanner pfSenseVirtHost ICMP 60 Echo (ping) request id=0x1e20, seq=7712/8222, ttl=20 (reply in 148)
148 202.575220810 pfSenseVirtHost QualysScanner ICMP 60 Echo (ping) reply id=0x1e20, seq=7712/8222, ttl=127 (request in 147)
149 203.978816545 QualysScanner pfSenseWAN ICMP 60 Echo (ping) request id=0x0b0b, seq=2827/2827, ttl=1 (reply in 150)
150 203.978934486 pfSenseWAN QualysScanner ICMP 60 Echo (ping) reply id=0x0b0b, seq=2827/2827, ttl=64 (request in 149)
151 203.979088724 pfSenseWAN QualysScanner ICMP 82 Time-to-live exceeded (Time to live exceeded in transit)

where pfSenseWANVirt, pfSenseWAN, and pfSenseVirtHost are consecutive hosts in the same subnet. Look at the sequence in packets 138 to 148.

That I can reproduce with 2.6 using the Qualys scanner.

Regards,

Serge Caron

Actions #8

Updated by Marcos M 10 days ago

It's not listed there because the VIP address doesn't actually reach pfSense in my test, only the primary interface address. All reports so far show replies from the WAN address as well, hence I'd expect the same behavior here if it was an issue on 23.01/2.7.

Actions #9

Updated by Infra Weavers 9 days ago

It's not listed there because the VIP address doesn't actually reach pfSense in my test, only the primary interface address. All reports so far show replies from the WAN address as well, hence I'd expect the same behavior here if it was an issue on 23.01/2.7.

Ahh! So it's critical that both IP's are reachable and on the firewalls. We tested a plethora of different configurations, unless two sequential IP's on the same PFSense host are being tested by Qualys in the same test, the behaviour doesn't happen.

If you look at the traffic capture that we provided above, both the primary IP (.155) and the IP alias (.156) respond with netmask responses.

Actions #10

Updated by Serge Caron 9 days ago

To add to these observations, the issue does NOT occur for Address Mask requests even when sequential IPs are used.

There are various conditions under which you want to use virtual IPs as a mean to protect internal equipments but ICMP is always difficult to handle : it should be forwarded to the internal equipment to satisfy normal trafic flow unless it is used with some malicious intent.

In our environment, we don't want timestamp, address mask, or info requests/replies to cross any subnet, hence the floating rules ... which don't seem to work as I understand them.

Regards,

Serge Caron

Actions #11

Updated by Marcos M 9 days ago

I didn't expect there to be a difference between a single address and multiple address, but I've now tested with multiple addresses on an old 22.09 build - results below. ICMP echo request/replies are not included in this pcap:

No.    Time    Source    Destination    Protocol    Length    Info
600    98.114132    64.39.98.235    192.0.2.5    ICMP    60    Timestamp request    id=0x71e2, seq=55697/37337, ttl=248
602    98.440478    64.39.98.235    192.0.2.4    ICMP    60    Timestamp request    id=0x71e8, seq=2449/37129, ttl=248
616    100.116273    64.39.98.235    192.0.2.5    ICMP    60    Address mask request id=0x71e2, seq=55697/37337, ttl=248
617    100.442648    64.39.98.235    192.0.2.4    ICMP    60    Address mask request id=0x71e8, seq=2449/37129, ttl=248
669    108.123466    64.39.98.235    192.0.2.5    ICMP    60    Timestamp request    id=0x71e2, seq=55697/37337, ttl=248
671    108.450430    64.39.98.235    192.0.2.4    ICMP    60    Timestamp request    id=0x71e8, seq=2449/37129, ttl=248
675    110.125602    64.39.98.235    192.0.2.5    ICMP    60    Address mask request id=0x71e2, seq=55697/37337, ttl=248
678    110.452432    64.39.98.235    192.0.2.4    ICMP    60    Address mask request id=0x71e8, seq=2449/37129, ttl=248
724    118.133618    64.39.98.235    192.0.2.5    ICMP    60    Timestamp request    id=0x71e2, seq=55697/37337, ttl=248
728    118.460800    64.39.98.235    192.0.2.4    ICMP    60    Timestamp request    id=0x71e8, seq=2449/37129, ttl=248
757    120.135662    64.39.98.235    192.0.2.5    ICMP    60    Address mask request id=0x71e2, seq=55697/37337, ttl=248
759    120.462774    64.39.98.235    192.0.2.4    ICMP    60    Address mask request id=0x71e8, seq=2449/37129, ttl=248

Actions #12

Updated by Infra Weavers 8 days ago

Hiya Marcos,

We've just reproduced this on a totally stock PFsense 2.6.0 install. The only things we did was to configure an IP alias to add a second address and add the firewall rule. I'm attaching the backup of the configuration (config-TESTpfSense.home.arpa-20230124161406.xml).

We scanned using 81.143.222.156-81.143.222.157 as the range in Qualys. I've attached the bottom part of the report so you can see the scan options we used (Scan_profile.html). As this is a test box I have also included the entire packet capture (Scan_2023-01-24_16-19-36.pcap) which shows (packets no 1179, 1180, 1189,1190) timestamp requests and timestamp responses. You can also see addressmask requests however they are not responded to.

We're going to upgrade to a 2.7.0 snapshot now and see if the problem is still reproducable.

Thanks,
Rob

Actions #13

Updated by Infra Weavers 7 days ago

Hello,

We have just tested pfSense-CE-memstick-2.7.0-DEVELOPMENT-amd64-20230125-0600.img.gz and we are seeing the ICMP timestamp responses on there as well.

We have attached the backup of the 2.7.0 configuration (config-testpfsense.home.arpa-20230125115540.xml) so you can see the rule that's added etc, it's basically identical to the 2.6.0 configuration uploaded yesterday.

I have also attached the pcap of all the traffic so it can be verified that there are ICMP timestamp responses.

The target of the scan was: 81.143.222.156-81.143.222.157

The option profile within Qualys was as follows:

Title:
DEFAULT_VM_OPTIONS_PROFILE_2022
Options:
Standard TCP port list and Additional TCP Port List : 8080, 8081, 8082, 8083, 8084, 8085, 8090, 50000, 50001, 50003, 33848, Standard UDP port list and Additional UDP Port List : 8080, 8081, 8082, 8083, 8084, 8085, 8090, 50000, 50001, 50002, 33848, parallel ML scaling disabled for appliances, Load balancer detection ON, Intrusive Checks: Excluded, Windows Authentication ON, Unix Authentication ON, SNMP Authentication ON, VMware Authentication ON, HTTP Authentication ON, Ignore firewall-generated RST packets, Ignore firewall-generated SYN-ACK packets, Custom Host Discovery TCP Port list and Additional Host Discovery TCP Port list 21-23, 25, 53, 80, 88, 110-111, 135, 139, 443, 445, 1433, 1521, 1525, 1526, 1527, 1529, 1571, ICMP Host Discovery, Overall Performance: Normal, Allow Parallel Scanning: Disabled, Hosts to Scan in Parallel - External Scanners: 15, Hosts to Scan in Parallel - Scanner Appliances: 30, Total Processes to Run in Parallel: 10, HTTP Processes to Run in Parallel: 10, Packet (Burst) Delay: Medium, Intensity: Normal

I am currently getting our Qualys guys to configure a profile that is ICMP checks only, so we can see if the issue only occurs with mixed traffic (which is my hunch).

I will report back if we get any more information.

Thanks

Actions #14

Updated by Infra Weavers 7 days ago

Hiya,

So we think we have got this down the smallest scan we can (takes about 90 seconds). There unfortunately isn't a way of exporting or saving the policies out of qualysguard.qg2.apps.qualys.com; so I've hit "view" and saved the HTML of the profile. It's not the cleanest/easiest to read however all the settings are there. Myself and someone else set up two different scans in Qualys, mine didn't show the problem, his did. Turned out that I had ticked "Include dead hosts in scans" and "Enable host alive testing". That combination made the scan not report it back.

We have also attached the traffic capture using that profile (which is much smaller than the others as you would expect) which was run 139.87.112.105 (Scanner 12.12.48-1, Vulnerability Signatures 2.5.684-2)

Just to clarify, we are running from the Qualys Cloud Platform, not from any on-prem agent or anything.

Thanks,

Actions #15

Updated by Marcos M 1 day ago

  • Status changed from Not a Bug to Confirmed
I was able to reproduce this in 23.01. The scan options required are:
  • A Search List with the 82003¬†ICMP Timestamp Request QID.
  • An Options Profile with Host Discovery: ICMP turned on; all other tests/scans/options disabled (in particular, Include dead hosts in scans and Enable host alive testing).
  • A consecutive target range (confirmed it fails with a single host as the target).

On pfSense, allow ICMP echo requests (needed for the scanner to detect the host and subsequently run the timestamp scan), and block all other ICMP:

pass  in  quick  on {  vmx0.99  } reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp  from any to 198.51.100.0/24 icmp-type echoreq ridentifier 1675104047 keep state label "USER_RULE: testa" label "id:1675104047" 
block  in log  quick  on {  vmx0.99  } reply-to ( vmx0.99 198.51.100.1 ) inet proto icmp  from any to 198.51.100.0/24 ridentifier 1675103204 label "USER_RULE: testb" label "id:1675103204" 

Packet capture of scan:

No.    Time    Delta    Source    Destination    Protocol    Length    Info
987    97.946388    0.008637    64.39.98.3    198.51.100.5    ICMP    60    Echo (ping) request  id=0x0000, seq=0/0, ttl=246 (reply in 988)
988    97.946412    0.000024    198.51.100.5    64.39.98.3    ICMP    42    Echo (ping) reply    id=0x0000, seq=0/0, ttl=64 (request in 987)
1055    99.849817    0.030909    64.39.98.3    198.51.100.5    ICMP    60    Echo (ping) request  id=0x4bc5, seq=19755/11085, ttl=246 (reply in 1056)
1056    99.849844    0.000027    198.51.100.5    64.39.98.3    ICMP    42    Echo (ping) reply    id=0x4bc5, seq=19755/11085, ttl=64 (request in 1055)
1103    101.850952    0.102859    64.39.98.3    198.51.100.5    ICMP    60    Timestamp request    id=0x4bc5, seq=19755/11085, ttl=246
1104    101.850965    0.000013    198.51.100.5    64.39.98.3    ICMP    54    Timestamp reply      id=0x4bc5, seq=19755/11085, ttl=64
1169    103.853321    0.045354    64.39.98.3    198.51.100.5    ICMP    60    Address mask request id=0x4bc5, seq=19755/11085, ttl=246
1253    109.860308    0.176726    64.39.98.3    198.51.100.5    ICMP    60    Address mask request id=0x4bc5, seq=19755/11085, ttl=246
1300    115.866207    0.023488    64.39.98.3    198.51.100.5    ICMP    60    Address mask request id=0x4bc5, seq=19755/11085, ttl=246

Actions #16

Updated by Marcos M 1 day ago

  • Status changed from Confirmed to Closed

I've created a separate report with specific details and easily reproducible steps; I'm going to close this one out and link it as a related issue.

Actions #17

Updated by Marcos M 1 day ago

  • Related to Bug #13918: ICMP timestamp requests are passed by states created from ICMP echo requests if they use the same ID added
Actions

Also available in: Atom PDF