pfSense » pfSense Packages

I can't seem to edit my initial post but wanted to clarify the Squid debug option should be debug_options ALL,1 rotate=7 - the previous underlining obscured the necessary underline in the option.

https://redmine.pfsense.org/issues/14786

I have also seen "UPP" utilizing this to get around non transparent mode when you have a single URL set to splice UPP takes it and uses that to piggyback to other URLs with the same IP address. Essentially a VPN to all the other URLs that are not spliced normally, utilizing the open spliced connection. A Get Headder Forgery

Could Squids storeID help resolve this?

https://wiki.squid-cache.org/Features/StoreID

https://forum.netgate.com/topic/182487/storeid-and-squid-helper-program/

https://github.com/rudiservo/pfsense_storeid

This program was made for CDN maybe it can be expanded

host verify strict is set to OFF by default so technically we souldnt be having these /409 errors.
My suspicion is that Squid package is not respecting this setting. This is a regression in the package that hasnt been caught until now. IMO.

Hi Mike, (and others)

Thanks for commenting and having a look at this - I agree, with "host_verify_strict off", which is the default, 409 errors should not be returned, so squid is not behaving as documented at http://www.squid-cache.org/Doc/config/host_verify_strict/ and doesn't seem to be respecting the setting correctly.

From reading I've done since my original post it seems that requests that fail the hostname/IP "coherency check", lets call it, should be marked as un-cachable assets (to avoid someone else retrieving them from the cache) and should also not be forwarded to squid peers but go directly to the destination server using the original IP address the client requested, and that these measures address the original security concern.

Neither of these restrictions are major issues in most use cases as they would only affect performance, and only for those who actually use peers or have caching enabled, which can't be many people these days with SSL everywhere unless they have certificate based MITM interception and decryption enabled. (We don't)

Probably like many, we only use Squid for Squidguard to allow domain name based filtering and to provide some logging mostly for troubleshooting purposes.

Rapidly changing/cycling DNS entries with very short (often 30 seconds) TTL for CDN's seems to be the norm now unfortunately, so even with Squid using the same caching DNS server as the clients (which isn't even always feasible) this coherency test is failing at a significantly high percentage rate resulting in random 409's to client devices, frequently enough for visible problems to occur for users, usually in the form of intermittent page load failures and difficulty logging into and using services like Office 365, which seems to be a particularly bad offender now.

My workaround for devices that can't get explicit proxy settings using mechanisms like group policy has been to set up WPAD to inform client devices of an explicit proxy setting but not all devices support this either, and Apple devices which we have a lot of on our network have defaulted to Automatic proxy configuration disabled for the last several major iOS versions which means trying to convince owners of several hundred BYO devices to correctly enable automatic proxy settings.

A fix for this issue would be a massive quality of life improvement for anyone who uses transparent proxying with PFSense and solve a lot of headaches.

I'd also add that I'm running 2.7.0 now and the issue is the same - not surprising as I think it's running the same or nearly the same version of squid. So the ticket can be updated to list 2.7.0.

I have a transparent deployment with pfSense 2.7.0, and a mitigation has been to rely on pfBlockerNG and custom NAT rules for interception. In essence, to bypass interception for range of IPs that are extremely likely to implement DNS based load balancing for their web services. I am bypassing Squid for:
1- Azure Cloud and Azure Front Door, solutions who often relies on Azure Traffic Manager for DNS load balancing.
2- Akamai. This CDN also implements DNS based load balancing with low TTL.
3- AWS, solutions are somewhat likely to implement Route 53 for load balancing. But this isn't as frequent as #1 #2.
4- Custom solutions. Often larger corporations that owns multiple datacenters, and rely on other commercial solutions like F5 GLSB.

It is getting increasingly difficult to maintain, and a real solution would be greatly appreciated.

An alternative to disabling the check/marking un-cachable, could be to replace the validation routine. Instead of a DNS based validation, verify that he SNI from the Client Hello request matches the certificate obtained from the remote web server, and if the certificate is trusted AND valid then the request can be considered legitimate. For future proofing as a web security gateway, we could someday look at leveraging certificate transparency to ensure we are getting a certificate that has been logged (known to be in use).

At this point, I would be happy with whatever solution prevents Squid from dropping perfectly legitimate HTTPS traffic.

Denis Roy wrote in #note-9:

I have a transparent deployment with pfSense 2.7.0, and a mitigation has been to rely on pfBlockerNG and custom NAT rules for interception. In essence, to bypass interception for range of IPs that are extremely likely to implement DNS based load balancing for their web services.

Could you not just use "Bypass Proxy for These Destination IPs" under "Transparent Proxy Settings" in the Squid General settings page to achieve the same effect much easier ? Even though it doesn't autocomplete when you type in the box you can put an alias here and then add networks/IP's to the alias. I use an alias here for a few specific destinations I want to bypass the transparent proxy for albeit for different reasons than you. As far as I know, behind the scenes this setting just adds exclusions in the ipfw redirect rules, essentially what you are doing manually.

I am bypassing Squid for:
1- Azure Cloud and Azure Front Door, solutions who often relies on Azure Traffic Manager for DNS load balancing.
2- Akamai. This CDN also implements DNS based load balancing with low TTL.
3- AWS, solutions are somewhat likely to implement Route 53 for load balancing. But this isn't as frequent as #1 #2.
4- Custom solutions. Often larger corporations that owns multiple datacenters, and rely on other commercial solutions like F5 GLSB.

It is getting increasingly difficult to maintain, and a real solution would be greatly appreciated.

I'm surprised you're able to maintain this at all to be honest, that seems like a never ending job. My workaround is WPAD based proxy auto configuration but it still requires convincing end users (mostly school children....) to set their proxy settings correctly. Also a never ending job. :)

Simon Byrnand wrote in #note-10:

Could you not just use "Bypass Proxy for These Destination IPs" under "Transparent Proxy Settings" in the Squid General settings page to achieve the same effect much easier ? Even though it doesn't autocomplete when you type in the box you can put an alias here and then add networks/IP's to the alias. I use an alias here for a few specific destinations I want to bypass the transparent proxy for albeit for different reasons than you. As far as I know, behind the scenes this setting just adds exclusions in the ipfw redirect rules, essentially what you are doing manually.

You are correct: I am just doing this stuff manually, and "Bypass Proxy for These Destination IPs" work as well. I just find it easier to maintain if I can look at the NAT rules, get statistics on it from the WebUI, etc. I do it manually for the convenience. Takes a minute to setup, and it makes everything easier down the road.

I'm surprised you're able to maintain this at all to be honest, that seems like a never ending job. My workaround is WPAD based proxy auto configuration but it still requires convincing end users (mostly school children....) to set their proxy settings correctly. Also a never ending job. :)

It's actually not that hard. pfBlockerNG does it for me. I can share most of the lists that I currently have:

https://saasedl.paloaltonetworks.com/feeds/akamai/all/ipv4
https://ip-ranges.amazonaws.com/ip-ranges.json
https://saasedl.paloaltonetworks.com/feeds/azure/public/azurecloud/ipv4
https://saasedl.paloaltonetworks.com/feeds/azure/public/azurefrontdoor/ipv4
AS32934 [ FACEBOOK, US ] #Note pfBlockerNG automatically supports resolving ASNs too!
https://api.fastly.com/public-ip-list
https://saasedl.paloaltonetworks.com/feeds/googleworkspace/all/ipv4
https://saasedl.paloaltonetworks.com/feeds/m365/worldwide/skype/all/ipv4

I feed all my access.log into a SIEM so I can easily identify all the 409s that I get, get a count for each unique domain name, etc. I've built that list fairly quick TBH. And it is quite static at this point.

The real issue with my approach is that whitelisting stuff like AWS could eventually means opening up stuff I'd like to block. Lots of domains hosted on AWS these days. But what I can't block through Squidguard, is blocked through DNSBL. And that's enough for my needs.

Or….
We could have a proper fix for this issue then the workarounds that aren’t scalable

Can anyone advise on the feasibility of building a custom patched version of Squid (at least for testing purposes to further investigate possible solutions) from source to install in PFSense as a package ?

This problem affects us badly enough that I'd be willing to look into doing this, however I'm not familiar with the build environment and processes of PFSense packages to know what is involved in this and whether everything needed to build an identical package to the released ones is made available to those outside Netgate. (I'm also a Linux and Windows guy not a FreeBSD guy but I'm fairly comfortable building open source software from source on Linux, so it's not too far out of my wheelhouse with some pointers in the right direction)

Is there a mechanism in PFSense to install 3rd party custom built packages that are not listed in the official repos ? Could a package be built with a slightly later version number and installed from the command line using pkg, etc ? If anyone can point me in the right direction it would be appreciated.