Bug #13926
closedpfBlockerNG-devel 3.2.0 - Slow MaxMind Database Downloads under PHP 8.1
100%
Description
Prior to the update to PHP 8.1, downloads of the MaxMind database would take approximately 4 seconds. After the update to PHP 8.1, downloads now take over 13 minutes:
Download Process Starting [ 02/3/23 01:00:00 ]
Download Process Starting [ 02/3/23 01:00:00 ]
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
Download Process Ended [ 02/3/23 01:13:24 ]
Download Process Ended [ 02/3/23 01:13:24 ]
Similar issues with MaxMind download times have been noted with the Suricata package. See https://redmine.pfsense.org/issues/13839#change-65485. Modifying the default cURL options appearing at lines 165 - 175 in the pfblockerng.inc file with changes similar to those made to the Suricata package restores download performance to pre-PHP 8.1 levels:
Download Process Starting [ 02/3/23 02:00:00 ]
Download Process Starting [ 02/3/23 02:00:00 ]
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
Download Process Ended [ 02/3/23 02:00:04 ]
Download Process Ended [ 02/3/23 02:00:04 ]
It may be worthwhile to consider importing the changes made in the Suricata Redmine referenced above into the pfBlocker package to avoid these download delays.
Related issues
Updated by Jim Pingle over 1 year ago
If you can easily reproduce this, try the following patch (path strip=1):
diff --git a/usr/local/pkg/pfblockerng/pfblockerng.inc b/usr/local/pkg/pfblockerng/pfblockerng.inc
index 1f5aceef7ee6..b3e9b11989ff 100644
--- a/usr/local/pkg/pfblockerng/pfblockerng.inc
+++ b/usr/local/pkg/pfblockerng/pfblockerng.inc
@@ -171,7 +171,13 @@ $pfb['curl_defaults'] = array( CURLOPT_USERAGENT => 'pfSense/pfBlockerNG cURL d
CURLOPT_FRESH_CONNECT => true,
CURLOPT_FILETIME => true,
CURLOPT_TCP_NODELAY => true,
- CURLOPT_CONNECTTIMEOUT => 15
+ CURLOPT_CONNECTTIMEOUT => 15,
+ CURLOPT_AUTOREFERER => true,
+ CURLOPT_MAXREDIRS => 10,
+ CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_NONE,
+ CURLOPT_FORBID_REUSE => true,
+ CURLOPT_SSL_ENABLE_ALPN => true,
+ CURLOPT_SSL_ENABLE_NPN => true,
);
// RFC7231 HTTP response codes
Updated by Steve Wilson over 1 year ago
Jim,
With your patch applied the download completes in about 5 seconds, so it solves the issue. But note that there are additional cURL options specified in /usr/local/www/pfblockerng/pfblockerng.php (starting at line 361). I haven't had time to parse the code there to see if the changes in the patch will conflict with any of the additional cURL options specified there, but with respect to the MaxMind download specifically it works fine.
Updated by Jim Pingle over 1 year ago
Steve Wilson wrote in #note-2:
Jim,
With your patch applied the download completes in about 5 seconds, so it solves the issue. But note that there are additional cURL options specified in /usr/local/www/pfblockerng/pfblockerng.php (starting at line 361). I haven't had time to parse the code there to see if the changes in the patch will conflict with any of the additional cURL options specified there, but with respect to the MaxMind download specifically it works fine.
The way that pfblockerng.php sets its options, it applies the defaults first and then sets its own. I don't see anything in there that would conflict, and even so, it would overwrite the "default" option with its own the way it's coded there. So if it works I'll go ahead and commit it for now since it's affecting quite a few people. Can always back it out if need be.
Updated by Jim Pingle over 1 year ago
- Status changed from New to Feedback
- % Done changed from 0 to 100
I merged the changes to the pfBlockerNG cURL defaults, so the next build will include them.
Updated by Glenn Hall over 1 year ago
This change fixed two issues I have seen with pfB since moving to 23.01: 1) slow MaxMind downloads; 2) slow block list downloads. Thanks!
Updated by Jim Pingle over 1 year ago
- Related to Bug #13874: pfBlocker -devel hanging on cron jobs added