Project

General

Profile

Actions

Bug #13926

closed

pfBlockerNG-devel 3.2.0 - Slow MaxMind Database Downloads under PHP 8.1

Added by Steve Wilson about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
pfBlockerNG
Target version:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
Affected Version:
Affected Plus Version:
23.01
Affected Architecture:

Description

Prior to the update to PHP 8.1, downloads of the MaxMind database would take approximately 4 seconds. After the update to PHP 8.1, downloads now take over 13 minutes:

Download Process Starting [ 02/3/23 01:00:00 ]

Download Process Starting [ 02/3/23 01:00:00 ]
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
Download Process Ended [ 02/3/23 01:13:24 ]

Download Process Ended [ 02/3/23 01:13:24 ]

Similar issues with MaxMind download times have been noted with the Suricata package. See https://redmine.pfsense.org/issues/13839#change-65485. Modifying the default cURL options appearing at lines 165 - 175 in the pfblockerng.inc file with changes similar to those made to the Suricata package restores download performance to pre-PHP 8.1 levels:

Download Process Starting [ 02/3/23 02:00:00 ]

Download Process Starting [ 02/3/23 02:00:00 ]
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
Download Process Ended [ 02/3/23 02:00:04 ]

Download Process Ended [ 02/3/23 02:00:04 ]

It may be worthwhile to consider importing the changes made in the Suricata Redmine referenced above into the pfBlocker package to avoid these download delays.


Related issues

Related to Bug #13874: pfBlocker -devel hanging on cron jobsResolved

Actions
Actions #1

Updated by Jim Pingle about 1 year ago

If you can easily reproduce this, try the following patch (path strip=1):

diff --git a/usr/local/pkg/pfblockerng/pfblockerng.inc b/usr/local/pkg/pfblockerng/pfblockerng.inc
index 1f5aceef7ee6..b3e9b11989ff 100644
--- a/usr/local/pkg/pfblockerng/pfblockerng.inc
+++ b/usr/local/pkg/pfblockerng/pfblockerng.inc
@@ -171,7 +171,13 @@ $pfb['curl_defaults'] = array(  CURLOPT_USERAGENT  => 'pfSense/pfBlockerNG cURL d
                                CURLOPT_FRESH_CONNECT   => true,
                                CURLOPT_FILETIME        => true,
                                CURLOPT_TCP_NODELAY     => true,
-                               CURLOPT_CONNECTTIMEOUT  => 15
+                               CURLOPT_CONNECTTIMEOUT  => 15,
+                               CURLOPT_AUTOREFERER     => true,
+                               CURLOPT_MAXREDIRS       => 10,
+                               CURLOPT_HTTP_VERSION    => CURL_HTTP_VERSION_NONE,
+                               CURLOPT_FORBID_REUSE    => true,
+                               CURLOPT_SSL_ENABLE_ALPN => true,
+                               CURLOPT_SSL_ENABLE_NPN  => true,
                                );

 // RFC7231 HTTP response codes
Actions #2

Updated by Steve Wilson about 1 year ago

Jim,

With your patch applied the download completes in about 5 seconds, so it solves the issue. But note that there are additional cURL options specified in /usr/local/www/pfblockerng/pfblockerng.php (starting at line 361). I haven't had time to parse the code there to see if the changes in the patch will conflict with any of the additional cURL options specified there, but with respect to the MaxMind download specifically it works fine.

Actions #3

Updated by Jim Pingle about 1 year ago

Steve Wilson wrote in #note-2:

Jim,

With your patch applied the download completes in about 5 seconds, so it solves the issue. But note that there are additional cURL options specified in /usr/local/www/pfblockerng/pfblockerng.php (starting at line 361). I haven't had time to parse the code there to see if the changes in the patch will conflict with any of the additional cURL options specified there, but with respect to the MaxMind download specifically it works fine.

The way that pfblockerng.php sets its options, it applies the defaults first and then sets its own. I don't see anything in there that would conflict, and even so, it would overwrite the "default" option with its own the way it's coded there. So if it works I'll go ahead and commit it for now since it's affecting quite a few people. Can always back it out if need be.

Actions #4

Updated by Jim Pingle about 1 year ago

  • Assignee set to Jim Pingle
Actions #5

Updated by Jim Pingle about 1 year ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

I merged the changes to the pfBlockerNG cURL defaults, so the next build will include them.

Actions #6

Updated by Glenn Hall about 1 year ago

This change fixed two issues I have seen with pfB since moving to 23.01: 1) slow MaxMind downloads; 2) slow block list downloads. Thanks!

Actions #7

Updated by Jim Pingle about 1 year ago

  • Status changed from Feedback to Resolved
Actions #8

Updated by Jim Pingle about 1 year ago

  • Related to Bug #13874: pfBlocker -devel hanging on cron jobs added
Actions

Also available in: Atom PDF