pfSense

How big are the resulting configs there? I'm not running PHP out of memory after throwing 9000 randomly-generated MACs in the passthrough list. Though the config is pretty bare outside the MAC list, it's a total of a bit under 500 KB config file. My <descr> fields are all blank, that might be enough to push it over the edge.

During boot, it sits there during boot at "Starting captive portal..." for roughly 15 minutes on an older i7 CPU VM.

Definitely more than we're going to get into for 2.2 given it's an extraordinarily rare edge case, will take some effort to fix, and we're too short on time.

I only tried on 2.2, haven't done any testing on previous releases.

Ermal suggests that if /cf/conf/use_xmlreader exists, that a faster parser will be used in 2.2.

Please re-test. Ermal will also look at speeding up loading of passthrough MACs.

Target set back to 2.2

I committed a change last night to shorten the <descr> text, which helps slightly, but still nothing works at 9000 MAC passthrough entries.

With an updated test case where the <descr> is filled out with the same text used on a real system with auto-add MAC passthrough, this fails pretty miserably. That takes the config up from about 500K to just over 1 MB. CP fails to start, after waiting for 15 minutes at "Starting zone" and failing with:

Maximum execution time of 900 seconds exceeded in /etc/inc/captiveportal.inc on line 1448

It fails the same way whether or not use_xmlreader is set.

Everything other than CP continues to work very well, and even management of CP works well. The huge table of MACs on services_captiveportal_mac.php fully loads in under 10 seconds.

Given time constraints on 2.2, and the high risk of introducing breakage when doing major performance improvements, we're not going to be able to make vast improvements here in 2.2. A work around for the one system where this is an issue is probably our best near-term bet. Adding a cron job to trim the list of the oldest entries and keep it to X length should work around.

I'll discuss options here with Ermal.

2.2 doesn't run out of memory doing this, so the problem as it existed in earlier versions is gone (probably with the switch to php-fpm). It's just so slow in some part of the processing that it hits the 15 minute time limit on execution. It's not in the config parsing portion though, it'd be more or less equally slow with any back end. We'll review in more detail post-2.2, in the mean time we can get them a script to trim the list.

Target version set back to 2.2.

The issue is that it's slow.

I made an assignment yesterday. I'm restoring it now.

I pushed some fixes related to this that impacted even 2.2
Normally on a capable machine for this it should be ok.

The only problem might be the max execution timeout which can be controlled differently.

This should be resolved properly now.

reassigned to mgsmith.

It really needs to be re-engineered

It is unreasonable to keep kicking this down the road to target_version++ Closing and recording it for future consideration.