Project

General

Profile

Bug #3932

Captive portal with greater than 9000 permanent MAC addresses causes timeout in loading CP

Added by Jeremy Porter over 2 years ago. Updated about 1 year ago.

Status:
Assigned
Priority:
Normal
Category:
Captive Portal
Target version:
Start date:
10/13/2014
Due date:
01/05/2015
% Done:

0%

Affected version:
All
Affected Architecture:
All

Description

When using captive portal, with all users become permanently learned after enough users have authenticated the system will run out of memory attempting to parse the config.xml.
Attempting to make changes to the config becomes impossible as the php process will crash.

Storing mac addresses in a sqlite database, will prevent the parsing from running out of memory.

Associated revisions

Revision 621fed0e
Added by Ermal Luçi over 2 years ago

Ticket #3932 For more than 100 entries create pipes in line with the rules file to speedup the process

Revision ab8d50ac
Added by Chris Buechler over 2 years ago

Shorten up the MAC pass-through descr. It was redundant, and for those with huge numbers of auto-added MAC passthrough entries, it adds up to a significant amount of config space (adding to delays when launching CP). helps Ticket #3932

Revision 64ed3e60
Added by Ermal Luçi about 2 years ago

Fix inherent issues with isset and empty values set as true by our parser. This made the piep configuration to be wrong at least for passthrough entries. Ticket #3932

Revision 384deecb
Added by Ermal Luçi about 2 years ago

Fix inherent issues with isset and empty values set as true by our parser. This made the piep configuration to be wrong at least for passthrough entries. Ticket #3932

Revision 7077addc
Added by Ermal Luçi about 2 years ago

Ticket #3932 Use array_map to get more parallelism when there are many entries. This makes it not reach the execution timeout with large entries.

Revision aa685f7a
Added by Ermal Luçi about 2 years ago

Ticket #3932 Use array_map to get more parallelism when there are many entries. This makes it not reach the execution timeout with large entries.

Revision 41196b69
Added by Ermal Luçi about 2 years ago

Split the work into different jobs called through fcgicli. Helps Ticket #3932

History

#1 Updated by Chris Buechler over 2 years ago

  • Target version changed from 2.2 to 2.3
  • Affected version changed from 2.1.x to All

How big are the resulting configs there? I'm not running PHP out of memory after throwing 9000 randomly-generated MACs in the passthrough list. Though the config is pretty bare outside the MAC list, it's a total of a bit under 500 KB config file. My <descr> fields are all blank, that might be enough to push it over the edge.

During boot, it sits there during boot at "Starting captive portal..." for roughly 15 minutes on an older i7 CPU VM.

Definitely more than we're going to get into for 2.2 given it's an extraordinarily rare edge case, will take some effort to fix, and we're too short on time.

I only tried on 2.2, haven't done any testing on previous releases.

#2 Updated by Jim Thompson over 2 years ago

  • Assignee set to Ermal Luçi
  • Priority changed from Low to Normal
  • Target version changed from 2.3 to 2.2

Ermal suggests that if /cf/conf/use_xmlreader exists, that a faster parser will be used in 2.2.

Please re-test. Ermal will also look at speeding up loading of passthrough MACs.

Target set back to 2.2

#3 Updated by Ermal Luçi over 2 years ago

The loading should be faster.

#4 Updated by Chris Buechler over 2 years ago

  • Status changed from New to Confirmed

I committed a change last night to shorten the <descr> text, which helps slightly, but still nothing works at 9000 MAC passthrough entries.

With an updated test case where the <descr> is filled out with the same text used on a real system with auto-add MAC passthrough, this fails pretty miserably. That takes the config up from about 500K to just over 1 MB. CP fails to start, after waiting for 15 minutes at "Starting zone" and failing with:

Maximum execution time of 900 seconds exceeded in /etc/inc/captiveportal.inc on line 1448

It fails the same way whether or not use_xmlreader is set.

Everything other than CP continues to work very well, and even management of CP works well. The huge table of MACs on services_captiveportal_mac.php fully loads in under 10 seconds.

Given time constraints on 2.2, and the high risk of introducing breakage when doing major performance improvements, we're not going to be able to make vast improvements here in 2.2. A work around for the one system where this is an issue is probably our best near-term bet. Adding a cron job to trim the list of the oldest entries and keep it to X length should work around.

I'll discuss options here with Ermal.

#5 Updated by Jim Thompson over 2 years ago

Could we try what Jeremy asked for?

#6 Updated by Chris Buechler about 2 years ago

  • Subject changed from Captive portal with greater than 9000 permanent MAC addresses causes out of memory when loading config.xml to Captive portal with greater than 9000 permanent MAC addresses causes timeout in loading CP
  • Target version changed from 2.2 to 2.2.1

2.2 doesn't run out of memory doing this, so the problem as it existed in earlier versions is gone (probably with the switch to php-fpm). It's just so slow in some part of the processing that it hits the 15 minute time limit on execution. It's not in the config parsing portion though, it'd be more or less equally slow with any back end. We'll review in more detail post-2.2, in the mean time we can get them a script to trim the list.

#7 Updated by Jim Thompson about 2 years ago

  • Due date set to 01/05/2015
  • Target version changed from 2.2.1 to 2.2

Target version set back to 2.2.

The issue is that it's slow.

I made an assignment yesterday. I'm restoring it now.

#8 Updated by Jim Thompson about 2 years ago

  • Target version changed from 2.2 to 2.2.1

#9 Updated by Ermal Luçi about 2 years ago

  • Status changed from Confirmed to Feedback

I pushed some fixes related to this that impacted even 2.2
Normally on a capable machine for this it should be ok.

The only problem might be the max execution timeout which can be controlled differently.

#10 Updated by Ermal Luçi about 2 years ago

  • Target version changed from 2.2.1 to 2.2

This should be resolved properly now.

#11 Updated by Jim Thompson about 2 years ago

  • Target version changed from 2.2 to 2.2.1

#12 Updated by Chris Buechler about 2 years ago

  • Target version changed from 2.2.1 to 2.2.2

#13 Updated by Chris Buechler almost 2 years ago

  • Target version changed from 2.2.2 to 2.2.3

#14 Updated by Chris Buechler almost 2 years ago

  • Target version changed from 2.2.3 to 2.3

#15 Updated by Jim Thompson over 1 year ago

  • Assignee changed from Ermal Luçi to Matthew Smith

reassigned to mgsmith.

#16 Updated by Jim Thompson over 1 year ago

bump

as much as I dislike cRaptive portals, this needs to be fixed.

#17 Updated by Jim Thompson over 1 year ago

  • Status changed from Feedback to Assigned

#18 Updated by Jim Thompson about 1 year ago

  • Assignee changed from Matthew Smith to Renato Botelho
  • Target version changed from 2.3 to 2.4.0

Also available in: Atom PDF