Bug #3796
openStates summary fails and is very slow with large state tables
0%
Description
One of the scenarios where the states summary would be most useful is when you have a large number of states, in helping figure out what's happening. Out of the box, it'll fail to complete upwards of somewhere around 100,000 states, running out of memory. It will succeed after increasing the memory limit, but is very slow. At 200,000 states, it needs a memory_limit of 256M, and takes near 10 minutes to run on a pretty fast CPU:
Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz (3200.00-MHz K8-class CPU)
Increasing the memory limit on this page would fix some circumstances, but it would take prohibitively long to run, hanging up the entire web interface in the process. You can't lock up your web interface for several minutes at a minimum when in the midst of troubleshooting an issue.
Suggested fix:
at a minimum, the current code could be run in the background and generate static output, so the Diag>States Summary menu, where you have > say 50,000 states, would only show static content from a previous run, and give the option to update the stats.
The reporting could likely be handled more efficiently outside of PHP entirely, though that might be much more work.