pfSense

this seems to be even easier to replicate on 2.3. Take the attached config, restore it, and on boot it'll throw out:

"pf_busy";s:6:"notice";s:38:"PF was wedged/busy and has been reset.";s:3:"url";s:0:"";s:8:"category";s:7:"pf_busy";s:8:"priority";i:1;}i:1447221238;a:5:{s:2:"id";s:11:"filter_load";s:6:"notice";s:105:"There were error(s) loading the rules: pfctl: DIOCADDALTQ: Device busy - The line in question reads [0]: " 

post-boot, most of the time upon running /etc/rc.filter_configure_sync you'll end up with:

"There were error(s) loading the rules: /tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory - The line in question reads [42]: table <pfB_P2P> persist file "/var/db/aliastables/pfB_P2P.txt" 

adding the following lines for debug in filter.inc:

        @file_put_contents("{$g['tmp_path']}/rules.limits", $limitrules);
$debugtime = time();
mwexec("cp /tmp/rules.limits /root/$debugtime-rules.limits");
        mwexec("/sbin/pfctl -Of {$g['tmp_path']}/rules.limits");
        unset($rules, $limitrules);
mwexec("pfctl -sm > /root/$debugtime-limits.txt");

shows that the rules.limits it's applying are correct, and 'pfctl -sm' shows the correct limits, before it attempts to load the rules. The limits are well above the size of the table. Later trying to just load the rules may work, or may result in the same.

# pfctl -f /tmp/rules.debug
/tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory
pfctl: Syntax error in config file: pf rules not loaded
# pfctl -f /tmp/rules.debug
# 

worked fine the second time. Sometimes the second attempt fails as well and a third might succeed. I can't seem to find a sure fire differentiation between when it works and when it doesn't, but it fails frequently with the attached config.

Luiz tracked down the root cause to actual memory allocation failures. "it uses memory from the uma allocator, the uma tries to be smart and start to deny some bigger allocations under low memory conditions but before it actually exhaust the memory"

It's easily replicable with a big enough config and 256 or 384 MB RAM. But with 512 MB or more, it's not an issue.

In short, you can fail to allocate memory in this situation even if there is some memory or swap available, so huge tables aren't doable with small amounts of RAM.