Bug #4876
closed
Cannot define table: Cannot allocate memory with large table aliases
0%
Description
Reference: https://forum.pfsense.org/index.php?topic=95989.0
New alert found: There were error(s) loading the rules: /tmp/rules.debug:45: cannot define table pfB_P2P: Cannot allocate memory - The line in question reads [45]: table <pfB_P2P> persist file "/var/db/aliastables/pfB_P2P.txt"
# wc -l /var/db/aliastables/pfB_P2P.txt 443746 /var/db/aliastables/pfB_P2P.txt
# pfctl -sa | grep -C4 LIMITS | tail -n 5 LIMITS: states hard limit 197000 src-nodes hard limit 197000 frags hard limit 5000 table-entries hard limit 10000000
As discussed on forum, the only (semi)reliable way to reproduce it is running
playback gitsync RELENG_2_2
and it triggers the above error on the Filter reload part about 80% of cases. If after that I go Status - Filter Reload and do Reload Filter, it reloads without any problems.
Attached relevant part of pfBlockerNG configuration plus a screenshot of the FW (NAT) rule triggering this, hopefully will make reproducing the issue a bit easier. (The BittorrentPorts is just a simple two ports alias - 6881, 51413).
Files
Updated by Kill Bill almost 9 years ago
Perhaps also this (copied from pfBNG update log) - really cannot see how I'm hitting the 10M limit here.
Alias Table IP Counts
-----------------------------
485649 total
443746 /var/db/aliastables/pfB_P2P.txt
29253 /var/db/aliastables/pfB_CUST.txt
4848 /var/db/aliastables/pfB_IBlock.txt
2728 /var/db/aliastables/pfB_PRI3.txt
2101 /var/db/aliastables/pfB_Europe_v4.txt
1498 /var/db/aliastables/pfB_PRI1.txt
514 /var/db/aliastables/pfB_Europe_v6.txt
426 /var/db/aliastables/pfB_SEC1.txt
208 /var/db/aliastables/pfB_SEC2.txt
160 /var/db/aliastables/pfB_PRI2.txt
140 /var/db/aliastables/pfB_PS_v4.txt
27 /var/db/aliastables/pfB_DNSBLIP.txt
pfSense Table Stats
-------------------
table-entries hard limit 10000000
Table Usage Count 485681
Updated by Kill Bill over 8 years ago
Well I think I have found something (basically, kernel limits issue), but the hints there are not useful since kern.maxdsiz is 34359738368 B here (~33GiB).
https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html
Updated by Chris Buechler over 8 years ago
- File config-bootstrap-test1.pfmechanics.com-20151110235247.xml config-bootstrap-test1.pfmechanics.com-20151110235247.xml added
- Status changed from New to Confirmed
- Assignee set to Luiz Souza
- Target version set to 2.3
- Affected Architecture All added
- Affected Architecture deleted (
amd64)
this seems to be even easier to replicate on 2.3. Take the attached config, restore it, and on boot it'll throw out:
"pf_busy";s:6:"notice";s:38:"PF was wedged/busy and has been reset.";s:3:"url";s:0:"";s:8:"category";s:7:"pf_busy";s:8:"priority";i:1;}i:1447221238;a:5:{s:2:"id";s:11:"filter_load";s:6:"notice";s:105:"There were error(s) loading the rules: pfctl: DIOCADDALTQ: Device busy - The line in question reads [0]: "
post-boot, most of the time upon running /etc/rc.filter_configure_sync you'll end up with:
"There were error(s) loading the rules: /tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory - The line in question reads [42]: table <pfB_P2P> persist file "/var/db/aliastables/pfB_P2P.txt"
adding the following lines for debug in filter.inc:
@file_put_contents("{$g['tmp_path']}/rules.limits", $limitrules);
$debugtime = time();
mwexec("cp /tmp/rules.limits /root/$debugtime-rules.limits");
mwexec("/sbin/pfctl -Of {$g['tmp_path']}/rules.limits");
unset($rules, $limitrules);
mwexec("pfctl -sm > /root/$debugtime-limits.txt");
shows that the rules.limits it's applying are correct, and 'pfctl -sm' shows the correct limits, before it attempts to load the rules. The limits are well above the size of the table. Later trying to just load the rules may work, or may result in the same.
# pfctl -f /tmp/rules.debug /tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded # pfctl -f /tmp/rules.debug #
worked fine the second time. Sometimes the second attempt fails as well and a third might succeed. I can't seem to find a sure fire differentiation between when it works and when it doesn't, but it fails frequently with the attached config.
Updated by Kill Bill over 8 years ago
Chris Buechler wrote:
Later trying to just load the rules may work, or may result in the same.
worked fine the second time. Sometimes the second attempt fails as well and a third might succeed. I can't seem to find a sure fire differentiation between when it works and when it doesn't, but it fails frequently with the attached config.
Very much matches my observation. No clear pattern and sometimes requires more attempts to finally get the table loaded. Should this be filed in FreeBSD bugzilla perhaps? NFC what's this, but it's annoying for sure.
Updated by Luiz Souza over 8 years ago
I'm trying to reproduce this, but no success so far (trying with a RCC-VE 4860).
Updated by Chris Buechler about 8 years ago
this is still easily replicable with the config I attached. Sent Luiz a system that's running it and still sees the issue.
Updated by Chris Buechler about 8 years ago
- Status changed from Confirmed to Closed
- Target version deleted (
2.3) - Affected Version changed from 2.2.x to All
Luiz tracked down the root cause to actual memory allocation failures. "it uses memory from the uma allocator, the uma tries to be smart and start to deny some bigger allocations under low memory conditions but before it actually exhaust the memory"
It's easily replicable with a big enough config and 256 or 384 MB RAM. But with 512 MB or more, it's not an issue.
In short, you can fail to allocate memory in this situation even if there is some memory or swap available, so huge tables aren't doable with small amounts of RAM.