Project

General

Profile

Bug #4876

Cannot define table: Cannot allocate memory with large table aliases

Added by Kill Bill almost 4 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Operating System
Target version:
-
Start date:
07/24/2015
Due date:
% Done:

0%

Estimated time:
Affected Version:
All
Affected Architecture:
All

Description

Reference: https://forum.pfsense.org/index.php?topic=95989.0

New alert found: There were error(s) loading the rules: /tmp/rules.debug:45: cannot define table pfB_P2P: Cannot allocate memory - The line in question reads [45]: table <pfB_P2P> persist file "/var/db/aliastables/pfB_P2P.txt" 
# wc -l /var/db/aliastables/pfB_P2P.txt
  443746 /var/db/aliastables/pfB_P2P.txt
# pfctl -sa | grep -C4 LIMITS | tail -n 5
LIMITS:
states        hard limit   197000
src-nodes     hard limit   197000
frags         hard limit     5000
table-entries hard limit 10000000

As discussed on forum, the only (semi)reliable way to reproduce it is running

playback gitsync RELENG_2_2

and it triggers the above error on the Filter reload part about 80% of cases. If after that I go Status - Filter Reload and do Reload Filter, it reloads without any problems.

Attached relevant part of pfBlockerNG configuration plus a screenshot of the FW (NAT) rule triggering this, hopefully will make reproducing the issue a bit easier. (The BittorrentPorts is just a simple two ports alias - 6881, 51413).

History

#1 Updated by Kill Bill almost 4 years ago

Perhaps also this (copied from pfBNG update log) - really cannot see how I'm hitting the 10M limit here.

Alias Table IP Counts
-----------------------------
  485649 total
  443746 /var/db/aliastables/pfB_P2P.txt
   29253 /var/db/aliastables/pfB_CUST.txt
    4848 /var/db/aliastables/pfB_IBlock.txt
    2728 /var/db/aliastables/pfB_PRI3.txt
    2101 /var/db/aliastables/pfB_Europe_v4.txt
    1498 /var/db/aliastables/pfB_PRI1.txt
     514 /var/db/aliastables/pfB_Europe_v6.txt
     426 /var/db/aliastables/pfB_SEC1.txt
     208 /var/db/aliastables/pfB_SEC2.txt
     160 /var/db/aliastables/pfB_PRI2.txt
     140 /var/db/aliastables/pfB_PS_v4.txt
      27 /var/db/aliastables/pfB_DNSBLIP.txt

pfSense Table Stats
-------------------
table-entries hard limit 10000000
Table Usage Count        485681

#2 Updated by Kill Bill almost 4 years ago

Well I think I have found something (basically, kernel limits issue), but the hints there are not useful since kern.maxdsiz is 34359738368 B here (~33GiB).

https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html

#3 Updated by Chris Buechler over 3 years ago

this seems to be even easier to replicate on 2.3. Take the attached config, restore it, and on boot it'll throw out:

"pf_busy";s:6:"notice";s:38:"PF was wedged/busy and has been reset.";s:3:"url";s:0:"";s:8:"category";s:7:"pf_busy";s:8:"priority";i:1;}i:1447221238;a:5:{s:2:"id";s:11:"filter_load";s:6:"notice";s:105:"There were error(s) loading the rules: pfctl: DIOCADDALTQ: Device busy - The line in question reads [0]: " 

post-boot, most of the time upon running /etc/rc.filter_configure_sync you'll end up with:

"There were error(s) loading the rules: /tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory - The line in question reads [42]: table <pfB_P2P> persist file "/var/db/aliastables/pfB_P2P.txt" 

adding the following lines for debug in filter.inc:

        @file_put_contents("{$g['tmp_path']}/rules.limits", $limitrules);
$debugtime = time();
mwexec("cp /tmp/rules.limits /root/$debugtime-rules.limits");
        mwexec("/sbin/pfctl -Of {$g['tmp_path']}/rules.limits");
        unset($rules, $limitrules);
mwexec("pfctl -sm > /root/$debugtime-limits.txt");

shows that the rules.limits it's applying are correct, and 'pfctl -sm' shows the correct limits, before it attempts to load the rules. The limits are well above the size of the table. Later trying to just load the rules may work, or may result in the same.

# pfctl -f /tmp/rules.debug
/tmp/rules.debug:42: cannot define table pfB_P2P: Cannot allocate memory
pfctl: Syntax error in config file: pf rules not loaded
# pfctl -f /tmp/rules.debug
# 

worked fine the second time. Sometimes the second attempt fails as well and a third might succeed. I can't seem to find a sure fire differentiation between when it works and when it doesn't, but it fails frequently with the attached config.

#4 Updated by Kill Bill over 3 years ago

Chris Buechler wrote:

Later trying to just load the rules may work, or may result in the same.
worked fine the second time. Sometimes the second attempt fails as well and a third might succeed. I can't seem to find a sure fire differentiation between when it works and when it doesn't, but it fails frequently with the attached config.

Very much matches my observation. No clear pattern and sometimes requires more attempts to finally get the table loaded. Should this be filed in FreeBSD bugzilla perhaps? NFC what's this, but it's annoying for sure.

#5 Updated by Luiz Souza over 3 years ago

I'm trying to reproduce this, but no success so far (trying with a RCC-VE 4860).

#6 Updated by Chris Buechler over 3 years ago

this is still easily replicable with the config I attached. Sent Luiz a system that's running it and still sees the issue.

#7 Updated by Chris Buechler over 3 years ago

  • Status changed from Confirmed to Closed
  • Target version deleted (2.3)
  • Affected Version changed from 2.2.x to All

Luiz tracked down the root cause to actual memory allocation failures. "it uses memory from the uma allocator, the uma tries to be smart and start to deny some bigger allocations under low memory conditions but before it actually exhaust the memory"

It's easily replicable with a big enough config and 256 or 384 MB RAM. But with 512 MB or more, it's not an issue.

In short, you can fail to allocate memory in this situation even if there is some memory or swap available, so huge tables aren't doable with small amounts of RAM.

Also available in: Atom PDF