Bug #3083
closed
Firewall rule toggle fails after several enable/disable cycles : 2.1 RC0 and 2.0.3 release
0%
Description
Please see the description in my forum post for more details:
http://forum.pfsense.org/index.php/topic,64098.0.html
Synopsis:
A block rule fails completely if toggled between enabled and disabled using the icon to the left of the rule.
The error condition can occur after just 1 cycle, but may take 6 full cycles of enabled/disabled/reset-states:
- add new ICMP block rule (enabled) LAN->WAN, host->host
- save and apply
- start an ICMP echo request (constant ping to remote host on WAN if)
- disable rule
- wait for replies to commence
- enable rule
- reset states
- return to firewall rules LAN page
(possible multiple repeat required here...)
At this point the ICMP echo replies continue, despite the block rule showing as enabled. The firewall is essentially exhibiting the opposite of the expected, and indicated, behaviour.
This issue became apparent by chance during initial evaluation of pfSense 2.0.3 RELEASE (i386) 2 weeks ago (as a possible addition to our data-centre hosting setup).
Several configurations of hardware, routing and rulesets were tried, but for thorough repeated testing I reverted to a completely "factory default" setup behind an ADSL broadband router with DHCP on WAN/em1, and just 1 user-added firewall rule on the LAN interface.
I emphasise that this has nothing to do with resetting states. The only remedial action available via the web GUI is to either add another rule or re-save the existing rule. I have also tried waiting up to 30 minutes after entering the error state and then accessing various web GUI pages, then resetting states, then stopping the ping, waiting 30min etc.; the error condition remains.
Actions via the console shell that trigger an OS file stat (e.g. an "ls" on the /tmp/rules.debug file) result in the block rule being reinstated to the live ruleset, and after a state reset the block rule becomes effective again (a volatile buffer being committed?).
On a system that is installed to HDD a reboot also flushes the block rule such that it is correctly reinstated.
I have only tested on 2.0.3 and the July 4th snapshot of 2.1 RC0. However, testing within a carefully controlled set of conditions (as described) has been as rigorous and exhaustive as I could make time for. I am satisfied that the effect I am seeing is not due to hardware or user error.
Updated by Chris Buechler over 11 years ago
- Status changed from New to Rejected
not true. Has to be states. Maybe that in combination with something delaying your filter reload enough that a new state can get added, almost certainly an installed package.
If you can show it's not in the running ruleset "pfctl -vvsr", which it most certainly is, then that would be a problem.
Updated by Mark Tiramani over 11 years ago
Chris Buechler wrote:
not true. Has to be states. Maybe that in combination with something delaying your filter reload enough that > a new state can get added, almost certainly an installed package.
As stated in my posts and bug report: This is a "factory default" setup plus 1 LAN->WAN rule.
There is nothing added, no additional packages. All the tests described were run on i386 ISO live CD archives downloaded from the 2 UK mirrors, and the 2.1 RC0 ISO from:
http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/i386/pfSense_RELENG_2_1/livecd_installer/pfSense-LiveCD-2.1-RC0-i386-20130708-2125.iso.gz
If you can show it's not in the running ruleset "pfctl -vvsr", which it most certainly is, then that would be > a problem.
As stated in my forum post, the rule is not appearing in "pfctl -sa" until after an OS stat, reboot, or rule "save".
I just provoked the error again and ran "pfctl -vvsr" and the rule is not in the running set.
Updated by Jim Pingle over 11 years ago
If you trigger too many filter reloads too fast it's possible that check_reload_status is eating the reload events because they're coming too frequently and it is throttling the reload. This happened with some other events previously.
Rather than doing a rule save, reboot, etc, wait a few moments and go to Status > Filter Reload, and press the button there to see if that also brings it back.
Updated by Mark Tiramani over 11 years ago
Jim P wrote:
If you trigger too many filter reloads too fast it's possible that check_reload_status is eating the reload
events because they're coming too frequently and it is throttling the reload. This happened with some other
events previously.
Please be more precise. What do you mean exactly by "too many filter reloads too fast"?
How many?
How fast?
e.g. What would you consider a reasonable interval between a single user toggling a single ruleset, clicking on "Apply changes" then "Close" and then clicking on the "Diagnostics->States->Reset States->Reset" button? (It just took me 13 seconds with a mousepad and clumsy fingers. I guess I could reduce it to 9 seconds with sufficient practice).
Please note: The error occurs in both 2.0.3 RELEASE as well as 21. RC0
Rather than doing a rule save, reboot, etc, wait a few moments and go to Status > Filter Reload, and press the button there to see if that also brings it back.
I will test this and post an update. However, if "Status->Filter Reload" does restore the ruleset displayed to the user then the behaviour is still highly unexpected for users of all experience levels.
Updated by Mark Tiramani over 11 years ago
Jim P wrote:
Rather than doing a rule save, reboot, etc, wait a few moments and go to Status > Filter Reload, and press the
button there to see if that also brings it back.
I produced the error state once more, waiting at least 15 seconds between rule toggles and the state reset.
With the block rule showing as enabled packets continued to pass.
I then did "Status->Filter Reload" and waited for the status field with the dashed border to refresh stating "Done. The filter rules have been reloaded."
I then did "Diagnostics->States->Reset States->Reset". The error remains. Packets continue to pass the block.
I repeated the above 4 times sloooowly, with the same result each time.
I then did "Status->Filter Reload" but this time I clicked on the "Reload Filter" button manually.
Bingo. The rules as they appear in the GUI are then reloaded.
This confirms that there is a bug.
Please reinstate Bug #3083 (or open another one to suit) and take a more careful look at the processes I have described. The error condition can probably be reproduced in less than 5 minutes by anyone using 2.0.3 or 2.1 RC0.
Updated by Doktor Notor over 11 years ago
I then did "Status->Filter Reload" but this time I clicked on the "Reload Filter" button manually. Bingo. The rules as they appear in the GUI are then reloaded.
Uhm... what do you mean by "manually"? Without clicking the button, the filter does not reload at all, of course.
Updated by Mark Tiramani over 11 years ago
Doktor Notor wrote:
Uhm... what do you mean by "manually"? Without clicking the button, the filter does not reload at all, of
course.
Where is that stated? Watch the page and you will clearly see it automatically refreshes after ~3 seconds and the text "Done. The filter rules have been reloaded." appears, without clicking the button.
If that does not mean what it says then it should obviously not say it.
Also, where is it stated that to reload the filter set after a rule change the user must navigate to this page and must press the button?
The button is clearly intended for a manual/forced reload. But the logic and wording of the interface tells the user that the reload has been applied even without further intervention. Indeed, the button on the firewall rules page says "Apply changes" and subsequently that those changes are being applied in the background.
The rules are clearly not reloaded as the devs or the user expects. Bug.
Updated by Doktor Notor over 11 years ago
Also, where is it stated that to reload the filter set after a rule change the user must navigate to this page and must press the button?
Not what I'm talking about at all. I'm merely saying that browsing to Status > Filter Reload without pressing the button does nothing.
Updated by Jim Pingle over 11 years ago
That page shows the status of the previous filter reload. If it was successful, it will say so.
Pressing the button causes a manual reload to happen, and the status will update accordingly.
Updated by Mark Tiramani over 11 years ago
Jim P wrote:
That page shows the status of the previous filter reload. If it was successful, it will say so.
Pressing the button causes a manual reload to happen, and the status will update accordingly.
OK, but there is no mention of "_previous_" on the "Status->Filter Reload" page. It is completely unclear to the new user. The language used on the page is seemingly very clear but is very likely to lead to the wrong conclusion. After clicking the "Apply changes" button on the "Firewall->Rules" page the text next to the "Ok" button includes a link to "monitor" the filter reload progress. It says "monitor", not "to activate".
And this still does not address the actual problem: The filter reload is not happening as intended by the developers, or by the user. This much is clear is it not?
What is not clear is why we are talking around the edges of an obvious bug ;)
Why is the browser code that triggers the current filter reload as applied on the "Firewall->Rules" interface "Apply changes" not always functioning?
Is this possibly within the scope of "check_reload_status ... eating the reload events"?
Updated by Jim Pingle over 11 years ago
- Status changed from Rejected to New
- Priority changed from Urgent to Normal
I was finally able to reproduce this, but it wasn't quite so easy. I was able to do it for several minutes straight until I sped up the pace of the procedure. If I gave it plenty of time between reloads it did eventually stop but it seemed to take much longer. The faster you go, the quicker it seems to appear.
However if I went to Status > Filter Reload and hit the button, the rules did update.
I'd still guess it's check_reload_status trying to rate limit the updates.
Updated by Ermal Luçi over 11 years ago
Yes check reload does limit this events.
You have other issues if you do not rate limit it that this rate limit was supposed to fix!
Updated by Jim Pingle over 11 years ago
- Status changed from New to Feedback
After Ermal's check_reload_status commits this morning I can no longer reproduce this after copying over a binary from the builder including the changes. The next available snapshot will include the new binary and should work as expected.
Rather than ignoring the suppressed "too fast" updates, it schedules a later update to handle the subsequent reload so the last change isn't lost. It can take 10+ seconds for the rule to show up due to the rate limiting, but it does always come back now.
Updated by Chris Buechler over 11 years ago
- Status changed from Feedback to Resolved