Project

General

Profile

Actions

Regression #15320

open

XMLRPC Sync Trigger on CARP Maintenance Mode Causes webConfigurator Hangs on VIPs

Added by Kris Phillips 10 months ago. Updated 9 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
High Availability
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Release Notes:
Default
Affected Plus Version:
23.09.1
Affected Architecture:

Description

When syncing a large configuration file with a large number of Virtual IPs, XMLRPC Sync can cause the webConfigurator to completely hang on a secondary unit in an HA pair for several minutes. This can also lead to 504 Timeout messages. The webConfigurator will typically recover on it's own, but this will often take several minutes.

Disabling the Virtual IP portion of the XMLRPC sync resolves this issue and the failover is nearly instantaneous, along with complete responsiveness from the webConfigurator.

Likely improvements can be made to the PHP code to not just blindly copy and rebuild the entire Virtual IP configuration on the secondary unit, as these hangs can lead to high CPU load and responsiveness issues for the secondary firewall that you just failed over to. This is obviously less than ideal since that unit is supposed to be taking over traffic in a manual failover scenario.

Actions #1

Updated by Kris Phillips 10 months ago

Customer ticket 2453691225 is an example of this issue.

Actions #2

Updated by Kris Phillips 9 months ago

Customer ticket 2575064585 is another example of this issue. They did not have webConfigurator hangs, but they did have high CPU interrupts that caused gateway monitoring to fall over, which caused all three of their gateways to experience 10-40% packet loss.

Actions #3

Updated by Marcos M 9 months ago

It'd be good to test this on 24.03 as there have been general efficiency improvements that may help here.

Actions

Also available in: Atom PDF