Project

General

Profile

Actions

Bug #6478

closed

Master not sending XML RPC sync data to Backup node

Added by Michael Schmid almost 8 years ago. Updated almost 8 years ago.

Status:
Not a Bug
Priority:
Normal
Assignee:
-
Category:
XMLRPC
Target version:
-
Start date:
06/10/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
Affected Architecture:

Description

I noticed that my backup node is out of XML sync although it's activated on master.
State sync is working fine.

The sync is configured to work on an dedicated PfSync VLAN where the state sync as well as the XML sync should go.

I doublechecked the following:
  • both nodes have the same protocol (HTTPS) and port (443)
  • traffic on PfSync interface is allowed on both nodes
  • no log entries for related firewall events on both
  • no alerts in my WebGUI on both
  • no error logs under System > General on master: {{{
    <27>Jun 8 09:44:13 php-fpm29604: /system_hasync.php: waiting for pfsync...
    <27>Jun 8 09:44:44 php-fpm29604: /system_hasync.php: pfsync done in 30 seconds.
    <27>Jun 8 09:44:44 php-fpm29604: /system_hasync.php: Configuring CARP settings finalize...
    }}}
  • PfSync is enabled on both
  • XML RPC Sync enabled on Master only pointing to -> Backup
  • XML RPC Sync settings on Backup are empty
  • both nodes have the same admin user and password installed

Generally no changes were made to the HA settings compared to the last working state.

TCPDumped the traffic on the PfSync Interface on Master and I can only see "state" Packets
leaving the interface towards backup.

I suppose the XML data never hits the wire.

My Version: 2.3.1p1

Actions #1

Updated by Chris Buechler almost 8 years ago

  • Status changed from New to Feedback

almost certainly not a bug. where no traffic, probably set to an IP other than what is correct

Actions #2

Updated by Michael Schmid almost 8 years ago

Chris Buechler wrote:

almost certainly not a bug. where no traffic, probably set to an IP other than what is correct

hi chris,

i've been running this for over a year without any changes...that's simply strange.
using a cluster setup with XML syncing in an other project for years (since 1.2.2) without any issues.

the only thing i changed in the last weeks was adding suricata and a elastic filebeat daemon to the
malfunctioning master could this affect syncing? ...as the pkgs are only installed on the master?

what confuses me is that i don't get any errors in the logs or alerts in the web gui.
I had once an issue when the user pass was accidentially changed on backup.
but herein i got immediately an alert (was under 2.1 i think).

could you give me an advice where i could look at or which scripts are involved in triggering
and executing the sync?

thanks for your help!

Michl

Actions #3

Updated by Michael Schmid almost 8 years ago

Hi Chris,

I found the bug.
It's not directly Pfsense related, it happens when I start the 3rd party binary

filebeat-freebsd-amd64

from Elastic.
The Filebeat gathers my Suricata log data from:
/var/log/suricata/*/eve.json

and passes it to my ELK Server.

The problem occurs ONLY if I start the binary via the "shellcmd" package with the following parameters:

Command: /etc/filebeat/filebeat-freebsd-amd64
Shellcmd Type: shellcmd

If I run it logged in from a SSH session I have no influences on the XML sync.

Any tips on how to further debug this to make it boot proof?
How could it block the sync as it's communicating on a completly different Interface, IP and Port (:5044)?

My FileBeat setup is based on: Another question is:
  • "Why is there no error visible" although the sync process doesn't exit correctly?

Thanks,

Michl

Actions #4

Updated by Chris Buechler almost 8 years ago

  • Status changed from Feedback to Not a Bug

what you did prevented the system from completing bootup, and while the system is booting, it doesn't (and shouldn't) config sync.

Actions #5

Updated by Michael Schmid almost 8 years ago

Thanks chris for pointing me on this!

I placed a little helper script that sends the program to the background:

/etc/filebeat/filebeat-freebsd-amd64 &

That script is symlinked to the startup folder:

/usr/local/etc/rc.d folder

The "shellcmd" entry is deleted and now it's boot proof.

I just don't know how I could do that with shellcmd directly?
The following setups have all been tried without success (boot hangs every time):

Command: /etc/filebeat/filebeat-freebsd-amd64 &
Shellcmd Type: shellcmd

Command: /etc/filebeat/filebeat.sh
Shellcmd Type: shellcmd

Actions

Also available in: Atom PDF