Bug #6478
closed
Master not sending XML RPC sync data to Backup node
Added by Michael Schmid almost 8 years ago.
Updated almost 8 years ago.
Description
I noticed that my backup node is out of XML sync although it's activated on master.
State sync is working fine.
The sync is configured to work on an dedicated PfSync VLAN where the state sync as well as the XML sync should go.
I doublechecked the following:
- both nodes have the same protocol (HTTPS) and port (443)
- traffic on PfSync interface is allowed on both nodes
- no log entries for related firewall events on both
- no alerts in my WebGUI on both
- no error logs under System > General on master:
{{{
<27>Jun 8 09:44:13 php-fpm29604: /system_hasync.php: waiting for pfsync...
<27>Jun 8 09:44:44 php-fpm29604: /system_hasync.php: pfsync done in 30 seconds.
<27>Jun 8 09:44:44 php-fpm29604: /system_hasync.php: Configuring CARP settings finalize...
}}}
- PfSync is enabled on both
- XML RPC Sync enabled on Master only pointing to -> Backup
- XML RPC Sync settings on Backup are empty
- both nodes have the same admin user and password installed
Generally no changes were made to the HA settings compared to the last working state.
TCPDumped the traffic on the PfSync Interface on Master and I can only see "state" Packets
leaving the interface towards backup.
I suppose the XML data never hits the wire.
My Version: 2.3.1p1
- Status changed from New to Feedback
almost certainly not a bug. where no traffic, probably set to an IP other than what is correct
Chris Buechler wrote:
almost certainly not a bug. where no traffic, probably set to an IP other than what is correct
hi chris,
i've been running this for over a year without any changes...that's simply strange.
using a cluster setup with XML syncing in an other project for years (since 1.2.2) without any issues.
the only thing i changed in the last weeks was adding suricata and a elastic filebeat daemon to the
malfunctioning master could this affect syncing? ...as the pkgs are only installed on the master?
what confuses me is that i don't get any errors in the logs or alerts in the web gui.
I had once an issue when the user pass was accidentially changed on backup.
but herein i got immediately an alert (was under 2.1 i think).
could you give me an advice where i could look at or which scripts are involved in triggering
and executing the sync?
thanks for your help!
Michl
Hi Chris,
I found the bug.
It's not directly Pfsense related, it happens when I start the 3rd party binary
filebeat-freebsd-amd64
from Elastic.
The Filebeat gathers my Suricata log data from:
/var/log/suricata/*/eve.json
and passes it to my ELK Server.
The problem occurs ONLY if I start the binary via the "shellcmd" package with the following parameters:
Command: /etc/filebeat/filebeat-freebsd-amd64
Shellcmd Type: shellcmd
If I run it logged in from a SSH session I have no influences on the XML sync.
Any tips on how to further debug this to make it boot proof?
How could it block the sync as it's communicating on a completly different Interface, IP and Port (:5044)?
My FileBeat setup is based on:
Another question is:
- "Why is there no error visible" although the sync process doesn't exit correctly?
Thanks,
Michl
- Status changed from Feedback to Not a Bug
what you did prevented the system from completing bootup, and while the system is booting, it doesn't (and shouldn't) config sync.
Thanks chris for pointing me on this!
I placed a little helper script that sends the program to the background:
/etc/filebeat/filebeat-freebsd-amd64 &
That script is symlinked to the startup folder:
/usr/local/etc/rc.d folder
The "shellcmd" entry is deleted and now it's boot proof.
I just don't know how I could do that with shellcmd directly?
The following setups have all been tried without success (boot hangs every time):
Command: /etc/filebeat/filebeat-freebsd-amd64 &
Shellcmd Type: shellcmd
Command: /etc/filebeat/filebeat.sh
Shellcmd Type: shellcmd
Also available in: Atom
PDF
Updated by 
Updated by