pfSense » pfSense Packages

Zachary Cohen wrote:

As originally reported here (https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed), changes made to the HAProxy configuration require a reboot to take effect.

Also related to: https://forum.netgate.com/topic/184401/haproxy-seems-to-forward-to-wrong-backend-port and #15182

Kris Phillips wrote in #note-2:

Tested this on 23.09.1 with HAProxy 0.63_2. I'm not able to reproduce this. Changing any frontend or backend settings has no bearing on HAProxy's functionality for me. I also deleted and added a new backend. This did not result in any 503 errors.

There must be something specific in configs that are different from the basics that are causing this.

Zachary,

Do you have anything besides an HTTP and OPTIONS check method for health checks? Perhaps the type of health check makes a difference.

Marking as Incomplete for now until we have more information.

Here's a simplified version of the config. This particular frontend/backend has given me trouble in the past.

I only recently added the HTTP check, but the issue occurred before I added it. Most of my backends have their health check set to "none".

<haproxy>
    <ha_backends>
        <item>
            <name>IOT</name>
            <descr><![CDATA[Reverse Proxy for the IOT Interface]]></descr>
            <status>active</status>
            <primary_frontend>Docker</primary_frontend>
            <type>http</type>
            <httpclose>http-keep-alive</httpclose>
            <ssloffloadcert>51001fee6ea42</ssloffloadcert>
            <advanced></advanced>
            <ha_acls>
                <item>
                    <name>ecowitt</name>
                    <expression>host_matches</expression>
                    <value>ecowitt.REDACTED.com</value>
                    <backendservercountbackend>Blackbox-exporter</backendservercountbackend>
                    <_index></_index>
                </item>
            </ha_acls>
            <ha_certificates>
                <item>
                    <ssl_certificate>51001fee6ea42</ssl_certificate>
                    <_index></_index>
                </item>
            </ha_certificates>
            <clientcert_ca></clientcert_ca>
            <clientcert_crl></clientcert_crl>
            <a_extaddr>
                <item>
                    <extaddr>10.0.0.254_ipv4</extaddr>
                    <extaddr_port>443</extaddr_port>
                    <extaddr_ssl>yes</extaddr_ssl>
                    <_index></_index>
                </item>
            </a_extaddr>
            <a_actionitems>
                <item>
                    <action>use_backend</action>
                    <acl>ecowitt</acl>
                    <use_backendbackend>Ecowitt</use_backendbackend>
                    <_index></_index>
                </item>
            </a_actionitems>
            <a_errorfiles></a_errorfiles>
            <forwardfor>yes</forwardfor>
        </item>
    </ha_backends>
    <ha_pools>
        <item>
            <ha_servers>
                <item>
                    <status>active</status>
                    <name>ecowitt</name>
                    <address>10.0.0.28</address>
                    <port>80</port>
                    <id>158</id>
                    <_index></_index>
                </item>
            </ha_servers>
            <a_acl></a_acl>
            <a_actionitems></a_actionitems>
            <errorfiles></errorfiles>
            <advanced></advanced>
            <advanced_backend></advanced_backend>
            <name>Ecowitt</name>
            <balance></balance>
            <balance_urilen></balance_urilen>
            <balance_uridepth></balance_uridepth>
            <balance_uriwhole></balance_uriwhole>
            <transparent_clientip></transparent_clientip>
            <transparent_interface>wan</transparent_interface>
            <check_type>HTTP</check_type>
            <checkinter>60000</checkinter>
            <log-health-checks>yes</log-health-checks>
            <httpcheck_method>GET</httpcheck_method>
            <monitor_uri>/</monitor_uri>
            <monitor_httpversion>HTTP/1.1\r\nHost:\ www</monitor_httpversion>
            <monitor_username></monitor_username>
            <monitor_domain></monitor_domain>
            <monitor_agentport>Ecowitt</monitor_agentport>
            <agent_check></agent_check>
            <agent_port></agent_port>
            <agent_inter></agent_inter>
            <connection_timeout></connection_timeout>
            <server_timeout></server_timeout>
            <retries></retries>
            <stats_enabled></stats_enabled>
            <stats_username></stats_username>
            <stats_password></stats_password>
            <stats_uri></stats_uri>
            <stats_scope></stats_scope>
            <stats_realm></stats_realm>
            <stats_admin></stats_admin>
            <stats_node></stats_node>
            <stats_desc></stats_desc>
            <stats_refresh></stats_refresh>
            <persist_stick_expire></persist_stick_expire>
            <persist_stick_tablesize></persist_stick_tablesize>
            <persist_stick_length></persist_stick_length>
            <persist_stick_cookiename></persist_stick_cookiename>
            <persist_sticky_type>none</persist_sticky_type>
            <persist_cookie_enabled></persist_cookie_enabled>
            <persist_cookie_name></persist_cookie_name>
            <persist_cookie_mode>passive</persist_cookie_mode>
            <persist_cookie_cachable></persist_cookie_cachable>
            <persist_cookie_postonly></persist_cookie_postonly>
            <persist_cookie_httponly></persist_cookie_httponly>
            <persist_cookie_secure></persist_cookie_secure>
            <haproxy_cookie_maxidle></haproxy_cookie_maxidle>
            <haproxy_cookie_maxlife></haproxy_cookie_maxlife>
            <haproxy_cookie_domains></haproxy_cookie_domains>
            <haproxy_cookie_dynamic_cookie_key></haproxy_cookie_dynamic_cookie_key>
            <strict_transport_security></strict_transport_security>
            <cookie_attribute_secure></cookie_attribute_secure>
            <email_level><![CDATA[alert]]></email_level>
            <email_to><![CDATA[REDACTED]]></email_to>
            <id>155</id>
        </item>
    </ha_pools>
    <configversion>00.58</configversion>
    <files></files>
    <email_mailers></email_mailers>
    <dns_resolvers></dns_resolvers>
    <maxconn>1000</maxconn>
    <logfacility>local0</logfacility>
    <loglevel>info</loglevel>
    <nbthread></nbthread>
    <hard_stop_after></hard_stop_after>
    <localstats_refreshtime></localstats_refreshtime>
    <localstats_sticktable_refreshtime></localstats_sticktable_refreshtime>
    <log-send-hostname></log-send-hostname>
    <sslcompatibilitymode>intermediate</sslcompatibilitymode>
    <ssldefaultdhparam></ssldefaultdhparam>
    <email_level></email_level>
    <email_myhostname></email_myhostname>
    <email_from></email_from>
    <email_to></email_to>
    <resolver_retries></resolver_retries>
    <resolver_timeoutretry></resolver_timeoutretry>
    <resolver_holdvalid></resolver_holdvalid>
    <localstatsport>2200</localstatsport>
    <advanced>REDACTED</advanced>
    <remotesyslog>/var/run/log</remotesyslog>
    <enable></enable>
</haproxy>

Also discussed here.
https://forum.netgate.com/topic/178348/haproxy-backend-port-changes-are-not-applied

workaround is to delete /tmp/haproxy_server_state

Please see my last comment here: https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed/7

This seems to be related to the haproxy state file and the IDs you assign to the backend servers: /tmp/haproxy_server_state

If i delete the state file and reload haproxy the changes get applied correctly!

Kris Phillips I found what is adding these directives to the config file!

The behavior is hard-coded in /usr/local/pkg/haproxy/haproxy.inc which is what generates /var/etc/haproxy/haproxy.cfg

Line 727
function write_backend($configpath, $fd, $name, $pool, $backendsettings) {
                    ...
    Line 1006  (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

Line 1520
function haproxy_writeconf($configpath) {
                    ...
    Line 1612 (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tserver-state-file /tmp/haproxy_server_state\n");

Line 2498
function haproxy_check_run($reload) {
                    ...
    Lines 2507-2508 (executes if haproxy is running when a reload is requested, which is almost always true)
    if ($reload) {
        if (haproxy_is_running()) {
        $r = haproxy_socket_command("show servers state");
            file_put_contents("/tmp/haproxy_server_state", $r);
        }

These lines result in the following behavior:

• Whenever HAProxy's settings are saved using the GUI, the server-state-file directive is added to the global section of the resulting config file.
• Whenever HAProxy's settings are saved using the GUI, the load-server-state-from-file directive is added to each backend of the resulting config file.
• Whenever HAProxy reloads, it writes the backend server state to /tmp/haproxy_server_state, and due to the server-state-file directive, it then always reads and applies the previous state data.

There is no GUI text that mentions that this behavior is hard-coded, and there is no way to disabled it other than commenting out the lines (1006, 1612, 2507, 2508) in haproxy.inc that are responsible for adding the directives to the config file.

The GUI option Reload behaviour is responsible for the directive hard_stop_after which is unrelated.

The line

fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

https://github.com/pfsense/FreeBSD-ports/commit/9f7d258917ece10b6d55435776d2db85370e289c

I believe this might be what caused the behavior to change. Previously, the server-state-file directive was already being added, but there was no directive to tell the backends to use the saved state data.

Commit 9f7d258 added the load-server-state-from-file directive to all backends, which results in each backend loading the saved state data whenever HAProxy starts or reloads.

Adding
load-server-state-from-file none
to the Advanced Settings > Backend pass thru section of each backend overrides the behavior and makes backend changes apply immediately when reloading.

To resolve this issue, I think that either the default backend behavior needs to be changed to NOT load state from file (remove the load-server-state-from-file line, or change the value from global to none), or there needs to be GUI options for enabling/disabling the load-server-state-from-file and server-state-file directives.

Looking through older Redmine issues for HAProxy and I found Issue #11756 which is asking for HAProxy to keep backend states during reload, 2 years prior to the commit that introduced the behavior change.

Since there are cases for using one behavior or the other, a GUI option to change the setting is definitely needed.

Marcos M Thanks for taking care of this so quickly!
Will a patch or updated HAProxy package be released for 25.07, or will we need to wait until the 25.11 beta is released to start testing?

It relies on a change to the main tree hence the updated package will be available starting with 25.11.