Project

General

Profile

Actions

Bug #15274

open

HAProxy Configuration Changes Require pfSense Reboot to Take Effect

Added by Zachary Cohen 9 months ago. Updated 4 months ago.

Status:
Incomplete
Priority:
Normal-package
Assignee:
-
Category:
haproxy
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
2.7.x
Affected Plus Version:
Affected Architecture:
amd64

Description

As originally reported here (https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed), changes made to the HAProxy configuration require a reboot to take effect.

I'm consistently able to reproduce this issue when adding new backends.

When browsing to the new backend, I receive a 503 - "no server is available to handle this request". After rebooting, it works as expected.

Other users have been able to validate that this issue was present starting with pfSense 2.6.0 and HAProxy version haproxy-devel 0.62.10.

While I was able to replicate that issue starting on that version, I'm currently replicating it in pfSense 2.7.2-RELEASE (amd64) and haproxy-devel 0.63_2.

Actions #1

Updated by Zachary Cohen 9 months ago

Zachary Cohen wrote:

As originally reported here (https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed), changes made to the HAProxy configuration require a reboot to take effect.

Also related to: https://forum.netgate.com/topic/184401/haproxy-seems-to-forward-to-wrong-backend-port and #15182

Actions #2

Updated by Kris Phillips 9 months ago

  • Status changed from New to Incomplete

Tested this on 23.09.1 with HAProxy 0.63_2. I'm not able to reproduce this. Changing any frontend or backend settings has no bearing on HAProxy's functionality for me. I also deleted and added a new backend. This did not result in any 503 errors.

There must be something specific in configs that are different from the basics that are causing this.

Zachary,

Do you have anything besides an HTTP and OPTIONS check method for health checks? Perhaps the type of health check makes a difference.

Marking as Incomplete for now until we have more information.

Actions #3

Updated by Zachary Cohen 9 months ago

Kris Phillips wrote in #note-2:

Tested this on 23.09.1 with HAProxy 0.63_2. I'm not able to reproduce this. Changing any frontend or backend settings has no bearing on HAProxy's functionality for me. I also deleted and added a new backend. This did not result in any 503 errors.

There must be something specific in configs that are different from the basics that are causing this.

Zachary,

Do you have anything besides an HTTP and OPTIONS check method for health checks? Perhaps the type of health check makes a difference.

Marking as Incomplete for now until we have more information.

Here's a simplified version of the config. This particular frontend/backend has given me trouble in the past.

I only recently added the HTTP check, but the issue occurred before I added it. Most of my backends have their health check set to "none".

<haproxy>
    <ha_backends>
        <item>
            <name>IOT</name>
            <descr><![CDATA[Reverse Proxy for the IOT Interface]]></descr>
            <status>active</status>
            <primary_frontend>Docker</primary_frontend>
            <type>http</type>
            <httpclose>http-keep-alive</httpclose>
            <ssloffloadcert>51001fee6ea42</ssloffloadcert>
            <advanced></advanced>
            <ha_acls>
                <item>
                    <name>ecowitt</name>
                    <expression>host_matches</expression>
                    <value>ecowitt.REDACTED.com</value>
                    <backendservercountbackend>Blackbox-exporter</backendservercountbackend>
                    <_index></_index>
                </item>
            </ha_acls>
            <ha_certificates>
                <item>
                    <ssl_certificate>51001fee6ea42</ssl_certificate>
                    <_index></_index>
                </item>
            </ha_certificates>
            <clientcert_ca></clientcert_ca>
            <clientcert_crl></clientcert_crl>
            <a_extaddr>
                <item>
                    <extaddr>10.0.0.254_ipv4</extaddr>
                    <extaddr_port>443</extaddr_port>
                    <extaddr_ssl>yes</extaddr_ssl>
                    <_index></_index>
                </item>
            </a_extaddr>
            <a_actionitems>
                <item>
                    <action>use_backend</action>
                    <acl>ecowitt</acl>
                    <use_backendbackend>Ecowitt</use_backendbackend>
                    <_index></_index>
                </item>
            </a_actionitems>
            <a_errorfiles></a_errorfiles>
            <forwardfor>yes</forwardfor>
        </item>
    </ha_backends>
    <ha_pools>
        <item>
            <ha_servers>
                <item>
                    <status>active</status>
                    <name>ecowitt</name>
                    <address>10.0.0.28</address>
                    <port>80</port>
                    <id>158</id>
                    <_index></_index>
                </item>
            </ha_servers>
            <a_acl></a_acl>
            <a_actionitems></a_actionitems>
            <errorfiles></errorfiles>
            <advanced></advanced>
            <advanced_backend></advanced_backend>
            <name>Ecowitt</name>
            <balance></balance>
            <balance_urilen></balance_urilen>
            <balance_uridepth></balance_uridepth>
            <balance_uriwhole></balance_uriwhole>
            <transparent_clientip></transparent_clientip>
            <transparent_interface>wan</transparent_interface>
            <check_type>HTTP</check_type>
            <checkinter>60000</checkinter>
            <log-health-checks>yes</log-health-checks>
            <httpcheck_method>GET</httpcheck_method>
            <monitor_uri>/</monitor_uri>
            <monitor_httpversion>HTTP/1.1\r\nHost:\ www</monitor_httpversion>
            <monitor_username></monitor_username>
            <monitor_domain></monitor_domain>
            <monitor_agentport>Ecowitt</monitor_agentport>
            <agent_check></agent_check>
            <agent_port></agent_port>
            <agent_inter></agent_inter>
            <connection_timeout></connection_timeout>
            <server_timeout></server_timeout>
            <retries></retries>
            <stats_enabled></stats_enabled>
            <stats_username></stats_username>
            <stats_password></stats_password>
            <stats_uri></stats_uri>
            <stats_scope></stats_scope>
            <stats_realm></stats_realm>
            <stats_admin></stats_admin>
            <stats_node></stats_node>
            <stats_desc></stats_desc>
            <stats_refresh></stats_refresh>
            <persist_stick_expire></persist_stick_expire>
            <persist_stick_tablesize></persist_stick_tablesize>
            <persist_stick_length></persist_stick_length>
            <persist_stick_cookiename></persist_stick_cookiename>
            <persist_sticky_type>none</persist_sticky_type>
            <persist_cookie_enabled></persist_cookie_enabled>
            <persist_cookie_name></persist_cookie_name>
            <persist_cookie_mode>passive</persist_cookie_mode>
            <persist_cookie_cachable></persist_cookie_cachable>
            <persist_cookie_postonly></persist_cookie_postonly>
            <persist_cookie_httponly></persist_cookie_httponly>
            <persist_cookie_secure></persist_cookie_secure>
            <haproxy_cookie_maxidle></haproxy_cookie_maxidle>
            <haproxy_cookie_maxlife></haproxy_cookie_maxlife>
            <haproxy_cookie_domains></haproxy_cookie_domains>
            <haproxy_cookie_dynamic_cookie_key></haproxy_cookie_dynamic_cookie_key>
            <strict_transport_security></strict_transport_security>
            <cookie_attribute_secure></cookie_attribute_secure>
            <email_level><![CDATA[alert]]></email_level>
            <email_to><![CDATA[REDACTED]]></email_to>
            <id>155</id>
        </item>
    </ha_pools>
    <configversion>00.58</configversion>
    <files></files>
    <email_mailers></email_mailers>
    <dns_resolvers></dns_resolvers>
    <maxconn>1000</maxconn>
    <logfacility>local0</logfacility>
    <loglevel>info</loglevel>
    <nbthread></nbthread>
    <hard_stop_after></hard_stop_after>
    <localstats_refreshtime></localstats_refreshtime>
    <localstats_sticktable_refreshtime></localstats_sticktable_refreshtime>
    <log-send-hostname></log-send-hostname>
    <sslcompatibilitymode>intermediate</sslcompatibilitymode>
    <ssldefaultdhparam></ssldefaultdhparam>
    <email_level></email_level>
    <email_myhostname></email_myhostname>
    <email_from></email_from>
    <email_to></email_to>
    <resolver_retries></resolver_retries>
    <resolver_timeoutretry></resolver_timeoutretry>
    <resolver_holdvalid></resolver_holdvalid>
    <localstatsport>2200</localstatsport>
    <advanced>REDACTED</advanced>
    <remotesyslog>/var/run/log</remotesyslog>
    <enable></enable>
</haproxy>
Actions #4

Updated by Brendon Baumgartner 5 months ago

Also discussed here.
https://forum.netgate.com/topic/178348/haproxy-backend-port-changes-are-not-applied

workaround is to delete /tmp/haproxy_server_state

Actions #5

Updated by Kilian Ries 4 months ago

Please see my last comment here: https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed/7

This seems to be related to the haproxy state file and the IDs you assign to the backend servers: /tmp/haproxy_server_state

If i delete the state file and reload haproxy the changes get applied correctly!

Actions

Also available in: Atom PDF