Bug #15274
openHAProxy Configuration Changes Require pfSense Reboot to Take Effect
Added by Zachary Cohen 6 months ago. Updated about 1 month ago.
0%
Description
As originally reported here (https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed), changes made to the HAProxy configuration require a reboot to take effect.
I'm consistently able to reproduce this issue when adding new backends.
When browsing to the new backend, I receive a 503 - "no server is available to handle this request". After rebooting, it works as expected.
Other users have been able to validate that this issue was present starting with pfSense 2.6.0 and HAProxy version haproxy-devel 0.62.10.
While I was able to replicate that issue starting on that version, I'm currently replicating it in pfSense 2.7.2-RELEASE (amd64) and haproxy-devel 0.63_2.
Updated by Zachary Cohen 6 months ago
Zachary Cohen wrote:
As originally reported here (https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed), changes made to the HAProxy configuration require a reboot to take effect.
Also related to: https://forum.netgate.com/topic/184401/haproxy-seems-to-forward-to-wrong-backend-port and #15182
Updated by Kris Phillips 6 months ago
- Status changed from New to Incomplete
Tested this on 23.09.1 with HAProxy 0.63_2. I'm not able to reproduce this. Changing any frontend or backend settings has no bearing on HAProxy's functionality for me. I also deleted and added a new backend. This did not result in any 503 errors.
There must be something specific in configs that are different from the basics that are causing this.
Zachary,
Do you have anything besides an HTTP and OPTIONS check method for health checks? Perhaps the type of health check makes a difference.
Marking as Incomplete for now until we have more information.
Updated by Zachary Cohen 6 months ago
Kris Phillips wrote in #note-2:
Tested this on 23.09.1 with HAProxy 0.63_2. I'm not able to reproduce this. Changing any frontend or backend settings has no bearing on HAProxy's functionality for me. I also deleted and added a new backend. This did not result in any 503 errors.
There must be something specific in configs that are different from the basics that are causing this.
Zachary,
Do you have anything besides an HTTP and OPTIONS check method for health checks? Perhaps the type of health check makes a difference.
Marking as Incomplete for now until we have more information.
Here's a simplified version of the config. This particular frontend/backend has given me trouble in the past.
I only recently added the HTTP check, but the issue occurred before I added it. Most of my backends have their health check set to "none".
<haproxy>
<ha_backends>
<item>
<name>IOT</name>
<descr><![CDATA[Reverse Proxy for the IOT Interface]]></descr>
<status>active</status>
<primary_frontend>Docker</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>51001fee6ea42</ssloffloadcert>
<advanced></advanced>
<ha_acls>
<item>
<name>ecowitt</name>
<expression>host_matches</expression>
<value>ecowitt.REDACTED.com</value>
<backendservercountbackend>Blackbox-exporter</backendservercountbackend>
<_index></_index>
</item>
</ha_acls>
<ha_certificates>
<item>
<ssl_certificate>51001fee6ea42</ssl_certificate>
<_index></_index>
</item>
</ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>10.0.0.254_ipv4</extaddr>
<extaddr_port>443</extaddr_port>
<extaddr_ssl>yes</extaddr_ssl>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems>
<item>
<action>use_backend</action>
<acl>ecowitt</acl>
<use_backendbackend>Ecowitt</use_backendbackend>
<_index></_index>
</item>
</a_actionitems>
<a_errorfiles></a_errorfiles>
<forwardfor>yes</forwardfor>
</item>
</ha_backends>
<ha_pools>
<item>
<ha_servers>
<item>
<status>active</status>
<name>ecowitt</name>
<address>10.0.0.28</address>
<port>80</port>
<id>158</id>
<_index></_index>
</item>
</ha_servers>
<a_acl></a_acl>
<a_actionitems></a_actionitems>
<errorfiles></errorfiles>
<advanced></advanced>
<advanced_backend></advanced_backend>
<name>Ecowitt</name>
<balance></balance>
<balance_urilen></balance_urilen>
<balance_uridepth></balance_uridepth>
<balance_uriwhole></balance_uriwhole>
<transparent_clientip></transparent_clientip>
<transparent_interface>wan</transparent_interface>
<check_type>HTTP</check_type>
<checkinter>60000</checkinter>
<log-health-checks>yes</log-health-checks>
<httpcheck_method>GET</httpcheck_method>
<monitor_uri>/</monitor_uri>
<monitor_httpversion>HTTP/1.1\r\nHost:\ www</monitor_httpversion>
<monitor_username></monitor_username>
<monitor_domain></monitor_domain>
<monitor_agentport>Ecowitt</monitor_agentport>
<agent_check></agent_check>
<agent_port></agent_port>
<agent_inter></agent_inter>
<connection_timeout></connection_timeout>
<server_timeout></server_timeout>
<retries></retries>
<stats_enabled></stats_enabled>
<stats_username></stats_username>
<stats_password></stats_password>
<stats_uri></stats_uri>
<stats_scope></stats_scope>
<stats_realm></stats_realm>
<stats_admin></stats_admin>
<stats_node></stats_node>
<stats_desc></stats_desc>
<stats_refresh></stats_refresh>
<persist_stick_expire></persist_stick_expire>
<persist_stick_tablesize></persist_stick_tablesize>
<persist_stick_length></persist_stick_length>
<persist_stick_cookiename></persist_stick_cookiename>
<persist_sticky_type>none</persist_sticky_type>
<persist_cookie_enabled></persist_cookie_enabled>
<persist_cookie_name></persist_cookie_name>
<persist_cookie_mode>passive</persist_cookie_mode>
<persist_cookie_cachable></persist_cookie_cachable>
<persist_cookie_postonly></persist_cookie_postonly>
<persist_cookie_httponly></persist_cookie_httponly>
<persist_cookie_secure></persist_cookie_secure>
<haproxy_cookie_maxidle></haproxy_cookie_maxidle>
<haproxy_cookie_maxlife></haproxy_cookie_maxlife>
<haproxy_cookie_domains></haproxy_cookie_domains>
<haproxy_cookie_dynamic_cookie_key></haproxy_cookie_dynamic_cookie_key>
<strict_transport_security></strict_transport_security>
<cookie_attribute_secure></cookie_attribute_secure>
<email_level><![CDATA[alert]]></email_level>
<email_to><![CDATA[REDACTED]]></email_to>
<id>155</id>
</item>
</ha_pools>
<configversion>00.58</configversion>
<files></files>
<email_mailers></email_mailers>
<dns_resolvers></dns_resolvers>
<maxconn>1000</maxconn>
<logfacility>local0</logfacility>
<loglevel>info</loglevel>
<nbthread></nbthread>
<hard_stop_after></hard_stop_after>
<localstats_refreshtime></localstats_refreshtime>
<localstats_sticktable_refreshtime></localstats_sticktable_refreshtime>
<log-send-hostname></log-send-hostname>
<sslcompatibilitymode>intermediate</sslcompatibilitymode>
<ssldefaultdhparam></ssldefaultdhparam>
<email_level></email_level>
<email_myhostname></email_myhostname>
<email_from></email_from>
<email_to></email_to>
<resolver_retries></resolver_retries>
<resolver_timeoutretry></resolver_timeoutretry>
<resolver_holdvalid></resolver_holdvalid>
<localstatsport>2200</localstatsport>
<advanced>REDACTED</advanced>
<remotesyslog>/var/run/log</remotesyslog>
<enable></enable>
</haproxy>
Updated by Brendon Baumgartner about 1 month ago
Also discussed here.
https://forum.netgate.com/topic/178348/haproxy-backend-port-changes-are-not-applied
workaround is to delete /tmp/haproxy_server_state
Updated by Kilian Ries about 1 month ago
Please see my last comment here: https://forum.netgate.com/topic/172972/haproxy-config-changes-not-loaded-pfsense-restart-needed/7
This seems to be related to the haproxy state file and the IDs you assign to the backend servers: /tmp/haproxy_server_state
If i delete the state file and reload haproxy the changes get applied correctly!