Project

General

Profile

Actions

Bug #6186

open

race conditions in service startup

Added by Chris Buechler almost 8 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Services
Target version:
Start date:
04/17/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
All
Affected Architecture:

Description

There have always been a variety of possibilities for race conditions in service startup because of the nature of how multiple different things can call the functions that do the startup. That's a larger architectural issue which we're discussing options for properly addressing in the future.

The more immediate issue is after removing the "exit if booting" check from rc.newwanip(v6) in 2.3, which fixed a variety of edge case bugs with interfaces that are slow to come online during boot, some systems end up running certain things twice at almost exactly the same time. For instance, #6160, and probably #6132.

Adding locks in vpn_ipsec_configure was fine for strongswan in #6160. Might be fine in other areas, though adding locking like that can be risky in potentially breaking things that are fine now, if some of those functions end up recursing.

Actions #1

Updated by Chris Buechler almost 8 years ago

  • Description updated (diff)
Actions #2

Updated by sebastian nielsen almost 8 years ago

Same applies for services that start up in the wrong order. So if a VPN client interface is slow to start up, and a unbound DNS forwarder has the VPN client interface as outgoing, unbound will sometimes start before the VPN client interface has came up, causing the unbound server to permanently return "SERVFAIL" as it reports a configuration error since said interface didn't exist at start.

Same applies to routing.

I think the whole architecture needs recoding such as it will first always bring up the interfaces including starting services related to interfaces such as VPN clients/servers, (blocking operation), then start any services not related to interfaces (also a blocking operation), and then apply any firewall rules, custom routes, default gateway and NAT.

Actions #3

Updated by Jim Pingle almost 8 years ago

VPN and DNS is not that clear a solution. You have a chicken-and-egg scenario there. In plenty of cases you need working DNS before the VPN can be brought up, especially if you are using a hostname for the VPN peer. In that case you'd have to start DNS, then the VPN, then restart DNS if it doesn't (re)attach to the VPN interface.

Actions #4

Updated by sebastian nielsen almost 8 years ago

yes that would be a good idea. Forced restart on unbound after VPN success. However, it can be a good idea to then delay the restart of unbound a few seconds after VPN success to ensure the interface has "settled" before attempting to attach DNS server to it.

Actions #5

Updated by Chris Buechler almost 8 years ago

  • Target version changed from 2.3.1 to 2.3.2

What I committed takes things back to 2.2.x and earlier behavior, plus retaining the fix for #5952. That's confirmed to fix/avoid the "pf wedged" issues, things like unbound and dhcpd starting twice at almost exactly the same time (though those don't hurt anything, ugly log spam), among other things.

Virtually all the race conditions people have encountered are from that change in rc.newwanip during boot. So we're at 2.2.6 and a bit better now for 2.3.1.

But there's a larger architectural issue to be addressed. This is a hack to avoid these kinds of issues.

Actions #6

Updated by Chris Buechler over 7 years ago

  • Target version changed from 2.3.2 to 2.4.0
Actions #7

Updated by Renato Botelho about 7 years ago

  • Assignee deleted (Marc Dye)
Actions #8

Updated by Jim Thompson about 7 years ago

  • Assignee set to Renato Botelho
Actions #9

Updated by John Cairns about 7 years ago

I've run into this issue as well on my pfSense machines that have ovpn client interfaces set as the outgoing interfaces for unbound. Although in my case, I don't see unbound fail to start, but rather unbound.conf reverts to its default of using all interfaces as outgoing interfaces. I don't know if this is helpful information or not, but I started a thread on the forums in which I include more detailed information:
[[https://forum.pfsense.org/index.php?topic=126925.0]]

Also, I understand the point about the DNS/VPN chicken and egg scenario, although I just use raw IPs for my VPN client connections. I feel that's a reasonable expectation, especially since most people using VPN client connections likely want all DNS traffic flowing through them as well. That said, I am by no means trying to evangelize here, just throwing in my two cents :)

Actions #10

Updated by Renato Botelho over 6 years ago

  • Target version changed from 2.4.0 to 2.4.1
Actions #11

Updated by Jim Pingle over 6 years ago

  • Target version changed from 2.4.1 to 2.4.2
Actions #12

Updated by Jim Pingle over 6 years ago

  • Target version changed from 2.4.2 to 2.4.3
Actions #13

Updated by Abuzer Rafey about 6 years ago

The more immediate issue is after removing the "exit if booting" check from rc.newwanip(v6) in 2.3, which fixed a variety of edge case bugs with interfaces that are slow to come online during boot, some systems end up running certain things twice at almost exactly the same time. For instance, #6160, and probably #6132.

Would this fix #5999?

Actions #14

Updated by Jim Pingle about 6 years ago

  • Target version changed from 2.4.3 to 2.4.4
Actions #15

Updated by Anonymous over 5 years ago

  • Status changed from Confirmed to New
Actions #16

Updated by Anonymous over 5 years ago

  • Status changed from New to 13
Actions #17

Updated by Anonymous over 5 years ago

  • Status changed from 13 to New
Actions #18

Updated by Anonymous over 5 years ago

  • Target version changed from 2.4.4 to 48
Actions #19

Updated by Jim Pingle about 5 years ago

  • Target version changed from 48 to 2.5.0
Actions #20

Updated by Jim Pingle over 4 years ago

  • Category set to Services
Actions #21

Updated by Anonymous over 3 years ago

  • Target version changed from 2.5.0 to Future
Actions #22

Updated by Renato Botelho almost 2 years ago

  • Assignee deleted (Renato Botelho)
Actions

Also available in: Atom PDF