pfSense

Is there a ntpd core with crash dump that you can share? It could help us to identify the issue.

If anyone tells me where the ntpd core crash dump is located, sure.

I've built ntpd binaries with debug symbols, there are binaries for 2.0.3 and 2.1, i386 and amd64:

ntpd-2.0.3-amd64
ntpd-2.0.3-i386
ntpd-2.1-amd64
ntpd-2.1-i386

Choose the one according your system version and arch and you can get that onto a system from the shell like so:

fetch -o /root/ntpd.new http://files.pfsense.org/garga/##NTPD_BINARY_NAME##
chmod a+x /root/ntpd.new
killall -9 ntpd
mv /usr/local/bin/ntpd /usr/local/bin/ntpd.old
mv /root/ntpd.new /usr/local/bin/ntpd

Then go to Status -> Services on control panel and restart ntpd.

If you need to back the original binary in place just run:

mv /usr/local/bin/ntpd.old /usr/local/bin/ntpd

After this procedure i can't start the NTPD service.

OK it's started now. I had to go Services - NTP - press Save button. Then press the restart service button. Now it's running.

Since implement your new file yesterday, i habe no more ntpd crashes. I will report again at the end of the week.

Not any single crash with the new file. The OpenNTPD service is running rock-stable. No crash, no error in system-logs.
Good Job!

Would be great if you can include i in the next snapshot.

Could you confirm if the problem persists on more recent snapshots?

For information, here is the stack :

$ gdb ntpd ntpd.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `ntpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x283d2df3 in ?? ()
(gdb) where
#0  0x283d2df3 in ?? ()
#1  0x08066c5b in more_pkt ()
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_request.c:327
#2  0x080519ff in ctlsettrap ()
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_control.c:2593
#3  0x0805b130 in input_handler (cts=0x0)
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_io.c:3049
#4  0x0805b582 in input_handler (cts=0x0)
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_io.c:2882
#5  0x0804c3e7 in getnetnum (num=0xbfbfedc8 "������\004\b\006", addr=0x750, 
    complain=-1077940716, a_type=671928416)
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_config.c:2289
#6  0x0804c358 in getnetnum (num=0x804c3e7 "", addr=0x5, complain=-1077940760, 
    a_type=3217026836)
    at /usr/src/usr.sbin/ntp/ntpd/../../../contrib/ntp/ntpd/ntp_config.c:2229
#7  0x00000006 in ?? ()

I can confirm that NTPD doesn't come up. It happens everytime after reboot the firewall.

Pierre POMES wrote:

For information, here is the stack :

[...]

Forget this, this stack is wrong.

Renato, I think the ntpd you built is from a BSD base system (ntp 4.2.4) and not from ports (4.2.6). The ntpd used by pfSense is in /usr/local/bin and is coming from ports.

So the stack I posted was wrong.

Pierre

Pierre POMES wrote:

Pierre POMES wrote:

For information, here is the stack :

[...]

Forget this, this stack is wrong.

Renato, I think the ntpd you built is from a BSD base system (ntp 4.2.4) and not from ports (4.2.6). The ntpd used by pfSense is in /usr/local/bin and is coming from ports.

So the stack I posted was wrong.

Yes, you are right. I apologize for the mistake. I'm going to provide new binaries.

Pierre POMES wrote:

Pierre POMES wrote:

For information, here is the stack :

[...]

Forget this, this stack is wrong.

Renato, I think the ntpd you built is from a BSD base system (ntp 4.2.4) and not from ports (4.2.6). The ntpd used by pfSense is in /usr/local/bin and is coming from ports.

So the stack I posted was wrong.

Pierre

New binaries are available at http://files.pfsense.org/garga/ and steps were updated on comment 6. Could you please try again?

Regards

Loaded the new ntpd binaries into my firewall.
Now i got: kernel: pid 21367 (ntpd), uid 0: exited on signal 11 (core dumped)

Benjamin H. wrote:

Loaded the new ntpd binaries into my firewall.
Now i got: kernel: pid 21367 (ntpd), uid 0: exited on signal 11 (core dumped)

Could you please let me know pfSense version and arch you are using and also send me the ntpd.core?

Updatet to latest snapshot, 2.1 RC1 amd64 and replaced the NTPD file again with your provided file in post #19.
After a reboot there is no more error with ntpd crash. But the main problem still exist; NTPD service doesn't come up by itself (never ever).

The system log lies, because it says ntpd is up and running.
Aug 7 17:14:22 ntpd⁷⁶¹¹⁹: ntpd 4.2.6p5@1.2349-o Mon Aug 5 15:15:14 UTC 2013 (1)
Aug 7 17:14:22 ntp: Starting NTP Daemon.

but in the dashboard and under services it is not running.
I have to go to Services - NTP - then press Save. After this its shown as running.

Benjamin H. wrote:

Updatet to latest snapshot, 2.1 RC1 amd64 and replaced the NTPD file again with your provided file in post #19.
After a reboot there is no more error with ntpd crash. But the main problem still exist; NTPD service doesn't come up by itself (never ever).

The system log lies, because it says ntpd is up and running.
Aug 7 17:14:22 ntpd⁷⁶¹¹⁹: ntpd 4.2.6p5@1.2349-o Mon Aug 5 15:15:14 UTC 2013 (1)
Aug 7 17:14:22 ntp: Starting NTP Daemon.

but in the dashboard and under services it is not running.
I have to go to Services - NTP - then press Save. After this its shown as running.

NTPd is starting fine on all my systems, could you share ntpd section of config.xml with me?

It is also incredibly inconsistent. If I wait an hour and try to start, perhaps ntpd will start without issue and run.

Problem seems to be related to line:
statsdir /var/log/ntp

in /var/etc/ntp.conf.

After commenting this line, ntpd is starting correctly (no more core dumps).

Could someone check if it might be a permission/jail/chroot/wrong option issues?

thank you

also line
driftfile /var/db/ntpd.drift

seems to be involved.

those really seems to be directory permission issues to me, as process ntpd seems not being able to read/write on those directory/files...so it crashes!

I commented both lines on my machines and ntpd is running right now...

Sorry for posting here again. But I still got the "exited on signal 11 (core dumped)" error sometimes.
NTP runs fine for a long time but suddenly (probably when IPs change) it crashes and cannot be started again.
Various tips here and on the forums didn't help (make logfile, driftfile, world-writable, remote config lines,..).

The only thing that helps is a reboot.

I can reproduce it very easily, I just have to restart the OpenVPN Serivce and NTP dies and never comes back to life again (until a reboot of the server).

It's version 2.1 amd64 on a Dell PowerEdge 210 II, also running OpenVPN. But I think the problem did also occur before using OpenVPN.

I can report that this happens on i386 and amd64. I've switched architectures trying to avoid this bug. I can also report that it occurs on boxes with openvpn running, and with boxes that do not have openvpn running. Its definitely broken, but no word from anyone about re-opening this bug? Still shows as Status: Resolved. It most definitely isn't. Can we at least get an acknowledgment from the pfsense team that this issue is known?

I did just a series of test, and the core dump of ntpd seems strictly related to the presence of the file /var/db/ntpd.drift.
I deleted it, and the ntpd services started well being stable.

What I was wondering is: may it be that /var/db/ directory can only be written during boot-up phase?

That could explain why, when you restart ntpd (and ntpd is restarted in many occasion, such as restarting other network services) it starts to core dump...

Fabio Giudici wrote:

I did just a series of test, and the core dump of ntpd seems strictly related to the presence of the file /var/db/ntpd.drift.
I deleted it, and the ntpd services started well being stable.

What I was wondering is: may it be that /var/db/ directory can only be written during boot-up phase?

That could explain why, when you restart ntpd (and ntpd is restarted in many occasion, such as restarting other network services) it starts to core dump...

/var/db is always read/write, on Full installation it's a regular directory on /, and on nanobsd /var is mounted on a memory partition, that is also always read/write.

I see a problem on my 2.1 64-bit system with NTPD that may be related to the issues reported here. Anytime the WAN interface bounces (I have cable modem service with DHCP on the pfSense WAN, and sometimes the cable signal goes out for a while), NTPD will crash/die and not restart. When I catch that it is not running, I can usually manually start it from the Status...Services menu.

Should it crash/stop running just because the WAN interface is bounced? I have it configured to actually listen on only the LAN interface since I have an internal NTP server on the network. I never saw this behavior with my old 2.0.3 box.

Bill

Good morning
Just one more question: is it ntpd running in jail/chroot?

Just to restrict the issue...but it seems to be a bug of ntpd to me...deleting /var/db/ntpd.drift always solves the issue.

Fabio

Renato Botelho wrote:

/var/db is always read/write, on Full installation it's a regular directory on /, and on nanobsd /var is mounted on a memory partition, that is also always read/write.

Fabio Giudici wrote:

Good morning
Just one more question: is it ntpd running in jail/chroot?

Just to restrict the issue...but it seems to be a bug of ntpd to me...deleting /var/db/ntpd.drift always solves the issue.

No, it's not chrooted.

What is the content of ntpd.drift when ntpd crashes?

simply one line contining:

-0.056

(or other numbers)

Please test on 2.1.1 and report back. https://forum.pfsense.org/index.php/topic,71546.0.html

Upgraded one of my clusters to 2.1.1-PRERELEASE (amd64) built on Wed Feb 19 19:46:29 EST 2014.

Can confirm NTPD is working as it should. Starts on boot on both nodes. Can be manually stopped/started without an issue.

Well done guys!

Thank you.