Bug #16341
closed
Error notification and log message ``"Updating repositories metadata" returned error code 1`` at boot due to ``certctl`` race condition
100%
Description
There are certain environments where users see an error notification when logging in after (re)booting which reads:
check_upgrade: "Updating repositories metadata" returned error code 1
This does not affect most installations, and can be difficult to reproduce. Most devices never encounter the error, some devices only encounter the error occasionally, a handful encounter the error at every boot. The error is too vague to know if each potential problem scenario is related.
At least at boot this error may be due to a race condition between /etc/rc.update_pkg_metadata attempting to run while certctl rehash is still running. On slow systems the certctl rehash process can take about 2 minutes to finish.
On a VM which can reproduce this error at every boot, delaying /etc/rc.update_pkg_metadata by 130 seconds allows it to complete without error. Similarly, commenting out the call to certctl rehash in the CA trust store process also allows it to complete without error.
Files
Updated by Jim Pingle 2 months ago
- Status changed from Confirmed to Feedback
- % Done changed from 0 to 100
They may be a non-issue now on current dev snapshots. The latest upstream src merge brought in the new certctl implementation from upstream which was rewritten in C.
That new implementation is much, much faster , sometimes finishing in under a second even on slow hardware, rather than taking several minutes.
Once on a current dev snapshot (as of today, at minimum) this particular race shouldn't be a concern. If it still happens, add a comment here with more info.
Updated by Jordan G about 2 months ago
I think this was occurring on one of my test systems, I haven't seen the error recently and have rebooted a few times while running pfSense+ 25.11.a.20250918.1857
Updated by Joe Cornelius about 1 month ago
I started getting this after I updated 10/05/2025 to 2.8.1. It shows up at every boot.
Screen shot:
Updated by Grid Solution 14 days ago
- File 2025-10-30_15h08_42.png 2025-10-30_15h08_42.png added
- File clipboard-202510301521-zamoy.png clipboard-202510301521-zamoy.png added
Updated by Grid Solution 14 days ago
- File clipboard-202510301533-pkyjb.png clipboard-202510301533-pkyjb.png added
- File clipboard-202510301534-o39xd.png added
Jim Pingle wrote in #note-1:
They may be a non-issue now on current dev snapshots. The latest upstream src merge brought in the new
certctlimplementation from upstream which was rewritten in C.That new implementation is much, much faster , sometimes finishing in under a second even on slow hardware, rather than taking several minutes.
Once on a current dev snapshot (as of today, at minimum) this particular race shouldn't be a concern. If it still happens, add a comment here with more info. ++
Completely fresh installation of pfSense Plus (25.07.1-RELEASE (amd64)) ------------- we couldn't even complete the full configuration.
Initially (during the first configuration steps), this problem did not occur and everything was fine. It has been present since restarting after adding interfaces.
Package repo not available.
This is a VM running on an ESXi8U3 host, but only pfSense is virtualized on this (dedicated), ----base HW ---Dell VEP4600 uCPE.
Updated by Jim Pingle 14 days ago
That's for the release, which was before the change I mentioned. The problem may be fixed on development snapshots of Plus 25.11 (and CE 2.9.0), I was requesting reports of it still happening on those development snapshots, we are already aware people are seeing it on the 25.07.1 release.
Updated by Grid Solution 14 days ago
- File clipboard-202510301650-7dgu7.png clipboard-202510301650-7dgu7.png added
- File clipboard-202510301650-2jp3u.png clipboard-202510301650-2jp3u.png added
- File clipboard-202510301650-7sidh.png clipboard-202510301650-7sidh.png added
- File clipboard-202510301651-hyphl.png clipboard-202510301651-hyphl.png added
- File clipboard-202510301656-fgzrw.png added
Jim Pingle wrote in #note-6:
That's for the release, which was before the change I mentioned. The problem may be fixed on development snapshots of Plus 25.11 (and CE 2.9.0), I was requesting reports of it still happening on those development snapshots, we are already aware people are seeing it on the 25.07.1 release. **
Okay Jim, but now, after restarting, it has completely prevented further configuration, so the problem is escalating. - no other kernel msg. ------ * pkg-static: Unable to open '/usr/local/etc/pkg/repos//pfSense.conf':No such file or directory*
Let me show you what we found:
(only in parentheses, not to mention that this pfSense unit has to be ready by the deadline + now we're starting almost from scratch with ESXi snapshot - unfortunately, we did not create one after the interface configuration - cca. 9 int.)
As I mentioned, there is no package repo, so I continued with this "Bug #15097" - this situation occurred after the restart....
https://redmine.pfsense.org/issues/15097
Here's a video. If you slow it down, you can see the system starting up:
https://mega.nz/file/R4lmhZ6A#dU1StMh_cHFz9LepxxwlJgpjg6-EnnxtSP1X4pPjj9Y
Now back to this point.......
Updated by Grid Solution 14 days ago
- File clipboard-202510301746-zcfty.png clipboard-202510301746-zcfty.png added
- File clipboard-202510301748-d4oxq.png clipboard-202510301748-d4oxq.png added
- File clipboard-202510301752-rqp0i.png clipboard-202510301752-rqp0i.png added
- File clipboard-202510301752-xil9g.png clipboard-202510301752-xil9g.png added
Jim, you know when this happens, we figured it out with a little research...
1. Restore the pfSense installation from a snapshot; this is the starting point, basic installation
In this state, there were 9 interfaces + VMware Tools installation. 4x X710 rNDC + 4x X710 rNDC + vmx0 for mgmt. (VMXNET3) + QAT 10VFs
-----The "pkg" files were here in the folder----
2. since it was necessary -- add an I350 with passthrough, - afterwards - REBOOT pfSense & ESXi host
3. after restarting , the "pkg" files are missing and * Bug #16341* occurs immediately.
So, the conclusion is that now adding an HW element (&/or change) deletes the files(??) from the /usr/local/etc/pfSense/pkg/repos folder. hmmmm....?
After restarting again, pfSense will no longer start up.
Updated by 
Updated by Jim Pingle 14 days ago
This site is not for support or diagnostic discussion and those issues are unlikely to be directly related to this problem. It may have a similar symptom, but it is not the same root cause.
For assistance in solving problems, please post on the Netgate Forum .
See Reporting Issues with pfSense Software for more information.
Updated by Jim Pingle about 15 hours ago
- Subject changed from Error notification and log message ``"Updating repositories metadata" returned error code 1`` at boot to Error notification and log message ``"Updating repositories metadata" returned error code 1`` at boot due to ``certctl`` race condition
- Status changed from Feedback to Closed
I still occasionally see this while running and not at boot, but infrequently and I can't repeat it on demand like I could with the error condition caused by the certctl race at boot. So there may still be an issue that gives this error message but the most frequent cause we could reliably repeat at boot time should be resolved.