Regression #14452
closed
Prometheus node_exporter generates errors with the default config
100%
Description
Installing and enabling the node exporter generates errors whenever it is queried.
Jun 5 14:47:50 node_exporter 29215 ts=2023-06-05T13:47:50.492Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=1.3109e-05 err="cannot allocate memory"
This can be triggered by querying it with curl:
curl http://172.21.16.246:9100/metrics
Tested:
23.05-REL
node_exporter-1.5.0_4
pfSense-pkg-node_exporter-0.18.1_3
This was also failing in 23.01.
Files
| node_exporter_metrics.txt (48.5 KB) node_exporter_metrics.txt | |||
| grafana.jpg (92.4 KB) grafana.jpg | grafana |
Updated by Steve Wheeler over 1 year ago
Full metrics output attached.
Note that both 'uname' and 'os' report failure:
# TYPE node_scrape_collector_success gauge
node_scrape_collector_success{collector="boottime"} 1
node_scrape_collector_success{collector="cpu"} 1
node_scrape_collector_success{collector="exec"} 1
node_scrape_collector_success{collector="filesystem"} 1
node_scrape_collector_success{collector="loadavg"} 1
node_scrape_collector_success{collector="meminfo"} 1
node_scrape_collector_success{collector="netdev"} 1
node_scrape_collector_success{collector="os"} 0
node_scrape_collector_success{collector="textfile"} 1
node_scrape_collector_success{collector="time"} 1
node_scrape_collector_success{collector="uname"} 0
node_scrape_collector_success{collector="zfs"} 1
Updated by Jim Pingle over 1 year ago
- Target version deleted (
2.7.0) - Plus Target Version deleted (
23.09)
Updated by Steven Hostetler over 1 year ago
I stumbled upon this today. This PR [[https://github.com/prometheus/node_exporter/pull/2584]] may provide additional context on the fix needed.
Updated by Steve Wheeler over 1 year ago
Confirmed this also affects the 2.7 package:
https://forum.netgate.com/topic/180575/node_exporter-is-not-working-properly-on-23-05/5
Updated by Steve Wheeler about 1 year ago
A fix for this issue appears to have been merged upstream:
https://github.com/prometheus/node_exporter/issues/2593
But that bug is still open at this time.
Updated by Craig Coonrad about 1 year ago
Still not working with pfSense 23.09 and node_exporter-1.5.0_6. I tested FreeBSD 14.0-RELEASE w/node_exporter-1.6.1. That works!
Updated by Craig Coonrad about 1 year ago
I enabled FreeBSD repos on 23.09 and installed node_exporter-1.6.1. Same error, does not resolve the issue.
Updated by Logan Marchione about 1 year ago
I'm still seeing this on CE 2.7.2 with node_exporter 0.18.1_3 (upstream node_exporter-1.6.1) installed. Note that the upstream bug was fixed though, so I'm not sure what's causing this.
https://github.com/prometheus/node_exporter/issues/2593
Updated by Craig Coonrad 12 months ago
Tested 24.03 development snapshot. Error persists.
2024-01-13T15:38:42.627092-08:00 pf100.local node_exporter 98035 - - ts=2024-01-13T23:38:42.627Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=3.736e-06 err="cannot allocate memory"
Updated by Craig Coonrad 10 months ago
The FreeBSD repos contain a newer version of node_exporter: node_exporter-1.6.1_2
I downloaded this pkg copied it to pfSense, installed and started node_exporter successfully.
Same error persists. Something in pfSense is preventing this from working.
Updated by Craig Coonrad 4 months ago
- File grafana.jpg grafana.jpg added
I copied over `node_exporter-1.8.2.pkg` from the FreeBSD repos. Installed, no errors. It works!
Updated by Craig Coonrad about 1 month ago
This is resolved with 24.11-RELEASE. I don't have the ability to change the status on this one.
node_exporter-1.8.2 Prometheus exporter for machine metrics
Updated by Jim Pingle about 1 month ago
- Status changed from New to Resolved
- % Done changed from 0 to 100