Project

General

Profile

Actions
Copy link

Regression #14452

open

Prometheus node_exporter generates errors with the default config

Added by Steve Wheeler 11 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
node_exporter
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
2.7.0
Affected Plus Version:
23.05
Affected Architecture:

Description

Installing and enabling the node exporter generates errors whenever it is queried. 

Jun 5 14:47:50     node_exporter     29215     ts=2023-06-05T13:47:50.492Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=1.3109e-05 err="cannot allocate memory"

This can be triggered by querying it with curl:

curl http://172.21.16.246:9100/metrics

Tested:
23.05-REL
node_exporter-1.5.0_4
pfSense-pkg-node_exporter-0.18.1_3

This was also failing in 23.01.

Files

node_exporter_metrics.txt (48.5 KB) node_exporter_metrics.txt Steve Wheeler, 06/05/2023 01:59 PM
Actions
Copy link
 #1

Updated by Steve Wheeler 11 months ago

Full metrics output attached.

Note that both 'uname' and 'os' report failure:

# TYPE node_scrape_collector_success gauge
node_scrape_collector_success{collector="boottime"} 1
node_scrape_collector_success{collector="cpu"} 1
node_scrape_collector_success{collector="exec"} 1
node_scrape_collector_success{collector="filesystem"} 1
node_scrape_collector_success{collector="loadavg"} 1
node_scrape_collector_success{collector="meminfo"} 1
node_scrape_collector_success{collector="netdev"} 1
node_scrape_collector_success{collector="os"} 0
node_scrape_collector_success{collector="textfile"} 1
node_scrape_collector_success{collector="time"} 1
node_scrape_collector_success{collector="uname"} 0
node_scrape_collector_success{collector="zfs"} 1

Actions
Copy link
 #2

Updated by Jim Pingle 11 months ago

  • Target version deleted (2.7.0)
  • Plus Target Version deleted (23.09)
Actions
Copy link
 #3

Updated by Steven Hostetler 10 months ago

I stumbled upon this today. This PR [[https://github.com/prometheus/node_exporter/pull/2584]] may provide additional context on the fix needed.

Actions
Copy link
 #5

Updated by Steve Wheeler 7 months ago

A fix for this issue appears to have been merged upstream:
https://github.com/prometheus/node_exporter/issues/2593

But that bug is still open at this time.

Actions
Copy link
 #6

Updated by Craig Coonrad 5 months ago

Still not working with pfSense 23.09 and node_exporter-1.5.0_6. I tested FreeBSD 14.0-RELEASE w/node_exporter-1.6.1. That works!

Actions
Copy link
 #7

Updated by Craig Coonrad 5 months ago

I enabled FreeBSD repos on 23.09 and installed node_exporter-1.6.1. Same error, does not resolve the issue.

Actions
Copy link
 #8

Updated by Logan Marchione 4 months ago

I'm still seeing this on CE 2.7.2 with node_exporter 0.18.1_3 (upstream node_exporter-1.6.1) installed. Note that the upstream bug was fixed though, so I'm not sure what's causing this.
https://github.com/prometheus/node_exporter/issues/2593

Actions
Copy link
 #9

Updated by Craig Coonrad 3 months ago

Tested 24.03 development snapshot. Error persists.

2024-01-13T15:38:42.627092-08:00 pf100.local node_exporter 98035 - - ts=2024-01-13T23:38:42.627Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=3.736e-06 err="cannot allocate memory"
Actions
Copy link
 #10

Updated by Craig Coonrad about 2 months ago

The FreeBSD repos contain a newer version of node_exporter: node_exporter-1.6.1_2
I downloaded this pkg copied it to pfSense, installed and started node_exporter successfully.
Same error persists. Something in pfSense is preventing this from working.

Actions
Copy link

Also available in: Atom PDF