Bug #15756
open
NTOPNG db sits in ramdisk on pfsense, filling it up and crashing pfsense
0%
Description
Environment:
OS name: PFSENSE
OS version: 2.7.2-RELEASE
Architecture: amd64
ntopng version/revision: ntopng Community v.5.6.240304 rev.0 (FreeBSD 14.0)
What happened:
by default, ntopng puts it's db files in /var/db/ntopng
when ramdisk is enabled in pfsense, the files are now sitting on the ramdisk and it can be quickly filled up, resulting in various services in pfsense crashing.
How did you reproduce it?
ntopng on pfsense \w ramdisk enabled.
Debug Information:
/var/db: du -sh * | sort -hr | head
317M ntopng
12M rrd
8.1M pkg
748K captiveportaldn.rules
560K cpelements
i think ntopng is using redis ( .rdb ) which loads to the memory, so i think this shouldn't be in ramdisk.
changing the default location from:
/var/db/ntopng
to:
/var/db/pkg/ntopng
should do the trick.
i found this commit is where the problem begins:
https://www.freshports.org/commit.php?category=net&port=ntopng&files=yes&message_id=201812222105.wBML51Ma094362@repo.freebsd.org
you can see the data dir used to sit on the hdd.
---
posted the issue here as well:
https://github.com/ntop/ntopng/issues/8735
Updated by Danilo Zrenjanin 11 months ago
- Status changed from New to Confirmed
I was able to replicate the observed behavior.
Tested against:
pfSense Plust 24.03, ntopng version 5.6.0
Please be aware that if the entire /var directory is stored in RAM when the ramdisk feature is enabled on pfSense, relocating the ntopng data to a subdirectory within /var, such as /var/db/pkg/ntopng, will not resolve the issue, as it will still reside in RAM.
Updated by Denny Page 4 days ago
The general purpose of using ramdisk is to reduce writes to physical storage. ntopng writes a lot, and working around the ramdisk to place ntopng's storage files on the physical storage would be very bad.
The ntopng package requires an ssd/hdd/nvme drive. It isn't expected to be compatible with use of ramdisk.
Updated by cody wasser 4 days ago
Denny Page wrote in #note-2:
The general purpose of using ramdisk is to reduce writes to physical storage. ntopng writes a lot, and working around the ramdisk to place ntopng's storage files on the physical storage would be very bad.
The ntopng package requires an ssd/hdd/nvme drive. It isn't expected to be compatible with use of ramdisk.
Redis can do caching. Therefore ssd shouldn't be fully required. Likewise an oversized ssd can overcome any write wear concerns. Newer ssds also have significantly more life, so wear concerns should be disregarded.
Random catastrophic failure in potential production environments is concerning.
Updated by Denny Page 4 days ago
Redis caching really isn't pertinent to the issue. ntopng simply writes a lot, and the majority of the writes are to rrd and sqlite files rather than to Redis. These files are also the big consumers of disk space used by ntopng. The Redis database is only 1-2% of the requirement. No way around this without a complete redesign of ntopng itself.
The documentation explicitly states that the package requires ssd/hdd.
Updated by cody wasser 4 days ago
Denny Page wrote in #note-4:
Redis caching really isn't pertinent to the issue. ntopng simply writes a lot, and the majority of the writes are to rrd and sqlite files rather than to Redis. These files are also the big consumers of disk space used by ntopng. The Redis database is only 1-2% of the requirement. No way around this without a complete redesign of ntopng itself.
The documentation explicitly states that the package requires ssd/hdd.
What's the point of even using redis if the files just sit on the ramdisk? Redis is already in the ram.
I've never used redis myself, but aren't the writes based on snapshots? So... just do snapshots less often and p000f... less writes.
And how can a ssd be required if the files just sit in ramdisk...
Redis db should not be on a ram disk. Period.
Updated by Denny Page 4 days ago
As previously noted, the Redis database is only 1-2% of the requirement. ntopng uses Redis to store frequently read data, such as configuration information. There is only one Redis database file, 100-200 KB in size. Comparatively tiny. Most of the actual accumulated runtime information for ntopng is actually stored in rrd and sqlite files. 10s to 100s of these, totaling 10-20+ MB in size depending on how ntopng is configured. This is where the writes happen.
You may not agree with the design choices that the architects of ntopng have made, but it is what it is. It's their product and their choices to make. Changing this would require a complete redesign of ntopng. No way around that.
The short version: If you are using ramdisks on pfSense, you should not attempt to run ntopng.
Updated by cody wasser 4 days ago
Denny Page wrote in #note-6:
As previously noted, the Redis database is only 1-2% of the requirement. ntopng uses Redis to store frequently read data, such as configuration information. There is only one Redis database file, 100-200 KB in size. Comparatively tiny. Most of the actual accumulated runtime information for ntopng is actually stored in rrd and sqlite files. 10s to 100s of these, totaling 10-20+ MB in size depending on how ntopng is configured. This is where the writes happen.
You may not agree with the design choices that the architects of ntopng have made, but it is what it is. It's their product and their choices to make. Changing this would require a complete redesign of ntopng. No way around that.
The short version: If you are using ramdisks on pfSense, you should not attempt to run ntopng.
or... you could... you know... move the db off the ramdisk...
and stop having two copies of the same data in the memory.
and the redis db sure as hell easily gets into the GB range.... cause it fills up the ramdisk and crashes everything.
16GB ram, 32GB ram... you name it.