Bug #4567
closed
ntopNG Geo files missing
100%
Description
The ntopNG package does not install the necessary GeoIP files. Form syslog:
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoLiteCityv6.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoLiteCity.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
Updated by Jim Pingle over 9 years ago
- Status changed from New to Needs Patch
Last July the GeoIP distribution stopped including the data files, they must be manually fetched and put in place. I believe it's due to their license or distribution model changing. There is an update script that will grab some files but they are named differently. I'm not sure there's much of anything we can do for this at the moment. It may be possible to add a button or automate the script but it may need some sort of legal click-through. For now, the files must be manually fetched and renamed.
We're open to better suggestions/patches for this, so long as it doesn't introduce any other problems (legal or otherwise)
Updated by Kill Bill over 9 years ago
Hmmm, the "lite" data files are CC BY-SA 3.0 so there should be no problem with distributing those as long as you credit those in the package description/GUI.
http://dev.maxmind.com/geoip/legacy/geolite/
http://dev.maxmind.com/geoip/geoip2/geolite2/
http://creativecommons.org/licenses/by-sa/3.0/
Updated by Denny Page over 9 years ago
Perhaps they stopped distributing the files because they have introduced a new database format and don't want to double distribute. Speculation on my part.
I really can't see how it could be a license issue.
Maxmind's page for the legacy database is here: http://dev.maxmind.com/geoip/legacy/geolite/
It states that the database is distributed under a creative commons license It is free to use and further distribute, even for commercial use. The only requirement is appropriate attribution if you distribute.
The following script handles the missing files. It should be run monthly.
#!/bin/sh mkdir -p /usr/pbi/ntopng-amd64/local/share/ntopng/httpdocs/geoip cd /usr/pbi/ntopng-amd64/local/share/ntopng/httpdocs/geoip fetch http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz fetch http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz fetch http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz fetch http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz gunzip -f GeoLiteCity.dat.gz GeoLiteCityv6.dat.gz GeoIPASNum.dat.gz GeoIPASNumv6.dat.gz
Updated by Jim Pingle over 9 years ago
- Status changed from Needs Patch to Assigned
- Assignee set to Jim Pingle
OK, great. I didn't have time to dig into all that yet. Looks promising. That script would need to be a bit more robust but with some attribution text in the GUI and an update button / cron job it could work out.
I'd prefer to place the GeoIP files in a more central spot and symlink them where ntopng wants them, that way if other packages need the files they could share an update mechanism rather than duplicating the data.
Updated by Jim Pingle over 9 years ago
Actually there was already a script to do that but it was being called by the wrong name.
Try running /usr/pbi/ntopng-`uname -m`/bin/ntopng-geoipupdate.sh
I'll see if I can work in at least a button to update that.
Updated by Jim Pingle over 9 years ago
- Status changed from Assigned to Feedback
- % Done changed from 0 to 100
Applied in changeset commit:a9d7f2034af6a28287ef85682a2fa1b6c8e0d05b.
Updated by Jos van de Ven over 9 years ago
Still some problems. See this topic were I made some remarks:
https://forum.pfsense.org/index.php?topic=91909.0
Maybe the files are damaged during download.
Manually downloading and replacing them works ok.
Updated by Denny Page over 9 years ago
Still receiving the following GeoIP related errors...
On startup:
ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
When selecting the "Update GeoIP Data" button:
php-fpm78474: /pkg_edit.php: The command '/usr/bin/gzip -d '/usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNum.dat.gz'' returned exit code '1', the output was 'gzip: /usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNum.dat.gz: not in gzip format'
php-fpm78474: /pkg_edit.php: The command '/usr/bin/gzip -d '/usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNumv6.dat.gz'' returned exit code '1', the output was 'gzip: /usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNumv6.dat.gz: not in gzip format'
ntopng: [Redis.cpp:67] ERROR: ntopng requires redis server to be up and running
ntopng: [Redis.cpp:68] ERROR: Please start it and try again or use -r
ntopng: [Redis.cpp:69] ERROR: to specify a redis server other than the default
Updated by Basile Caillens over 9 years ago
Issue is still there is 2.2.2 package version 1.2.1 v0.5
/usr/pbi/ntopng-amd64/bin/ntopng-geoipupdate.sh doesn't correctly download file GeoIPASNum.dat.gz and GeoIPASNumv6.dat.gz
Witch is causing gzip faillure.
Updated by Kill Bill over 9 years ago
That /usr/pbi/ntopng-amd64/bin/ntopng-geoipupdate.sh is definitely not a shell script, plus it downloads corrupt crap. Best replaced with simple fetch, like in comment #3.
Updated by Kill Bill over 9 years ago
Kill Bill wrote:
That /usr/pbi/ntopng-amd64/bin/ntopng-geoipupdate.sh is definitely not a shell script, plus it downloads corrupt crap. Best replaced with simple fetch, like in comment #3.
Updated by