Bug #4567
closed
Added by Denny Page over 9 years ago.
Updated over 9 years ago.
Affected Architecture:
amd64
Description
The ntopNG package does not install the necessary GeoIP files. Form syslog:
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoLiteCityv6.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoLiteCity.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
Mar 31 20:51:26 ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
- Status changed from New to Needs Patch
Last July the GeoIP distribution stopped including the data files, they must be manually fetched and put in place. I believe it's due to their license or distribution model changing. There is an update script that will grab some files but they are named differently. I'm not sure there's much of anything we can do for this at the moment. It may be possible to add a button or automate the script but it may need some sort of legal click-through. For now, the files must be manually fetched and renamed.
We're open to better suggestions/patches for this, so long as it doesn't introduce any other problems (legal or otherwise)
Perhaps they stopped distributing the files because they have introduced a new database format and don't want to double distribute. Speculation on my part.
I really can't see how it could be a license issue.
Maxmind's page for the legacy database is here: http://dev.maxmind.com/geoip/legacy/geolite/
It states that the database is distributed under a creative commons license It is free to use and further distribute, even for commercial use. The only requirement is appropriate attribution if you distribute.
The following script handles the missing files. It should be run monthly.
#!/bin/sh
mkdir -p /usr/pbi/ntopng-amd64/local/share/ntopng/httpdocs/geoip
cd /usr/pbi/ntopng-amd64/local/share/ntopng/httpdocs/geoip
fetch http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
fetch http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
fetch http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
fetch http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz
gunzip -f GeoLiteCity.dat.gz GeoLiteCityv6.dat.gz GeoIPASNum.dat.gz GeoIPASNumv6.dat.gz
- Status changed from Needs Patch to Assigned
- Assignee set to Jim Pingle
OK, great. I didn't have time to dig into all that yet. Looks promising. That script would need to be a bit more robust but with some attribution text in the GUI and an update button / cron job it could work out.
I'd prefer to place the GeoIP files in a more central spot and symlink them where ntopng wants them, that way if other packages need the files they could share an update mechanism rather than duplicating the data.
Actually there was already a script to do that but it was being called by the wrong name.
Try running /usr/pbi/ntopng-`uname -m`/bin/ntopng-geoipupdate.sh
I'll see if I can work in at least a button to update that.
- Status changed from Assigned to Feedback
- % Done changed from 0 to 100
Applied in changeset commit:a9d7f2034af6a28287ef85682a2fa1b6c8e0d05b.
Still receiving the following GeoIP related errors...
On startup:
ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
ntopng: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/local/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
When selecting the "Update GeoIP Data" button:
php-fpm78474: /pkg_edit.php: The command '/usr/bin/gzip -d '/usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNum.dat.gz'' returned exit code '1', the output was 'gzip: /usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNum.dat.gz: not in gzip format'
php-fpm78474: /pkg_edit.php: The command '/usr/bin/gzip -d '/usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNumv6.dat.gz'' returned exit code '1', the output was 'gzip: /usr/pbi/ntopng-amd64/share/ntopng/GeoIPASNumv6.dat.gz: not in gzip format'
ntopng: [Redis.cpp:67] ERROR: ntopng requires redis server to be up and running
ntopng: [Redis.cpp:68] ERROR: Please start it and try again or use -r
ntopng: [Redis.cpp:69] ERROR: to specify a redis server other than the default
Issue is still there is 2.2.2 package version 1.2.1 v0.5
/usr/pbi/ntopng-amd64/bin/ntopng-geoipupdate.sh doesn't correctly download file GeoIPASNum.dat.gz and GeoIPASNumv6.dat.gz
Witch is causing gzip faillure.
That /usr/pbi/ntopng-amd64/bin/ntopng-geoipupdate.sh is definitely not a shell script, plus it downloads corrupt crap. Best replaced with simple fetch, like in comment #3.
This should be fixed now.
- Status changed from Feedback to Resolved
Also available in: Atom
PDF
Updated by 
Updated by Kill Bill 
Updated by 
Updated by 
Updated by 
Updated by