Bug #3334
closedStatus/Traffic Graph isn't IPv6 ready
Added by Armin Fasold over 10 years ago. Updated over 3 years ago.
100%
Description
In /usr/local/www/bandwith_by_ip.php, which is called per AJAX/XHR to provide the data for the right table in "Traffic Graph" page (status_graph.php), the program "rate" is called to acquire that data.
Looking into the source code of "rate", to see what all the given parameters do, I noticed that it uses u_int32_t [0] for storing the IP addresses. This results in a silent fail if called with an IPv6 netmask and no listing of IPv6 traffic sources.
[0]: https://fossies.org/dox/rate-0.9/rate__abusers_8c_source.html#l00062
Files
bandwidth_by_ip.php (6.31 KB) bandwidth_by_ip.php | Joshua Sign, 01/18/2019 06:17 PM | ||
iftop_parse.awk (986 Bytes) iftop_parse.awk | Joshua Sign, 01/18/2019 06:17 PM | ||
iftop_parser.sh (1.07 KB) iftop_parser.sh | Joshua Sign, 01/18/2019 06:17 PM | ||
pfSense_status_graph.patch (7.95 KB) pfSense_status_graph.patch | Joshua Sign, 01/19/2019 12:24 AM | ||
Capture iftop.PNG (24.1 KB) Capture iftop.PNG | Daryl Morse, 01/20/2019 05:47 PM | ||
Capture iftop hn0.PNG (22.9 KB) Capture iftop hn0.PNG | Daryl Morse, 01/20/2019 05:55 PM | ||
Capture iftop hn1.PNG (22.9 KB) Capture iftop hn1.PNG | Daryl Morse, 01/20/2019 05:56 PM | ||
iftop status graph.PNG (77.6 KB) iftop status graph.PNG | Daryl Morse, 01/20/2019 06:02 PM | ||
iftop_parse.awk.gz (308 Bytes) iftop_parse.awk.gz | Joshua Sign, 01/21/2019 04:12 AM |
Updated by Chris Buechler almost 9 years ago
- Status changed from New to Confirmed
Updated by Jim Pingle over 7 years ago
- Affected Version changed from 2.1-IPv6 to All
Updated by Pim Pish over 7 years ago
When will this be resolved? This is a really old bug.
Updated by Jim Pingle over 7 years ago
The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.
I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.
Updated by Daryl Morse over 5 years ago
Jim Pingle wrote:
The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.
I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.
I looked at the man page for rate. It appears to have been written in 2003 by Mateusz Golicz <mteg@jaszczur.org>, aka github.com/mteg. There doesn't appear to be a repository for rate, so you have to think he abandoned it and moved on. I tried emailing him, but he didn't reply. In the absence of the author, the code isn't going to fix itself.
Updated by Joshua Sign over 5 years ago
Jim Pingle wrote:
I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.
Me too, I am looking for a way to replace rate by iftop.
Since iftop-1.0pre3 (2014-01-01), a text output mode was added, so it is possible to get an output like rate.
An awk script is good enough for that, i just test something like that :
# iftop -nNb -i eth0 -s 1 -o 2s -F 10.0.0.0/8 -t | awk -f iftop_parse.awk 10.0.50.233 3,27Kb 3,27Kb 1,87Kb 1,87Kb 216.58.198.206 1,87Kb 1,87Kb 3,27Kb 3,27Kb 10.0.50.233 1,59Kb 1,59Kb 2,18Kb 2,18Kb 216.58.204.101 2,18Kb 2,18Kb 1,59Kb 1,59Kb 10.0.48.251 896b 896b 0b 0b 224.0.0.18 0b 0b 896b 896b 10.0.51.217 660b 660b 0b 0b 239.255.255.250 0b 0b 660b 660b 10.0.51.9 644b 644b 0b 0b 239.255.255.250 0b 0b 644b 644b
The awk script is very simple :
BEGIN { FS = " " numlist = 0 } { if ( numlist == 1 && $1 == "--------------------------------------------------------------------------------------------" ) { numlist = 2 next } if ( numlist == 0 && $1 == "--------------------------------------------------------------------------------------------" ) { numlist = 1 next } if ( numlist == 1 ) { if ( $0 ~ "=>") { SENDER = $2 STX = $4 STXAVG = $5 getline RECEIVER = $1 RTX = $3 RTXAVG = $4 print SENDER" "STX" "STXAVG" "RTX" "RTXAVG print RECEIVER" "RTX" "RTXAVG" "STX" "STXAVG } next } if ( $1 == "============================================================================================" ) { numlist = 0 next } } END { }
I also notice no errors on disconnected interfaces or when there is no trafic.
(version 1.0pre4 of iftop, on 2.4.4 pfsense release)
Please tell me if I can help, or if I just waste our time.
Updated by Jim Pingle over 5 years ago
That does help a bit. It would be even better if iftop had an output mode like libxo where it would be trivial to parse the output, but I'm not sure how the project author(s) would feel about that.
Can you create a PR that changes rate to iftop? No need to worry about the dependency issues at first (adding iftop to base, that is), since we'd need to run it through manual tests before it gets merged anyhow.
Updated by Joshua Sign over 5 years ago
ok i will work on it and create a PR to change rate by iftop as soon as it works
(normaly it should be ok on sunday or before)
Updated by Joshua Sign over 5 years ago
Hi Jim,
FYI, I just finish some tests : it seems to works as expected.
All we need to test is :
This awk script 'iftop_parser.awk' :
BEGIN { FS = " " numlist = 0 nblines = 15 printf "\n\n" } { if ( numlist == 1 && $1 == "--------------------------------------------------------------------------------------------" ) { exit } if ( numlist == 0 && $1 == "--------------------------------------------------------------------------------------------" ) { numlist = 1 next } if ( numlist == 1 ) { if ( $0 ~ "=>" && nblines > 0 ) { SENDER = $2 STX = pfFormat($4) getline RECEIVER = $1 RTX = pfFormat($3) printf "%s;%s;%s\n", SENDER,RTX,STX printf "%s;%s;%s\n", RECEIVER,STX,RTX nblines-- if ( nblines < 1 ) { exit } } next } } END { } function pfFormat(str) { sub("b","",str) return str }
And these changes in /usr/local/www/bandwidth_by_ip.php :
- replace line 112 by the code bellow :
$_grb = exec("/usr/bin/timeout 5 /usr/local/sbin/iftop -nNb -i {$real_interface} -s 1 -o 2s -t | /usr/bin/awk -f /root/iftop_parse.awk", $listedIPs); // Functions format // TODO : replace by existing function or put this in pfsense-utils.inc function GMK2bits($val) { $val = strtolower($val); if ( strpos($val,"g") !== false ) { $val = str_replace("g","",$val) * 1000 * 1000 * 1000; } else if ( strpos($val,"m") !== false ) { $val = str_replace("m","",$val) * 1000 * 1000; } else if ( strpos($val,"k") !== false ) { $val = str_replace("k","",$val) * 1000; } return $val*1; } function bits2GMK($val) { $units = array ( 0 => "", 1 => "K", 2 => "M", 3 => "G" ); $u=0; while ($val > 1000) { $val = $val/1000; $u++; } return round($val,2).$units[$u]; } // END of Functions format // order and group by foreach ($listedIPs as $k => $line){ if ($line != "") { $arrLine = explode (";", $line); $ip = $arrLine[0]; $in = GMK2bits($arrLine[1]); $out = GMK2bits($arrLine[2]); if ( isset($arr_in[$ip]) ) { $arr_in[$ip] += $in; $arr_out[$ip] += $out; } else { $arr_in[$ip] = $in; $arr_out[$ip] = $out; } } } if ($sort == "out") { arsort($arr_out,SORT_NUMERIC); $arrIP = array_keys($arr_out); } else { arsort($arr_in,SORT_NUMERIC); $arrIP = array_keys($arr_in); } unset($listedIPs); $listedIPs[] = ""; $listedIPs[] = ""; foreach ($arrIP as $k => $ip) { $listedIPs[] = $ip.";".bits2GMK($arr_in[$ip]).";".bits2GMK($arr_out[$ip]); }
- change separator ":" by ";" on the line 180
(it was line 122 before adding the code above) :
$infoarray = explode (";", $bandwidthinfo);
Unfortunally i don't use IPV6, so i can't test this part.
But it was necessary to change this separator to avoid IPV6 confict.
The exec of iftop may take between 1 or 2 seconds most of time.
But i experienced some cases where iftop wait more than 7 seconds before giving a result (or no result because not enough traffic).
Maybe thats what you experienced when you was talking about issues, so i add 'timeout 5' to the command line to ensure that iftop will not stay stucked and overload the system.
Yes it is a crappy solution, but it is not the best we can do.
This is even more usefull if there are many users on the status_graph.php : multiples concurrent processes are created.
We can look for another way to avoid multiples concurrent processes and increase performances. (i got some ideas)
Please tell me :
- if i miss something (i don't use '-F' or '-G' flag, maybe needed for ipv6?)
- if the order of IN/OUT bis are corrects... they seems to me
- where is the best place for the awk script ? (/etc ? /usr/local/bin ?)
- is there an equivalent of GMK2bits / bits2GMK in pfSense ? If there is not, should i add them to pfsense-utils.inc ?
- do we try to improve performances about concurrent processes if many users are on the same graph ?
It also should be possible to create a PR by adding iftop as an additional feature instead of replacing rate. (iftop must be added by pkg install)
This could be better to avoid any revert if somes have problems with iftop.
Otherwise, the old unused code can be removed.
I wait for your instructions and feedbacks if some of us can test it.
Updated by Daryl Morse over 5 years ago
Joshua Sign wrote:
Unfortunally i don't use IPV6, so i can't test this part.
I have IPv6 so I would be happy to test when you have something ready. Can you provide it as a patch based on the latest development snapshot?
Updated by Joshua Sign over 5 years ago
yes it will be possible soon.
I just wrote this script to avoid process concurrent creation when many users are on the same graph.
My tests looks good.
I'll put it in the PR.
#!/bin/sh # Usage if [ $# -ne 1 ]; then echo "Usage : $0 iface" exit fi # files paths pid_file=/var/run/iftop_$1.pid cache_file=/var/db/iftop_$1.log awk_script=/root/iftop_parse.awk # Binaries paths DATE=/bin/date STAT=/usr/bin/stat CUT=/usr/bin/cut PS=/bin/ps GREP=/usr/bin/grep CAT=/bin/cat RM=/bin/rm IFTOP=/usr/local/sbin/iftop AWK=/usr/bin/awk # test if pid file exist if [ -f $pid_file ]; then # check how old is the file curTS=`$DATE +%s` pidTS=`$STAT -r $pid_file | $CUT -d " " -f 10` # if more than 10 seconds, # it must be a dead pid file (process killed?) # or a stucked process that we should kill if [ $(( curTS - pidTS )) -gt 10 ]; then oldPID=`$CAT $pid_file` # test if pid still exist run=`$PS -p $oldPID | $GREP -F '$0'` if [ "$run" != "" ]; then kill -9 $oldPID fi $RM $pid_file $RM $cache_file exit fi if [ -f $cache_file ]; then $CAT $cache_file fi exit fi echo -n $$ > $pid_file $IFTOP -nNb -i $1 -s 1 -o 2s -t 2>> /dev/null | $AWK -f $awk_script > $cache_file.1 $CAT $cache_file.1 > $cache_file $CAT $cache_file $RM $pid_file
Then just change the exec in /usr/local/www/bandwidth_by_ip.php by
$_grb = exec("/root/iftop_parser.sh {$real_interface}", $listedIPs);
I'll create a PR soon.
Updated by Joshua Sign over 5 years ago
- File bandwidth_by_ip.php bandwidth_by_ip.php added
- File iftop_parse.awk iftop_parse.awk added
- File iftop_parser.sh iftop_parser.sh added
here are the files you need to easely test, it is faster thant the PR
just put the two scripts into the root directory, and replace the /usr/local/www/bandwidth_by_ip.php
Updated by Joshua Sign over 5 years ago
Here is the patch
/usr/local/bin/iftop_parser.sh must have +x
Updated by Joshua Sign over 5 years ago
Updated by Daryl Morse over 5 years ago
Joshua Sign wrote:
I installed this patch on the most recent snapshot, built on Sat Jan 19 05:32:33 EST 2019. I used http://ipv6-test.com/speedtest/ to test it, because there are separate speedtests for ipv4 and ipv6, making it easy to see traffic for both protocols. I tried both WAN and LAN, but the patch doesn't show anything for ipv6. For ipv4, I did not see any difference with the patch applied or not applied. I did not reboot after applying the patch. Should I have?
Updated by Joshua Sign over 5 years ago
can you chexk over console if iftop shows you some IPV6 adresses just by : `iftop -n` ?
as far as i didn't have any ipv6, it will be difficult to test this part.
Updated by Daryl Morse over 5 years ago
Joshua Sign wrote:
can you chexk over console if iftop shows you some IPV6 adresses just by : `iftop -n` ?
as far as i didn't have any ipv6, it will be difficult to test this part.
Sorry mate. iftop -n shows ipv4 and ipv6 traffic, but there is no indication of it in the list of hosts or ip addresses. iftop -n display formats ipv4 and ipv6 traffic exactly the same, except for the addresses (obviously). If you use iftop without -n, it's not possible to tell the difference between ipv4 and ipv6.
Updated by Joshua Sign over 5 years ago
ok,
to debbug it you can check if there is any ip6 in this output :
/usr/local/bin/iftop_parser.sh eth0
if there is no ipv6 in the output, please check the output of this command line :
iftop -nNb -i eth0 -s 1 -o 2s -t
You must find ipv6 here, so give me the output for example, the awk script may be wrong.
If there is no ipv6 here, i dont understand why.
Updated by Daryl Morse over 5 years ago
Joshua Sign wrote:
ok,
to debbug it you can check if there is any ip6 in this output :
[...]if there is no ipv6 in the output, please check the output of this command line :
[...]You must find ipv6 here, so give me the output for example, the awk script may be wrong.
If there is no ipv6 here, i dont understand why.
My system does not have a device called eth0. It's using hn1 for the wan and hn0 for the lan. I'd say that's part of the problem.
I got permission denied when I tried to run the script from the console shell.
When I executed the iftop command, there was ipv4 and ipv6 output for both hn0 and hn1. Except for the address format, the output is indistinguishable for ipv4 and ipv6. It's a shame there isn't a way to select ipv4 or ipv6 because if it resolves the hostnames, there is no way to tell them apart.
Updated by Daryl Morse over 5 years ago
just a thought, if you don't have ipv6, you could set up a tunnel with hurricane electric. It's free, it works very well, and it's easy to set up.
Updated by Joshua Sign over 5 years ago
Daryl Morse wrote:
I got permission denied when I tried to run the script from the console shell.
please check if the script got +x permission,
or give it anyway, type this in console :
chmod 755 /usr/local/bin/iftop_parser.sh
obiously, eth0 was un axemple,
you have to change it according your system.
in the iftop mode, the informations are grabbed by this command for your WAN card :
iftop -nNb -i hn1 -s 1 -o 2s -t
and this one for you LAN card :
iftop -nNb -i hn0 -s 1 -o 2s -t
Does it gives you ipv6 results ?
Updated by Daryl Morse over 5 years ago
- File Capture iftop.PNG Capture iftop.PNG added
- File Capture iftop hn0.PNG Capture iftop hn0.PNG added
- File Capture iftop hn1.PNG Capture iftop hn1.PNG added
Joshua Sign wrote:
Daryl Morse wrote:
I got permission denied when I tried to run the script from the console shell.
please check if the script got +x permission,
or give it anyway, type this in console :
[...]obiously, eth0 was un axemple,
you have to change it according your system.in the iftop mode, the informations are grabbed by this command for your WAN card :
[...]and this one for you LAN card :
[...]Does it gives you ipv6 results ?
I attached a screen capture of the output of the script. There are some errors.
I already told you the output of iftop shows ipv4 and ipv6 and they are identical except for the addresses. A couple of screen captures attached.
Updated by Daryl Morse over 5 years ago
- File iftop status graph.PNG iftop status graph.PNG added
Another screen capture from the status graph.
Updated by Joshua Sign over 5 years ago
ok,
the first file "File Capture iftop.PNG" show that there is a problem with the awk script.
This script is less than 40 lines and errors are about lines 43,76.
Can you check if this is the same file than this :
https://github.com/pfsense/pfsense/blob/8e73bf65ab51822b438872f13bdf7475ac370365/src/usr/local/bin/iftop_parse.awk
Updated by Daryl Morse over 5 years ago
Joshua Sign wrote:
ok,
the first file "File Capture iftop.PNG" show that there is a problem with the awk script.
This script is less than 40 lines and errors are about lines 43,76.Can you check if this is the same file than this :
https://github.com/pfsense/pfsense/blob/8e73bf65ab51822b438872f13bdf7475ac370365/src/usr/local/bin/iftop_parse.awk
I installed the script using the patch, so it must be the same.
Updated by Joshua Sign over 5 years ago
- File iftop_parse.awk.gz iftop_parse.awk.gz added
you are certainly in an issue with bads crlf in the awk script.
please update by this one, i gzip it to avoid any copy/paste problem.
fetch it in your pfsense and use this command line to replace the old one :
fetch https://redmine.pfsense.org/attachments/download/2695/iftop_parse.awk.gz gzcat iftop_parse.awk.gz > /usr/local/bin/iftop_parse.awk
Updated by Jim Pingle over 5 years ago
With some small modifications, it does work. See my comments on the PR (and future discussion should happen on the PR rather than here)
Updated by Daryl Morse over 5 years ago
Jim Pingle wrote:
The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.
I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.
As I mentioned, I emailed the author of rate. He replied this morning and confirmed that he is no longer developing rate. He offered up the source code, if you don't have it already. Let me know if you are interested in the details.
Updated by Jim Pingle over 5 years ago
Daryl Morse wrote:
As I mentioned, I emailed the author of rate. He replied this morning and confirmed that he is no longer developing rate. He offered up the source code, if you don't have it already. Let me know if you are interested in the details.
We already have access to the source code for rate, or it couldn't be compiled into our images. Adapting the GUI to use iftop is almost certainly easier than adapting rate to handle IPv6.
iftop is still being actively developed. If we attempt to adapt rate, then we'd be taking on all the technical debt of maintaining it going forward, on top of the work to make it function now.
Updated by Jim Pingle almost 5 years ago
- Status changed from Confirmed to Pull Request Review
- Target version set to 2.5.0
Updated by Renato Botelho over 4 years ago
- Status changed from Pull Request Review to Feedback
- Assignee set to Renato Botelho
- % Done changed from 0 to 100
Pull request has been merged. Thanks!
Updated by Viktor Gurov over 4 years ago
- Status changed from Feedback to Resolved
tested on 2.5.0.a.20200212.1057
works as expected
Updated by Jim Pingle over 4 years ago
- Status changed from Resolved to In Progress
There is still a problem or two here.
The iftop binary is not present unless you manually install the pkg. Also the pfSense package (net-mgmt/pfSense-pkg-iftop) needs to be removed since net-mgmt/iftop can now be set as a dependency of the security/pfSense meta pkg.
Updated by Jim Pingle over 4 years ago
Also it looks like there is an odd condition where the first time you switch to iftop, it doesn't want to display anything. But if you save it as default, navigate away for a few seconds, then go back, it shows data. I'd call it a fluke but it happened on two systems so far.
That could be moved to its own separate (potentially even post-2.5.0) issue if it persists after the pkg issue is sorted.
Updated by Renato Botelho almost 4 years ago
Jim Pingle wrote:
There is still a problem or two here.
The iftop binary is not present unless you manually install the pkg. Also the pfSense package (net-mgmt/pfSense-pkg-iftop) needs to be removed since net-mgmt/iftop can now be set as a dependency of the security/pfSense meta pkg.
It's done since Feb
Updated by Renato Botelho almost 4 years ago
- Status changed from In Progress to Feedback
Jim Pingle wrote:
Also it looks like there is an odd condition where the first time you switch to iftop, it doesn't want to display anything. But if you save it as default, navigate away for a few seconds, then go back, it shows data. I'd call it a fluke but it happened on two systems so far.
That could be moved to its own separate (potentially even post-2.5.0) issue if it persists after the pkg issue is sorted.
I couldn't reproduce it on recent snapshots. As you said, this ticket is now completed and we can open a new one for specific bugs.
Updated by Jim Pingle almost 4 years ago
- Status changed from Feedback to Resolved
I'd say it's working well enough for now. I tried it on a few more systems and it's OK, even on arm.
Updated by Jim Pingle almost 4 years ago
- Category changed from Web Interface to Traffic Graphs
Updated by Pim Pish over 3 years ago
I've upgraded to pfSense 2.5 RC but still there are no IPv6 addresses shown in the traffic graph for me. What am I doing wrong?
Updated by Jim Pingle over 3 years ago
It doesn't change to the new mode automatically. You have to change it from rate to iftop..
- Status > Traffic Graph
- Set Mode to iftop
- Click Save
Make sure it's set to display IP Addresses, not hostnames, or you may not notice when it does report IPv6 traffic.
You might have to navigate away and back to the page for it to start working after the first change to the new mode.
Updated by Pim Pish over 3 years ago
Thank you for pointing that out. Now it shows IPv6 addresses. Great work.