Project

General

Profile

Actions

Bug #3334

closed

Status/Traffic Graph isn't IPv6 ready

Added by Armin Fasold almost 11 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Category:
Traffic Graphs
Target version:
Start date:
11/21/2013
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
All
Affected Architecture:

Description

In /usr/local/www/bandwith_by_ip.php, which is called per AJAX/XHR to provide the data for the right table in "Traffic Graph" page (status_graph.php), the program "rate" is called to acquire that data.

Looking into the source code of "rate", to see what all the given parameters do, I noticed that it uses u_int32_t [0] for storing the IP addresses. This results in a silent fail if called with an IPv6 netmask and no listing of IPv6 traffic sources.

[0]: https://fossies.org/dox/rate-0.9/rate__abusers_8c_source.html#l00062


Files

bandwidth_by_ip.php (6.31 KB) bandwidth_by_ip.php Joshua Sign, 01/18/2019 06:17 PM
iftop_parse.awk (986 Bytes) iftop_parse.awk Joshua Sign, 01/18/2019 06:17 PM
iftop_parser.sh (1.07 KB) iftop_parser.sh Joshua Sign, 01/18/2019 06:17 PM
pfSense_status_graph.patch (7.95 KB) pfSense_status_graph.patch Joshua Sign, 01/19/2019 12:24 AM
Capture iftop.PNG (24.1 KB) Capture iftop.PNG Daryl Morse, 01/20/2019 05:47 PM
Capture iftop hn0.PNG (22.9 KB) Capture iftop hn0.PNG Daryl Morse, 01/20/2019 05:55 PM
Capture iftop hn1.PNG (22.9 KB) Capture iftop hn1.PNG Daryl Morse, 01/20/2019 05:56 PM
iftop status graph.PNG (77.6 KB) iftop status graph.PNG Daryl Morse, 01/20/2019 06:02 PM
iftop_parse.awk.gz (308 Bytes) iftop_parse.awk.gz Joshua Sign, 01/21/2019 04:12 AM
Actions #1

Updated by Chris Buechler almost 9 years ago

  • Status changed from New to Confirmed
Actions #2

Updated by Luca Moncelli about 8 years ago

IPv6 issue still present in 2.3.2

Actions #3

Updated by Jim Pingle over 7 years ago

  • Affected Version changed from 2.1-IPv6 to All
Actions #4

Updated by Pim Pish over 7 years ago

When will this be resolved? This is a really old bug.

Actions #5

Updated by Jim Pingle over 7 years ago

The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.

I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.

Actions #6

Updated by Daryl Morse over 5 years ago

Jim Pingle wrote:

The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.

I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.

I looked at the man page for rate. It appears to have been written in 2003 by Mateusz Golicz <>, aka github.com/mteg. There doesn't appear to be a repository for rate, so you have to think he abandoned it and moved on. I tried emailing him, but he didn't reply. In the absence of the author, the code isn't going to fix itself.

Actions #7

Updated by Joshua Sign over 5 years ago

Jim Pingle wrote:

I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.

Me too, I am looking for a way to replace rate by iftop.
Since iftop-1.0pre3 (2014-01-01), a text output mode was added, so it is possible to get an output like rate.
An awk script is good enough for that, i just test something like that :

# iftop -nNb -i eth0 -s 1 -o 2s -F 10.0.0.0/8 -t | awk -f iftop_parse.awk
10.0.50.233 3,27Kb 3,27Kb 1,87Kb 1,87Kb
216.58.198.206 1,87Kb 1,87Kb 3,27Kb 3,27Kb
10.0.50.233 1,59Kb 1,59Kb 2,18Kb 2,18Kb
216.58.204.101 2,18Kb 2,18Kb 1,59Kb 1,59Kb
10.0.48.251 896b 896b 0b 0b
224.0.0.18 0b 0b 896b 896b
10.0.51.217 660b 660b 0b 0b
239.255.255.250 0b 0b 660b 660b
10.0.51.9 644b 644b 0b 0b
239.255.255.250 0b 0b 644b 644b

The awk script is very simple :

BEGIN {
        FS              = " " 
        numlist = 0
}
{
        if ( numlist == 1 && $1 == "--------------------------------------------------------------------------------------------" ) { 
                numlist = 2 
                next
        }

        if ( numlist == 0 && $1 == "--------------------------------------------------------------------------------------------" ) { 
                numlist = 1 
                next
        }

        if ( numlist == 1 ) { 
                if ( $0 ~ "=>") {
                        SENDER = $2
                        STX = $4
                        STXAVG = $5
                        getline
                        RECEIVER = $1
                        RTX = $3
                        RTXAVG = $4
                        print SENDER" "STX" "STXAVG" "RTX" "RTXAVG
                        print RECEIVER" "RTX" "RTXAVG" "STX" "STXAVG

                }
                next
        }

        if ( $1 == "============================================================================================" ) {
                numlist = 0 
                next
        }

}
END {
}

I also notice no errors on disconnected interfaces or when there is no trafic.
(version 1.0pre4 of iftop, on 2.4.4 pfsense release)

Please tell me if I can help, or if I just waste our time.

Actions #8

Updated by Jim Pingle over 5 years ago

That does help a bit. It would be even better if iftop had an output mode like libxo where it would be trivial to parse the output, but I'm not sure how the project author(s) would feel about that.

Can you create a PR that changes rate to iftop? No need to worry about the dependency issues at first (adding iftop to base, that is), since we'd need to run it through manual tests before it gets merged anyhow.

Actions #9

Updated by Joshua Sign over 5 years ago

ok i will work on it and create a PR to change rate by iftop as soon as it works
(normaly it should be ok on sunday or before)

Actions #10

Updated by Joshua Sign over 5 years ago

Hi Jim,

FYI, I just finish some tests : it seems to works as expected.

All we need to test is :

This awk script 'iftop_parser.awk' :

BEGIN {
        FS              = " " 
        numlist  = 0
        nblines  = 15
        printf "\n\n" 
}
{
        if ( numlist == 1 && $1 == "--------------------------------------------------------------------------------------------" ) {
                exit
        }

        if ( numlist == 0 && $1 == "--------------------------------------------------------------------------------------------" ) {
                numlist = 1
                next
        }

        if ( numlist == 1 ) {
                if ( $0 ~ "=>" && nblines > 0 ) {
                        SENDER = $2
                        STX = pfFormat($4)
                        getline
                        RECEIVER = $1
                        RTX = pfFormat($3)
                        printf "%s;%s;%s\n", SENDER,RTX,STX
                        printf "%s;%s;%s\n", RECEIVER,STX,RTX
                        nblines--
                        if ( nblines < 1 ) {
                                exit
                        }
                }
                next
        }

}
END {
}

function pfFormat(str) {
        sub("b","",str)
        return str
}

And these changes in /usr/local/www/bandwidth_by_ip.php :

- replace line 112 by the code bellow :

$_grb = exec("/usr/bin/timeout 5 /usr/local/sbin/iftop -nNb -i {$real_interface} -s 1 -o 2s -t | /usr/bin/awk -f /root/iftop_parse.awk", $listedIPs);

// Functions format
// TODO : replace by existing function or put this in pfsense-utils.inc
function GMK2bits($val) {
        $val = strtolower($val);
        if ( strpos($val,"g") !== false ) {
                $val = str_replace("g","",$val) * 1000 * 1000 * 1000;
        } else if ( strpos($val,"m") !== false ) {
                $val = str_replace("m","",$val) * 1000 * 1000;
        } else if ( strpos($val,"k") !== false ) {
                $val = str_replace("k","",$val) * 1000;
        }

        return $val*1;
}

function bits2GMK($val) {
        $units = array ( 0 => "", 1 => "K", 2 => "M", 3 => "G" );
        $u=0;
        while ($val > 1000) {
                $val = $val/1000;
                $u++;
        }
        return round($val,2).$units[$u];
}
// END of Functions format

// order and group by
foreach ($listedIPs as $k => $line){
        if ($line != "") {
                $arrLine = explode (";", $line);
                $ip  = $arrLine[0];
                $in  = GMK2bits($arrLine[1]);
                $out = GMK2bits($arrLine[2]);
                if ( isset($arr_in[$ip]) ) {
                        $arr_in[$ip]  += $in;
                        $arr_out[$ip] += $out;
                } else {
                        $arr_in[$ip]  = $in;
                        $arr_out[$ip] = $out;
                }
        }
}

if ($sort == "out") {
        arsort($arr_out,SORT_NUMERIC);
        $arrIP = array_keys($arr_out);
} else {
        arsort($arr_in,SORT_NUMERIC);
        $arrIP = array_keys($arr_in);
}

unset($listedIPs);
$listedIPs[] = "";
$listedIPs[] = "";
foreach ($arrIP as $k => $ip) {
        $listedIPs[] = $ip.";".bits2GMK($arr_in[$ip]).";".bits2GMK($arr_out[$ip]);
}

- change separator ":" by ";" on the line 180
(it was line 122 before adding the code above) :

$infoarray = explode (";", $bandwidthinfo);

Unfortunally i don't use IPV6, so i can't test this part.
But it was necessary to change this separator to avoid IPV6 confict.

The exec of iftop may take between 1 or 2 seconds most of time.
But i experienced some cases where iftop wait more than 7 seconds before giving a result (or no result because not enough traffic).
Maybe thats what you experienced when you was talking about issues, so i add 'timeout 5' to the command line to ensure that iftop will not stay stucked and overload the system.
Yes it is a crappy solution, but it is not the best we can do.
This is even more usefull if there are many users on the status_graph.php : multiples concurrent processes are created.
We can look for another way to avoid multiples concurrent processes and increase performances. (i got some ideas)

Please tell me :
- if i miss something (i don't use '-F' or '-G' flag, maybe needed for ipv6?)
- if the order of IN/OUT bis are corrects... they seems to me
- where is the best place for the awk script ? (/etc ? /usr/local/bin ?)
- is there an equivalent of GMK2bits / bits2GMK in pfSense ? If there is not, should i add them to pfsense-utils.inc ?
- do we try to improve performances about concurrent processes if many users are on the same graph ?

It also should be possible to create a PR by adding iftop as an additional feature instead of replacing rate. (iftop must be added by pkg install)
This could be better to avoid any revert if somes have problems with iftop.
Otherwise, the old unused code can be removed.

I wait for your instructions and feedbacks if some of us can test it.

Actions #11

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

Unfortunally i don't use IPV6, so i can't test this part.

I have IPv6 so I would be happy to test when you have something ready. Can you provide it as a patch based on the latest development snapshot?

Actions #12

Updated by Joshua Sign over 5 years ago

yes it will be possible soon.

I just wrote this script to avoid process concurrent creation when many users are on the same graph.
My tests looks good.

I'll put it in the PR.

#!/bin/sh

# Usage
if [ $# -ne 1 ]; then
        echo "Usage : $0 iface" 
        exit
fi

# files paths
pid_file=/var/run/iftop_$1.pid
cache_file=/var/db/iftop_$1.log
awk_script=/root/iftop_parse.awk

# Binaries paths
DATE=/bin/date
STAT=/usr/bin/stat
CUT=/usr/bin/cut
PS=/bin/ps
GREP=/usr/bin/grep
CAT=/bin/cat
RM=/bin/rm
IFTOP=/usr/local/sbin/iftop
AWK=/usr/bin/awk

# test if pid file exist
if [ -f $pid_file ]; then
        # check how old is the file
        curTS=`$DATE +%s`
        pidTS=`$STAT -r $pid_file | $CUT -d " " -f 10`
        # if more than 10 seconds,
        # it must be a dead pid file (process killed?)
        # or a stucked process that we should kill
        if [ $(( curTS - pidTS )) -gt 10 ]; then
                oldPID=`$CAT $pid_file`
                # test if pid still exist
                run=`$PS -p $oldPID | $GREP -F '$0'`
                if [ "$run" != "" ]; then
                        kill -9 $oldPID
                fi
                $RM $pid_file
                $RM $cache_file
                exit
        fi
        if [ -f $cache_file ]; then
                $CAT $cache_file
        fi
        exit
fi

echo -n $$ > $pid_file

$IFTOP -nNb -i $1 -s 1 -o 2s -t 2>> /dev/null | $AWK -f $awk_script > $cache_file.1
$CAT $cache_file.1 > $cache_file
$CAT $cache_file
$RM $pid_file

Then just change the exec in /usr/local/www/bandwidth_by_ip.php by

$_grb = exec("/root/iftop_parser.sh {$real_interface}", $listedIPs);

I'll create a PR soon.

Actions #13

Updated by Joshua Sign over 5 years ago

here are the files you need to easely test, it is faster thant the PR
just put the two scripts into the root directory, and replace the /usr/local/www/bandwidth_by_ip.php

Actions #14

Updated by Joshua Sign over 5 years ago

Here is the patch

/usr/local/bin/iftop_parser.sh must have +x

Actions #16

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

PR : https://github.com/pfsense/pfsense/pull/4039

I installed this patch on the most recent snapshot, built on Sat Jan 19 05:32:33 EST 2019. I used http://ipv6-test.com/speedtest/ to test it, because there are separate speedtests for ipv4 and ipv6, making it easy to see traffic for both protocols. I tried both WAN and LAN, but the patch doesn't show anything for ipv6. For ipv4, I did not see any difference with the patch applied or not applied. I did not reboot after applying the patch. Should I have?

Actions #17

Updated by Joshua Sign over 5 years ago

can you chexk over console if iftop shows you some IPV6 adresses just by : `iftop -n` ?

as far as i didn't have any ipv6, it will be difficult to test this part.

Actions #18

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

can you chexk over console if iftop shows you some IPV6 adresses just by : `iftop -n` ?

as far as i didn't have any ipv6, it will be difficult to test this part.

Sorry mate. iftop -n shows ipv4 and ipv6 traffic, but there is no indication of it in the list of hosts or ip addresses. iftop -n display formats ipv4 and ipv6 traffic exactly the same, except for the addresses (obviously). If you use iftop without -n, it's not possible to tell the difference between ipv4 and ipv6.

Actions #19

Updated by Joshua Sign over 5 years ago

ok,

to debbug it you can check if there is any ip6 in this output :

 /usr/local/bin/iftop_parser.sh eth0 

if there is no ipv6 in the output, please check the output of this command line :

 iftop -nNb -i eth0  -s 1 -o 2s -t 

You must find ipv6 here, so give me the output for example, the awk script may be wrong.
If there is no ipv6 here, i dont understand why.

Actions #20

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

ok,

to debbug it you can check if there is any ip6 in this output :
[...]

if there is no ipv6 in the output, please check the output of this command line :
[...]

You must find ipv6 here, so give me the output for example, the awk script may be wrong.
If there is no ipv6 here, i dont understand why.

My system does not have a device called eth0. It's using hn1 for the wan and hn0 for the lan. I'd say that's part of the problem.

I got permission denied when I tried to run the script from the console shell.

When I executed the iftop command, there was ipv4 and ipv6 output for both hn0 and hn1. Except for the address format, the output is indistinguishable for ipv4 and ipv6. It's a shame there isn't a way to select ipv4 or ipv6 because if it resolves the hostnames, there is no way to tell them apart.

Actions #21

Updated by Daryl Morse over 5 years ago

just a thought, if you don't have ipv6, you could set up a tunnel with hurricane electric. It's free, it works very well, and it's easy to set up.

Actions #22

Updated by Joshua Sign over 5 years ago

Daryl Morse wrote:

I got permission denied when I tried to run the script from the console shell.

please check if the script got +x permission,
or give it anyway, type this in console :

chmod 755 /usr/local/bin/iftop_parser.sh

obiously, eth0 was un axemple,
you have to change it according your system.

in the iftop mode, the informations are grabbed by this command for your WAN card :

 iftop -nNb -i hn1 -s 1 -o 2s -t 

and this one for you LAN card :

 iftop -nNb -i hn0 -s 1 -o 2s -t 

Does it gives you ipv6 results ?

Actions #23

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

Daryl Morse wrote:

I got permission denied when I tried to run the script from the console shell.

please check if the script got +x permission,
or give it anyway, type this in console :
[...]

obiously, eth0 was un axemple,
you have to change it according your system.

in the iftop mode, the informations are grabbed by this command for your WAN card :
[...]

and this one for you LAN card :
[...]

Does it gives you ipv6 results ?

I attached a screen capture of the output of the script. There are some errors.

I already told you the output of iftop shows ipv4 and ipv6 and they are identical except for the addresses. A couple of screen captures attached.

Actions #24

Updated by Daryl Morse over 5 years ago

Another screen capture from the status graph.

Actions #25

Updated by Joshua Sign over 5 years ago

ok,

the first file "File Capture iftop.PNG" show that there is a problem with the awk script.
This script is less than 40 lines and errors are about lines 43,76.

Can you check if this is the same file than this :
https://github.com/pfsense/pfsense/blob/8e73bf65ab51822b438872f13bdf7475ac370365/src/usr/local/bin/iftop_parse.awk

Actions #26

Updated by Daryl Morse over 5 years ago

Joshua Sign wrote:

ok,

the first file "File Capture iftop.PNG" show that there is a problem with the awk script.
This script is less than 40 lines and errors are about lines 43,76.

Can you check if this is the same file than this :
https://github.com/pfsense/pfsense/blob/8e73bf65ab51822b438872f13bdf7475ac370365/src/usr/local/bin/iftop_parse.awk

I installed the script using the patch, so it must be the same.

Actions #27

Updated by Joshua Sign over 5 years ago

you are certainly in an issue with bads crlf in the awk script.
please update by this one, i gzip it to avoid any copy/paste problem.

fetch it in your pfsense and use this command line to replace the old one :

fetch https://redmine.pfsense.org/attachments/download/2695/iftop_parse.awk.gz
gzcat iftop_parse.awk.gz > /usr/local/bin/iftop_parse.awk
Actions #28

Updated by Jim Pingle over 5 years ago

With some small modifications, it does work. See my comments on the PR (and future discussion should happen on the PR rather than here)

Actions #29

Updated by Daryl Morse over 5 years ago

Jim Pingle wrote:

The underlying program, rate, still doesn't work with IPv6 as far as I'm aware.

I'd love to see rate swapped out for iftop (which does support IPv6) but the output of iftop isn't easy to parse and it has some issues running on interfaces that are disconnected or have no traffic, which makes it difficult to work with in an automated manner. iftop is also connection-oriented so it doesn't give per-host stats like rate. I have some testing code for it but I'm not terribly happy with how it operates yet. I don't see it replacing rate, but perhaps it might work as an add-on package.

As I mentioned, I emailed the author of rate. He replied this morning and confirmed that he is no longer developing rate. He offered up the source code, if you don't have it already. Let me know if you are interested in the details.

Actions #30

Updated by Jim Pingle over 5 years ago

Daryl Morse wrote:

As I mentioned, I emailed the author of rate. He replied this morning and confirmed that he is no longer developing rate. He offered up the source code, if you don't have it already. Let me know if you are interested in the details.

We already have access to the source code for rate, or it couldn't be compiled into our images. Adapting the GUI to use iftop is almost certainly easier than adapting rate to handle IPv6.

iftop is still being actively developed. If we attempt to adapt rate, then we'd be taking on all the technical debt of maintaining it going forward, on top of the work to make it function now.

Actions #31

Updated by Jim Pingle about 5 years ago

See also: #7224

Actions #32

Updated by Jim Pingle about 5 years ago

  • Status changed from Confirmed to Pull Request Review
  • Target version set to 2.5.0
Actions #33

Updated by Renato Botelho over 4 years ago

  • Status changed from Pull Request Review to Feedback
  • Assignee set to Renato Botelho
  • % Done changed from 0 to 100

Pull request has been merged. Thanks!

Actions #34

Updated by Viktor Gurov over 4 years ago

  • Status changed from Feedback to Resolved

tested on 2.5.0.a.20200212.1057
works as expected

Actions #35

Updated by Jim Pingle over 4 years ago

  • Status changed from Resolved to In Progress

There is still a problem or two here.

The iftop binary is not present unless you manually install the pkg. Also the pfSense package (net-mgmt/pfSense-pkg-iftop) needs to be removed since net-mgmt/iftop can now be set as a dependency of the security/pfSense meta pkg.

Actions #36

Updated by Jim Pingle over 4 years ago

Also it looks like there is an odd condition where the first time you switch to iftop, it doesn't want to display anything. But if you save it as default, navigate away for a few seconds, then go back, it shows data. I'd call it a fluke but it happened on two systems so far.

That could be moved to its own separate (potentially even post-2.5.0) issue if it persists after the pkg issue is sorted.

Actions #37

Updated by Renato Botelho about 4 years ago

Jim Pingle wrote:

There is still a problem or two here.

The iftop binary is not present unless you manually install the pkg. Also the pfSense package (net-mgmt/pfSense-pkg-iftop) needs to be removed since net-mgmt/iftop can now be set as a dependency of the security/pfSense meta pkg.

It's done since Feb

Actions #38

Updated by Renato Botelho about 4 years ago

  • Status changed from In Progress to Feedback

Jim Pingle wrote:

Also it looks like there is an odd condition where the first time you switch to iftop, it doesn't want to display anything. But if you save it as default, navigate away for a few seconds, then go back, it shows data. I'd call it a fluke but it happened on two systems so far.

That could be moved to its own separate (potentially even post-2.5.0) issue if it persists after the pkg issue is sorted.

I couldn't reproduce it on recent snapshots. As you said, this ticket is now completed and we can open a new one for specific bugs.

Actions #39

Updated by Jim Pingle about 4 years ago

  • Status changed from Feedback to Resolved

I'd say it's working well enough for now. I tried it on a few more systems and it's OK, even on arm.

Actions #40

Updated by Jim Pingle almost 4 years ago

  • Category changed from Web Interface to Traffic Graphs
Actions #41

Updated by Pim Pish over 3 years ago

I've upgraded to pfSense 2.5 RC but still there are no IPv6 addresses shown in the traffic graph for me. What am I doing wrong?

Actions #42

Updated by Jim Pingle over 3 years ago

It doesn't change to the new mode automatically. You have to change it from rate to iftop..

  • Status > Traffic Graph
  • Set Mode to iftop
  • Click Save

Make sure it's set to display IP Addresses, not hostnames, or you may not notice when it does report IPv6 traffic.

You might have to navigate away and back to the page for it to start working after the first change to the new mode.

Actions #43

Updated by Pim Pish over 3 years ago

Thank you for pointing that out. Now it shows IPv6 addresses. Great work.

Actions

Also available in: Atom PDF