https://redmine.pfsense.org/https://redmine.pfsense.org/favicon.ico?16780521162017-04-06T21:13:23ZpfSense bugtrackerpfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=324632017-04-06T21:13:23ZAnonymous
<ul></ul><p>Yeah, my DHCPv6 status page is only showing one lease, which happens to be a static reservation. None of the rest of my address leases show up. I don't have any PD going on since the interface is tracking another.</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=327092017-05-06T20:50:04ZAnders Lindanders.lind@gmail.com
<ul></ul><p>I think I found out what the problem is!</p>
<p>This commit changed how the lease file is handled:<br /><a class="external" href="https://github.com/pfsense/pfsense/commit/8f867225f4c2d3e61863f8d87d4ddb4143d5dda6#diff-b48bdab1db8f78638fbf6c7d98a01172">https://github.com/pfsense/pfsense/commit/8f867225f4c2d3e61863f8d87d4ddb4143d5dda6#diff-b48bdab1db8f78638fbf6c7d98a01172</a></p>
<p>Likely it was tested with GNU awk, because here the lease file is parsed without trouble.</p>
<p>The lease file can be found in: /var/dhcpd/var/db</p>
<p>So the line discussed here is line 164 in <a class="external" href="https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L164">https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L164</a> :<br /><pre>
exec("{$sed} {$cleanpattern} {$leasesfile} | {$awk} {$splitpattern}", $leases_content);
</pre></p>
<p>To be more specific take the lease file and run in Linux/bash:<br /><pre>
cat dhcpd6.leases | sed '/^ia-.. /, /^}/ !d; s,;$,,; s, *, ,g' | awk '{printf $0}; $0 ~ /^\}/ {printf "\n"}'
</pre></p>
<p>It should work fine. You might want while testing to switch history expansion off:<br />set +o histexpand</p>
<p>In pfSense where tsch is used I do not know how to switch history expansion off, but anyway to make sed run you will have to escape !d with a backslash.<br />The commands are then:<br /><pre>
cat dhcpd6.leases | sed '/^ia-.. /, /^}/ \!d; s,;$,,; s, *, ,g' | awk '{printf $0}; $0 ~ /^\}/ {printf "\n"}'
</pre></p>
<p>Now, when awk gets the output of sed and that output contains a percent character % awk will print out an error message like the below and stop executing - that is the reason why a lot of leases do not show up in the GUI:<br /><pre>
awk: weird printf conversion %" {
input record number 58, file
source line number 1
awk: not enough args in printf(ia-na "+\256\264\003\000\001\000\001\033\226Pg\264\256+\031\273%" {)
input record number 58, file
source line number 1
</pre></p>
<p>Now, there might be other characters than just % that makes awk throw an error so maybe you will get other results as well.<br />I do not see any % listed in the initial bug description, but it might be located somewhere earlier in the lease file unless as mentioned other characters make awk stop and throw error messages.</p>
<p>I do not know if the percent sign % needs to be escaped e.g. via<br />sed so that it says %% before being served to awk, but I am looking forward<br />to hear your results!</p>
<p>Thanks!</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=327102017-05-07T08:17:33ZAnders Lindanders.lind@gmail.com
<ul></ul><p>I can confirm adding an extra sed substitution for % to %% works!</p>
<p>Meaning changing <a class="external" href="https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L150">https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L150</a><br /><pre>
$cleanpattern = "'/^ia-.. /, /^}/ !d; s,;$,,; s, *, ,g'";
</pre><br />to:<br /><pre>
$cleanpattern = "'/^ia-.. /, /^}/ !d; s,;$,,; s, *, ,g; s/%/%%/'";
</pre><br />, makes it work.</p>
<p>But there is a huge but.</p>
<p>Currently the regular expression matching here <a class="external" href="https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L208">https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L208</a> has a problem:<br /><pre>
preg_match('/ia-.. "(.*)" { (.*)/ ', $leases_content[$i], $duid_split);
</pre></p>
<p>Main problem is that new lines have already been removed from the input when we reach here.</p>
<p>This means that it is not deterministic when the 1st group matching of<br />"(.*)" will end. Why?<br />Because space character and curly braces {} all are legal characters in the lease file.<br />Why is that so?<br />Because quotify_buf <a class="external" href="https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=common/print.c">https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=common/print.c</a><br />checks for isascii() and isprint().</p>
<p>Now isprint allows characters from 0x20 – 0x7E to become printed directly in the lease file.<br />So here we go - that breaks the regular expression e.g. should it happen someone has a IAID+DUID that looks like<br />'" { '<br />(single quotes are boundaries)</p>
<p>Therefore the idea about that newlines should not be removed too soon.</p>
<p>Now if that problem gets solved and newlines are not removed too soon (by rewriting the code)<br />we can also avoid having the redundant code with a FIXME in lines 238 - 256 (we could remove all of these lines completely) <a class="external" href="https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L238">https://github.com/pfsense/pfsense/blob/1a8b65541c3a3185792f41ed293aaf763a4caf01/src/usr/local/www/status_dhcpv6_leases.php#L238</a> :<br /><pre>
case "ia-pd":
$is_prefix = true;
case "ia-na":
$entry['iaid'] = $tmp_iaid;
$entry['duid'] = $tmp_duid;
if ($data[$f+1][0] == '"') {
$duid = "";
/* FIXME: This needs a safety belt to prevent an infinite loop */
while ($data[$f][strlen($data[$f])-1] != '"') {
$duid .= " " . $data[$f+1];
$f++;
}
$entry['duid'] = $duid;
} else {
$entry['duid'] = $data[$f+1];
}
$entry['type'] = $dynamic_string;
$f = $f+2;
break;
</pre></p>
<p>, which would make the code more clean and less error prone. :)<br />That would mean lines 208 - 216 needs to be rewritten.</p>
<p>Eager to hear your comments on this!</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=334532017-08-11T12:52:02ZRenato Botelhorenato@netgate.com
<ul><li><strong>Assignee</strong> set to <i>Renato Botelho</i></li></ul> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=338592017-09-11T15:58:59ZRenato Botelhorenato@netgate.com
<ul><li><strong>Target version</strong> changed from <i>2.4.0</i> to <i>2.4.1</i></li></ul> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=341582017-10-08T14:04:13ZKill Bill
<ul></ul><p>Am I the only one thinking that this absolutely unreadable regex madness needs to go to /dev/null and ISC DHCP server needs an API to show leases instead?</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=341592017-10-08T14:17:58ZJim Pingle
<ul></ul><p>I agree, but last I looked OMAPI didn't quite do everything we need, plus it required making a program in C to interface with it since there wasn't any PHP interface that we could find. That may have changed recently, but it'll need more investigation.</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=341602017-10-08T15:36:56ZKill Bill
<ul></ul><p>Jim Pingle wrote:</p>
<blockquote>
<p>I agree, but last I looked OMAPI didn't quite do everything we need. ... That may have changed recently, but it'll need more investigation.</p>
</blockquote>
<p>After a quick search, afraid this is a waste of time as far as ISC DHCP is concerned, perhaps time to look into Kea instead.</p>
<p><a class="external" href="http://isc-dhcp-users.2343191.n4.nabble.com/DHCPv6-omshell-OMAPI-tp2072p2078.html">http://isc-dhcp-users.2343191.n4.nabble.com/DHCPv6-omshell-OMAPI-tp2072p2078.html</a></p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=342192017-10-12T09:51:18ZJim Pingle
<ul><li><strong>Target version</strong> changed from <i>2.4.1</i> to <i>2.4.2</i></li></ul><p>Moving target to 2.4.2 as we need 2.4.1 sooner than anticipated.</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=346352017-10-23T12:20:55ZJim Pingle
<ul><li><strong>Target version</strong> changed from <i>2.4.2</i> to <i>2.4.3</i></li></ul> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=352882017-12-03T15:06:33ZAnders Lindanders.lind@gmail.com
<ul></ul><p>I have made a patch that addresses the issue, but it is<br />also a rewrite of a large part of the status leases<br />page, that does the following:</p>
<ul>
<li>Proper handling of the lease file for<br />getting leases, prefixes and the mapping (between<br />the delegated network and the routed ip address.)<br />Everything has been tested. Regarding the Pools/failover<br />part: It has been deduced from looking at the source code<br />and other ISC documentation and not from a live configured<br />system, but some made up 'failover' lease content was<br />used and tested on a live system. E.g. this was pasted<br />into the dhcpv6 leases file:<br /><pre>
failover peer "Failover-Pair-Name" state {
my state recover-wait at 1 2017/03/03 20:20:12;
partner state communications-interrupted at 1 2017/03/03 20:20:12;
mclt 123;
}
failover peer "Failover-2GETHER" state {
my state recover-done at 1 2017/03/03 21:24:12;
partner state unknown-state at 1 2017/03/03 21:44:12;
mclt 456;
}
</pre><br />, but I welcome you to try verifying this on an actual<br />'failover' configured system to see the result.<br />(Feel free to upload/send me a lease file with<br />content from a real 'failover' configured system.)<br />Re. making proper retrieval of failover content I found the<br />relevant places in source (more is mentioned in detail within the patch<br />as well), but part of it can be seen here:<br /><a class="external" href="https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/db.c">https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/db.c</a><br />int write_failover_state (dhcp_failover_state_t *state)<br />and <a class="external" href="https://deepthought.isc.org/article/AA-00252/31/Putting-the-working-server-in-a-failover-pair-into-partner-down-state.html">https://deepthought.isc.org/article/AA-00252/31/Putting-the-working-server-in-a-failover-pair-into-partner-down-state.html</a> (although it lacks the time and MCLT info.)</li>
</ul>
<ul>
<li>An IPv6 address matcher/parser that is used.<br />It can be skipped if one trusts the lease content<br />and that noone manually edits the lease file.</li>
</ul>
<ul>
<li>Unit tests of the IPv6 matching/parsing.<br />Run with: php -f parser_ipv6_tester.php</li>
</ul>
<ul>
<li>A small debugging tool for testing lease file content<br />both on a pfSense router machine (with live content<br />from ndp) as well as offline (e.g. if you copy a lease<br />file over to a desktop computer.)<br />All in all made to make sure that the stuff works.<br />Run with: php -f parser_dhcpv6_lease_tester.php</li>
</ul>
<ul>
<li>Also there is a small correction to wake on lan<br />on the status lease page that did not transfer<br />the mac address to the Services - Wake-on-LAN - Edit page.</li>
</ul>
<ul>
<li>Also this patch cleans up the code and verifies<br />the content of the lease file to some extent.<br />During the cleaning up I have found, by looking<br />at the ISC source code, that these properties<br />are not used at all in DHCPv6 land.<br />They are only used by IPv4 DHCP:<br /><pre>
starts
tstp
tsfp
atsfp
hardware
uid
client-hostname
next (binding state)
</pre><br />, because DHCPv6 server uses the method write_ia<br /><a class="external" href="https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/db.c">https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/db.c</a> (referred to in <a class="external" href="https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/dhcpv6.c">https://source.isc.org/cgi-bin/gitweb.cgi?p=dhcp.git;a=blob;f=server/dhcpv6.c</a> ):<br />DHCPv6: int write_ia(const struct ia_xx *ia)<br />vs<br />DHCPv4: int write_lease (lease)<br />So therefore they are thrown away in this patch.</li>
</ul>
<p>For now I would just like to ask for some feedback<br />e.g. do you like what you see<br />before I submit a pull request?</p>
<p>The files can be found here:<br /><a class="external" href="https://github.com/al-right/pfSense-dhcpv6-gui-leases-patch">https://github.com/al-right/pfSense-dhcpv6-gui-leases-patch</a></p>
<p>Thanks</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=354122017-12-16T19:41:15ZAnders Lindanders.lind@gmail.com
<ul></ul><p>I have added a PR here: <a class="external" href="https://github.com/pfsense/pfsense/pull/3892">https://github.com/pfsense/pfsense/pull/3892</a><br />and updated <a class="external" href="https://forum.pfsense.org/index.php?topic=132791">https://forum.pfsense.org/index.php?topic=132791</a> , because<br />some people in the forum are willing to test.</p>
<p>Thanks</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=355742018-01-15T15:13:11ZAnders Lindanders.lind@gmail.com
<ul></ul><p>The PR has been updated with a second patch addressing the<br />requested changes and includes further amends e.g. visual changes.<br />Please see <a class="external" href="https://github.com/pfsense/pfsense/pull/3892">https://github.com/pfsense/pfsense/pull/3892</a></p>
<p>Thanks</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=357032018-02-01T12:02:19ZAnonymous
<ul><li><strong>Assignee</strong> changed from <i>Renato Botelho</i> to <i>Anonymous</i></li></ul><p>Re-assigned for testing</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=357922018-02-07T12:14:10ZJim Pingle
<ul></ul><p>PR looks good to me. I had a number of leases on my edge firewall that were not showing before, and now they all show up with this patch applied. Before, it showed me 8 leases and 0 delegated. Now it shows 18 leases and 4 delegations, which matches what is in the leases file.</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=359222018-02-22T06:52:03ZRenato Botelhorenato@netgate.com
<ul><li><strong>Status</strong> changed from <i>Confirmed</i> to <i>Feedback</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>PR has been merged</p> pfSense - Bug #7413: status_dhcpv6_leases.php: Some DHCPv6 leases are not displayed in the GUIhttps://redmine.pfsense.org/issues/7413?journal_id=359732018-03-02T15:55:15ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Works fine now</p>