Bug #10508
closedBackup does not skip all RRD data
100%
Description
If I backup with skip RRD data enabled I will get this:
$ grep filename config-pfsense.rad-az.local-20200429135803.xml <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename>1_LAN_RFK_GW-quality.rrd</filename> <filename>3_DSL_PPPOE-quality.rrd</filename> <filename>3_SDSL_PPPOE-quality.rrd</filename> <filename>4_GW_VDSL_MODEM-quality.rrd</filename> <filename>4_VDSL_DHCP6-quality.rrd</filename> <filename>4_VDSL_PPPOE-quality.rrd</filename> <filename>6_UNUSED_DHCP-quality.rrd</filename> <filename>GW_Inexio10M-quality.rrd</filename> <filename>GW_KV_SAFENET-quality.rrd</filename> <filename>GW_LAN-quality.rrd</filename> <filename>OV_DRK_ALZEY_VPNV4-quality.rrd</filename> <filename>OV_ROADWARRIORS_VPNV4-quality.rrd</filename> <filename>OV_SCHUBI_VPNV4-quality.rrd</filename> <filename>OV_ZMT_EXTERN1_VPNV4-quality.rrd</filename> <filename>OV_ZMT_HQ1_VPNV4-quality.rrd</filename> <filename>ipsec-packets.rrd</filename> <filename>ipsec-traffic.rrd</filename> <filename>lan-packets.rrd</filename> <filename>lan-traffic.rrd</filename> <filename>opt1-packets.rrd</filename> <filename>opt1-traffic.rrd</filename> <filename>opt10-packets.rrd</filename> <filename>opt10-traffic.rrd</filename> <filename>opt11-packets.rrd</filename> <filename>opt11-traffic.rrd</filename> <filename>opt12-packets.rrd</filename> <filename>opt12-traffic.rrd</filename> <filename>opt13-packets.rrd</filename> <filename>opt13-traffic.rrd</filename> <filename>opt14-packets.rrd</filename> <filename>opt14-traffic.rrd</filename> <filename>opt15-packets.rrd</filename> <filename>opt15-traffic.rrd</filename> <filename>opt2-packets.rrd</filename> <filename>opt2-traffic.rrd</filename> <filename>opt3-packets.rrd</filename> <filename>opt3-traffic.rrd</filename> <filename>opt4-packets.rrd</filename> <filename>opt4-traffic.rrd</filename> <filename>opt5-packets.rrd</filename> <filename>opt5-traffic.rrd</filename> <filename>opt6-packets.rrd</filename> <filename>opt6-traffic.rrd</filename> <filename>opt7-packets.rrd</filename> <filename>opt7-traffic.rrd</filename> <filename>opt8-packets.rrd</filename> <filename>opt8-traffic.rrd</filename> <filename>opt9-packets.rrd</filename> <filename>opt9-traffic.rrd</filename> <filename>ovpns4-packets.rrd</filename> <filename>ovpns4-traffic.rrd</filename> <filename>ovpns4-vpnusers.rrd</filename> <filename>ovpns5-packets.rrd</filename> <filename>ovpns5-traffic.rrd</filename> <filename>ovpns5-vpnusers.rrd</filename> <filename>ovpns6-packets.rrd</filename> <filename>ovpns6-traffic.rrd</filename> <filename>ovpns6-vpnusers.rrd</filename> <filename>system-mbuf.rrd</filename> <filename>system-memory.rrd</filename> <filename>system-processor.rrd</filename> <filename>system-states.rrd</filename> <filename>wan-packets.rrd</filename> <filename>wan-traffic.rrd</filename>
If disable skip RRD data I will get this:
$ grep filename config-pfsense.rad-az.local-20200429140100.xml <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename></filename> <filename32></filename32> <filename64></filename64> <filename>1_LAN_RFK_GW-quality.rrd</filename> <filename>3_DSL_PPPOE-quality.rrd</filename> <filename>3_SDSL_PPPOE-quality.rrd</filename> <filename>4_GW_VDSL_MODEM-quality.rrd</filename> <filename>4_VDSL_DHCP6-quality.rrd</filename> <filename>4_VDSL_PPPOE-quality.rrd</filename> <filename>6_UNUSED_DHCP-quality.rrd</filename> <filename>GW_Inexio10M-quality.rrd</filename> <filename>GW_KV_SAFENET-quality.rrd</filename> <filename>GW_LAN-quality.rrd</filename> <filename>OV_DRK_ALZEY_VPNV4-quality.rrd</filename> <filename>OV_ROADWARRIORS_VPNV4-quality.rrd</filename> <filename>OV_SCHUBI_VPNV4-quality.rrd</filename> <filename>OV_ZMT_EXTERN1_VPNV4-quality.rrd</filename> <filename>OV_ZMT_HQ1_VPNV4-quality.rrd</filename> <filename>ipsec-packets.rrd</filename> <filename>ipsec-traffic.rrd</filename> <filename>lan-packets.rrd</filename> <filename>lan-traffic.rrd</filename> <filename>opt1-packets.rrd</filename> <filename>opt1-traffic.rrd</filename> <filename>opt10-packets.rrd</filename> <filename>opt10-traffic.rrd</filename> <filename>opt11-packets.rrd</filename> <filename>opt11-traffic.rrd</filename> <filename>opt12-packets.rrd</filename> <filename>opt12-traffic.rrd</filename> <filename>opt13-packets.rrd</filename> <filename>opt13-traffic.rrd</filename> <filename>opt14-packets.rrd</filename> <filename>opt14-traffic.rrd</filename> <filename>opt15-packets.rrd</filename> <filename>opt15-traffic.rrd</filename> <filename>opt2-packets.rrd</filename> <filename>opt2-traffic.rrd</filename> <filename>opt3-packets.rrd</filename> <filename>opt3-traffic.rrd</filename> <filename>opt4-packets.rrd</filename> <filename>opt4-traffic.rrd</filename> <filename>opt5-packets.rrd</filename> <filename>opt5-traffic.rrd</filename> <filename>opt6-packets.rrd</filename> <filename>opt6-traffic.rrd</filename> <filename>opt7-packets.rrd</filename> <filename>opt7-traffic.rrd</filename> <filename>opt8-packets.rrd</filename> <filename>opt8-traffic.rrd</filename> <filename>opt9-packets.rrd</filename> <filename>opt9-traffic.rrd</filename> <filename>ovpns4-packets.rrd</filename> <filename>ovpns4-traffic.rrd</filename> <filename>ovpns4-vpnusers.rrd</filename> <filename>ovpns5-packets.rrd</filename> <filename>ovpns5-traffic.rrd</filename> <filename>ovpns5-vpnusers.rrd</filename> <filename>ovpns6-packets.rrd</filename> <filename>ovpns6-traffic.rrd</filename> <filename>ovpns6-vpnusers.rrd</filename> <filename>system-mbuf.rrd</filename> <filename>system-memory.rrd</filename> <filename>system-processor.rrd</filename> <filename>system-states.rrd</filename> <filename>wan-packets.rrd</filename> <filename>wan-traffic.rrd</filename> <filename>1_LAN_RFK_GW-quality.rrd</filename> <filename>1_WAN_DHCP-quality.rrd</filename> <filename>1_WAN_DHCP6-quality.rrd</filename> <filename>3_DSL_PPPOE-quality.rrd</filename> <filename>3_SDSL_PPPOE-quality.rrd</filename> <filename>3_VDSL_DHCP6-quality.rrd</filename> <filename>3_VDSL_PPPOE-quality.rrd</filename> <filename>3_VDSL_PPPOE_PPPOE-quality.rrd</filename> <filename>4_GW_VDSL_MODEM-quality.rrd</filename> <filename>4_VDSL_DHCP6-quality.rrd</filename> <filename>4_VDSL_PPPOE-quality.rrd</filename> <filename>6_UNUSED_DHCP-quality.rrd</filename> <filename>GW_Inexio10M-quality.rrd</filename> <filename>GW_KV_SAFENET-quality.rrd</filename> <filename>GW_LAN-quality.rrd</filename> <filename>GW_TI_KONNEKTOR-quality.rrd</filename> <filename>OV_DRK_ALZEY_VPNV4-quality.rrd</filename> <filename>OV_ROADWARRIORS_VPNV4-quality.rrd</filename> <filename>OV_SCHUBI_VPNV4-quality.rrd</filename> <filename>OV_ZMT_EXTERN1_VPNV4-quality.rrd</filename> <filename>OV_ZMT_HQ1_VPNV4-quality.rrd</filename> <filename>WAN_DHCP-quality.rrd</filename> <filename>ipsec-packets.rrd</filename> <filename>ipsec-traffic.rrd</filename> <filename>lan-packets.rrd</filename> <filename>lan-traffic.rrd</filename> <filename>opt1-packets.rrd</filename> <filename>opt1-traffic.rrd</filename> <filename>opt10-packets.rrd</filename> <filename>opt10-traffic.rrd</filename> <filename>opt11-packets.rrd</filename> <filename>opt11-traffic.rrd</filename> <filename>opt12-packets.rrd</filename> <filename>opt12-traffic.rrd</filename> <filename>opt13-packets.rrd</filename> <filename>opt13-traffic.rrd</filename> <filename>opt14-packets.rrd</filename> <filename>opt14-traffic.rrd</filename> <filename>opt15-packets.rrd</filename> <filename>opt15-traffic.rrd</filename> <filename>opt16-packets.rrd</filename> <filename>opt16-traffic.rrd</filename> <filename>opt18-packets.rrd</filename> <filename>opt18-traffic.rrd</filename> <filename>opt2-packets.rrd</filename> <filename>opt2-traffic.rrd</filename> <filename>opt3-packets.rrd</filename> <filename>opt3-traffic.rrd</filename> <filename>opt4-packets.rrd</filename> <filename>opt4-traffic.rrd</filename> <filename>opt5-packets.rrd</filename> <filename>opt5-traffic.rrd</filename> <filename>opt6-packets.rrd</filename> <filename>opt6-traffic.rrd</filename> <filename>opt7-packets.rrd</filename> <filename>opt7-traffic.rrd</filename> <filename>opt8-packets.rrd</filename> <filename>opt8-traffic.rrd</filename> <filename>opt9-packets.rrd</filename> <filename>opt9-traffic.rrd</filename> <filename>ovpns4-packets.rrd</filename> <filename>ovpns4-traffic.rrd</filename> <filename>ovpns4-vpnusers.rrd</filename> <filename>ovpns5-packets.rrd</filename> <filename>ovpns5-traffic.rrd</filename> <filename>ovpns5-vpnusers.rrd</filename> <filename>ovpns6-packets.rrd</filename> <filename>ovpns6-traffic.rrd</filename> <filename>ovpns6-vpnusers.rrd</filename> <filename>system-mbuf.rrd</filename> <filename>system-memory.rrd</filename> <filename>system-processor.rrd</filename> <filename>system-states.rrd</filename> <filename>wan-packets.rrd</filename> <filename>wan-traffic.rrd</filename>
-r-------- 1 ggz ggz 11M Apr 29 15:58 config-pfsense.rad-az.local-20200429135803.xml -r-------- 1 ggz ggz 22M Apr 29 16:01 config-pfsense.rad-az.local-20200429140100.xml
Related issues
Updated by Grischa Zengel over 4 years ago
$ ls -lah /var/db/rrd total 7435 drwxr-xr-x 2 nobody wheel 78B Apr 29 14:01 . drwxr-xr-x 14 root wheel 63B Apr 29 11:29 .. -rw-r--r-- 1 nobody wheel 144K Oct 25 2019 1_LAN_RFK_GW-quality.rrd -rw-r--r-- 1 nobody wheel 144K Oct 25 2019 1_WAN_DHCP-quality.rrd -rw-r--r-- 1 nobody wheel 144K Oct 25 2019 1_WAN_DHCP6-quality.rrd -rw-r--r-- 1 nobody wheel 70K Dec 5 23:37 3_DSL_PPPOE-quality.rrd -rw-r--r-- 1 nobody wheel 70K Oct 25 2019 3_SDSL_PPPOE-quality.rrd -rw-r--r-- 1 nobody wheel 144K Apr 29 14:33 3_VDSL_DHCP6-quality.rrd -rw-r--r-- 1 nobody wheel 144K Apr 29 14:33 3_VDSL_PPPOE-quality.rrd -rw-r--r-- 1 nobody wheel 144K Dec 5 23:44 3_VDSL_PPPOE_PPPOE-quality.rrd -rw-r--r-- 1 nobody wheel 144K Dec 5 23:37 4_GW_VDSL_MODEM-quality.rrd -rw-r--r-- 1 nobody wheel 144K Apr 29 14:33 4_VDSL_DHCP6-quality.rrd -rw-r--r-- 1 nobody wheel 144K Apr 29 14:33 4_VDSL_PPPOE-quality.rrd -rw-r--r-- 1 nobody wheel 70K Oct 25 2019 6_UNUSED_DHCP-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 GW_Inexio10M-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 GW_KV_SAFENET-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 GW_LAN-quality.rrd -rw-r--r-- 1 nobody wheel 144K Apr 29 14:33 GW_TI_KONNEKTOR-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 OV_DRK_ALZEY_VPNV4-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 OV_ROADWARRIORS_VPNV4-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 OV_SCHUBI_VPNV4-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 OV_ZMT_EXTERN1_VPNV4-quality.rrd -rw-r--r-- 1 nobody wheel 70K Apr 29 14:33 OV_ZMT_HQ1_VPNV4-quality.rrd -rw-r--r-- 1 nobody wheel 144K Oct 24 2019 WAN_DHCP-quality.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ipsec-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ipsec-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 lan-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 lan-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt1-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt1-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt10-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt10-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt11-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt11-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt12-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt12-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt13-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt13-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt14-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt14-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt15-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt15-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt16-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt16-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Nov 15 14:44 opt18-packets.rrd -rw-r--r-- 1 nobody wheel 384K Nov 15 14:44 opt18-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt2-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt2-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt3-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt3-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt4-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt4-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt5-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt5-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt6-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt6-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt7-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt7-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt8-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt8-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt9-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 opt9-traffic.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns4-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns4-traffic.rrd -rw-r--r-- 1 nobody wheel 49K Apr 29 14:33 ovpns4-vpnusers.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns5-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns5-traffic.rrd -rw-r--r-- 1 nobody wheel 49K Apr 29 14:33 ovpns5-vpnusers.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns6-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 ovpns6-traffic.rrd -rw-r--r-- 1 nobody wheel 49K Apr 29 14:33 ovpns6-vpnusers.rrd -rw-r--r-- 1 nobody wheel 575K Apr 29 14:33 system-mbuf.rrd -rw-r--r-- 1 nobody wheel 718K Apr 29 14:33 system-memory.rrd -rw-r--r-- 1 nobody wheel 240K Apr 29 14:33 system-processor.rrd -rw-r--r-- 1 nobody wheel 240K Apr 29 14:33 system-states.rrd -rw-r--r-- 1 root wheel 25K Apr 23 18:21 updaterrd.sh -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 wan-packets.rrd -rw-r--r-- 1 nobody wheel 384K Apr 29 14:33 wan-traffic.rrd
Updated by Jim Pingle over 4 years ago
- Status changed from New to Not a Bug
Your config must have had older data already inside due to a different bug fixed a while back (like #8994) or restoring a config using a method which doesn't support restoring RRD (like putting the config on the memstick when installing). If you skip RRD, there is no way new RRD entries could end up in the backup.
Check your config in /conf/config.xml and see what it has. If you need help fixing that, post on the forum.
Updated by Grischa Zengel over 4 years ago
If it's an evolution problem, why not filtering wrong tags on export?
If you don't skip you will skip the wrong section.
Why not skip this section always?
/* If the config on disk had rrddata tags already, remove that section first. * See https://redmine.pfsense.org/issues/8994 */ $data = preg_replace("/<rrddata>.*<\\/rrddata>/", "", $data);
Updated by Grischa Zengel over 4 years ago
If want to make everybody happy:
replace this
if ($_POST['backuparea'] !== "rrddata" && !$_POST['donotbackuprrd']) { $rrd_data_xml = rrd_data_xml(); $closing_tag = "</" . $g['xml_rootobj'] . ">"; /* If the config on disk had rrddata tags already, remove that section first. * See https://redmine.pfsense.org/issues/8994 */ $data = preg_replace("/<rrddata>.*<\\/rrddata>/", "", $data); $data = preg_replace("/<rrddata\\/>/", "", $data); $data = str_replace($closing_tag, $rrd_data_xml . $closing_tag, $data); }
with this
/* If the config on disk had rrddata tags already, remove that section first. * See https://redmine.pfsense.org/issues/8994 */ $data = preg_replace("/<rrddata>.*<\\/rrddata>/", "", $data); $data = preg_replace("/<rrddata\\/>/", "", $data); if ($_POST['backuparea'] !== "rrddata" && !$_POST['donotbackuprrd']) { $rrd_data_xml = rrd_data_xml(); $closing_tag = "</" . $g['xml_rootobj'] . ">"; $data = str_replace($closing_tag, $rrd_data_xml . $closing_tag, $data); }
Updated by Viktor Gurov over 4 years ago
it's safe to always skip the wrong rrd tags section on export:
https://github.com/pfsense/pfsense/pull/4297
Updated by Jim Pingle over 4 years ago
- Status changed from Not a Bug to Pull Request Review
- Target version set to 2.5.0
Updated by Jim Pingle over 4 years ago
- Target version changed from 2.5.0 to 2.4.5-p1
Updated by Jim Pingle over 4 years ago
- Status changed from Pull Request Review to Feedback
Updated by Jim Pingle over 4 years ago
- Status changed from Feedback to New
This doesn't appear to be working. Added some dummy RRD tags to a config and they are still there when downloading a backup. If I also backup RRD, I have two tags.
Updated by Jim Pingle over 4 years ago
The old code doesn't appear to have ever worked properly as it was. I pushed some changes to the regex which make it work as expected + some improvement in output.
Updated by Jim Pingle over 4 years ago
- Status changed from In Progress to Feedback
- % Done changed from 0 to 100
Applied in changeset 4213d677f6e665d1b391066c27c17155d8da1699.
Updated by Jim Pingle over 4 years ago
- Status changed from Feedback to Resolved
With an existing <rrddata>
section in the backup, now backups are generated without the tag entirely (skip RRD checked), and with only a single expected tag when exporting with fresh RRD contents.
Updated by Jim Pingle about 2 years ago
- Related to Bug #13132: Multiple ``<sshdata>`` or ``<rrddata>`` sections in ``config.xml`` lead to an XML parsing error during restore added