Project

General

Profile

Feature #2123

Backup RRD files using the xml dump and restore from RRD tools

Added by Seth Mos over 8 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Backup / Restore
Target version:
-
Start date:
01/20/2012
Due date:
% Done:

90%

Estimated time:
1.00 h

Description

By dumping the xml files in the backup process we can restore the RRD files from i386 onto a amd64.

Migration path!
Profit!

Associated revisions

Revision 8bdb6879 (diff)
Added by Darren Embry over 8 years ago

Feature #2123 Backup RRD files using the xml dump and restore from RRD tools

http://redmine.pfsense.org/issues/2123

History

#1 Updated by Seth Mos over 8 years ago

  • Assignee set to Darren Embry
  • Affected Version set to 2.1

Not sure if instructions here are clear enough, ask away.

The current RRD backups perform a tar command on /var/db/rrd/*.rrd, we then base64_encode this and tack it on the config.xml when performing a config backup.

The plan is to dump them to xml, add that to the config backup, then restore the xml files to rrd files on config restore.

foreach($rrdfiles as $ rrd)
rrdtool dump /var/db/rrd/$rrd /tmp/{$rrd}.xml
}

tar -czf rrdbackup.tgz /tmp/*.rrd.xml
base64_encode ^
tack that onto the config.xml

On restore
base64_decode($rrdbackup);
tar -xzf ^

foreach($rrdfiles as $rrd) {
rrdtool restore /tmp/{$rrd}.xml /var/db/rrd/{$rrd}
}

Something along those lines. Using this procedure we can switch architectures while saving the RRD history, which is currently not possible.

#2 Updated by Darren Embry over 8 years ago

  • Status changed from New to Assigned
  • Assignee changed from Darren Embry to Chris Buechler
  • % Done changed from 0 to 90

implemented in 8bdb6879

I did a backup/restore config with RRD data and it seems to work just fine.

But I believe this could use more testing than I can wrap my head around so reassigning to Chris.

#3 Updated by Seth Mos over 8 years ago

The quick test I just did, backup config.xml on i386, upgrade to amd64, then restore config.xml didn't fix the RRD files.
Needs more investigation.

Ok, just confirmed that it does not appear to overwrite the old XML files. I tried -f to overwrite the RRD files but that doesn't work either. I need to debug it further.

#4 Updated by Darren Embry over 8 years ago

I've confirmed that -f is actually necessary anyway and added -f and log_error calls on failure to all the rrdtool restore calls.
Please check and see if you're still having the same problem as before.
If so, there should also be some error logs now, and I'd like to see them.

#5 Updated by Darren Embry over 8 years ago

I've also added "RRD Data" as a backup option in the dropdowns. Requires a little special handling in diag_backup.php. :-)

#6 Updated by Seth Mos over 8 years ago

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 90164860 bytes) in /usr/local/www/diag_backup.php on line 368

This is a 256MB vm. Going to try with 512. I think it needs yo unset previous arrays to keep memory use down.

#7 Updated by Cyrill B over 8 years ago

You should try to upgrade the rrdtool as the memory consumption required during restore has been reduced drastically.

2009-05-25 22:23  oetiker

    * src/rrd_restore.c: While restoring, rrd_restore uses between 30
      and 40 times as much memory as the size of the rrd file to be
      restored. This is due to the use of libxml2 for parsing the the
      whole file prior to turning it into rrd structs. This patch
      switches rrd_restore to using the xmlTextReader interface for
      incremental consumtion of xml files. This reduces the memory
      requirements to about twice the size of the rrd file which should
      be quite manageable for most cases. The parsing is about 25% less
      efficient than with the original libxml2 version of rrd_restore.
      This could be in part due to the more liberal use of malloced
      memory. Optimizing this is for another day though. --tobi

http://oss.oetiker.ch/rrdtool/pub/CHANGES

#8 Updated by Jim Pingle over 8 years ago

We're open to that but it requires a lot of changes in RRD tool syntax, fonts, and so on that also require pulling in a bunch more dependencies for rrdtool. It's a can of worms we've been trying to avoid opening.

#9 Updated by Cyrill B over 8 years ago

Jim P wrote:

We're open to that but it requires a lot of changes in RRD tool syntax, fonts, and so on that also require pulling in a bunch more dependencies for rrdtool. It's a can of worms we've been trying to avoid opening.

Well I think in that case it might be better to revert to the previous method of restoring RRDs during normal firmware upgrade as I have seen it to break on my Alix system causing various problems. e.g. not executing the remaining rc.bootup commands.

#10 Updated by Chris Buechler over 7 years ago

  • Status changed from Assigned to New
  • Assignee deleted (Chris Buechler)
  • Target version deleted (2.1)
  • Affected Version deleted (2.1)

for 2.1 this will be reverted as #2935. This needs revisiting when we're on a new rrdtool.

#11 Updated by Seth Mos over 7 years ago

This can be made to work, the issue here is that the memory violation is thrown in PHP, not in the rrd_restore binary.

For the Upgrade path to work I went to great lengths to make it work on a embedded system and I eventually succeeded.

What I ended up doing there was nest the functions as much as possible, and unset each variable after use. This worked, atleast, I have not heard of RRD upgrade failures on embedded for a while now.

The backup/restore feature was written by Darren, and I'm not sure that these changes were reflected. Iirc those were committed after this enhancement was put in. I don't mind having a look this week.

#12 Updated by Seth Mos over 7 years ago

I've looked over the code and the RRD restore does take into account both file formats of pre 2.1 config files. Debugging further.

Also, the OOM listed above is from PHP, not rrd_restore, so it's nice to know, but does not appear to be related to this specific case. I believe Darren did implement the correct fix here.

#13 Updated by Seth Mos over 7 years ago

  • Status changed from New to Feedback

#14 Updated by Chris Buechler about 5 years ago

  • Status changed from Feedback to Resolved

completed ~2-3 years ago.

Also available in: Atom PDF