pfSense

The quick test I just did, backup config.xml on i386, upgrade to amd64, then restore config.xml didn't fix the RRD files.
Needs more investigation.

Ok, just confirmed that it does not appear to overwrite the old XML files. I tried -f to overwrite the RRD files but that doesn't work either. I need to debug it further.

I've confirmed that -f is actually necessary anyway and added -f and log_error calls on failure to all the rrdtool restore calls.
Please check and see if you're still having the same problem as before.
If so, there should also be some error logs now, and I'd like to see them.

I've also added "RRD Data" as a backup option in the dropdowns. Requires a little special handling in diag_backup.php. :-)

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 90164860 bytes) in /usr/local/www/diag_backup.php on line 368

This is a 256MB vm. Going to try with 512. I think it needs yo unset previous arrays to keep memory use down.

You should try to upgrade the rrdtool as the memory consumption required during restore has been reduced drastically.

2009-05-25 22:23  oetiker

    * src/rrd_restore.c: While restoring, rrd_restore uses between 30
      and 40 times as much memory as the size of the rrd file to be
      restored. This is due to the use of libxml2 for parsing the the
      whole file prior to turning it into rrd structs. This patch
      switches rrd_restore to using the xmlTextReader interface for
      incremental consumtion of xml files. This reduces the memory
      requirements to about twice the size of the rrd file which should
      be quite manageable for most cases. The parsing is about 25% less
      efficient than with the original libxml2 version of rrd_restore.
      This could be in part due to the more liberal use of malloced
      memory. Optimizing this is for another day though. --tobi

http://oss.oetiker.ch/rrdtool/pub/CHANGES

We're open to that but it requires a lot of changes in RRD tool syntax, fonts, and so on that also require pulling in a bunch more dependencies for rrdtool. It's a can of worms we've been trying to avoid opening.

Jim P wrote:

We're open to that but it requires a lot of changes in RRD tool syntax, fonts, and so on that also require pulling in a bunch more dependencies for rrdtool. It's a can of worms we've been trying to avoid opening.

Well I think in that case it might be better to revert to the previous method of restoring RRDs during normal firmware upgrade as I have seen it to break on my Alix system causing various problems. e.g. not executing the remaining rc.bootup commands.

This can be made to work, the issue here is that the memory violation is thrown in PHP, not in the rrd_restore binary.

For the Upgrade path to work I went to great lengths to make it work on a embedded system and I eventually succeeded.

What I ended up doing there was nest the functions as much as possible, and unset each variable after use. This worked, atleast, I have not heard of RRD upgrade failures on embedded for a while now.

The backup/restore feature was written by Darren, and I'm not sure that these changes were reflected. Iirc those were committed after this enhancement was put in. I don't mind having a look this week.

I've looked over the code and the RRD restore does take into account both file formats of pre 2.1 config files. Debugging further.

Also, the OOM listed above is from PHP, not rrd_restore, so it's nice to know, but does not appear to be related to this specific case. I believe Darren did implement the correct fix here.