Project

General

Profile

Bug #1758

Upgrade fails to upgrade RRD data for traffic and packets

Added by Eddie Atherton about 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Upgrade
Target version:
Start date:
08/05/2011
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.0
Affected Architecture:
i386

Description

Running a nanobsd upgrade from 1.2.3 to 2.0-RC3 gives the following errors:

Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-traffic.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'
Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-packets.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'
Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'
Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-packets.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'
Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-traffic.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'
Aug 4 22:22:20 roadblock php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-packets.rrd N:U:U:U:U' returned exit code '1', the output was 'ERROR: expected 2 data source readings (got 4) from N:U:U:U:U'

Following this, the graphs for traffic and packets is not available.

Cheers.

Associated revisions

Revision e34cf1f6 (diff)
Added by Seth Mos almost 8 years ago

Restore the RRD backup before attempting a migration as this bites the nanobsd users.
We immediately backup the new databases to a new rrd.tgz file. The old database will be moved to /root
Fix for ticket #1758

Revision 8fa054b1 (diff)
Added by Seth Mos almost 8 years ago

Move the old databases to the backup folder in conf
Fixes ticket #1758

Revision 7ad4b1f4 (diff)
Added by Seth Mos almost 8 years ago

Move the old databases to the backup folder in conf
Fixes ticket #1758

Revision eb5e790f (diff)
Added by Seth Mos almost 8 years ago

Restore the RRD backup before attempting a migration as this bites the nanobsd users.
We immediately backup the new databases to a new rrd.tgz file. The old database will be moved to /root
Fix for ticket #1758

Revision 1e3c94dd (diff)
Added by Seth Mos almost 8 years ago

Modify code to remove memory usage to prevent a Out of Memory condition when upgrading the RRD database on a 128MB system.
Ticket #1758

Revision 1005d4bf (diff)
Added by Seth Mos almost 8 years ago

Add the OOM memory restructuring fix from Ticket #1758 into mainline for nanoBSD upgrades.

Revision d7778a7b (diff)
Added by Seth Mos almost 8 years ago

Ok, let's try not to corrupt the RRD files on upgrade. Leave the RRA archives for the 720 minutes average at 1000.
Then run a rrdtool resize command to grow the RRA by 1000 and 2000 for the 60 and 720 average respectively.
Attempts to further fix ticket #1758

Revision eb346e0b (diff)
Added by Seth Mos almost 8 years ago

Ok, let's try not to corrupt the RRD files on upgrade. Leave the RRA arc
hives for the 720 minutes average at 1000.
Then run a rrdtool resize command to grow the RRA by 1000 and 2000 for the 60 and 720 average respectively.
Attempts to further fix ticket #1758

Revision b0b48a57 (diff)
Added by Seth Mos almost 8 years ago

Correct typo in command.
Ticket #1758

Revision 4fe54cdf (diff)
Added by Seth Mos almost 8 years ago

Add back the accidentally removed xml rrd restore line, otherwise we still don't have the new RRD file
Ticket #1758

Revision 12a2f395 (diff)
Added by Seth Mos almost 8 years ago

Fix the 2nd grow command, add space
Ticket #1758

Revision 56ee96ed (diff)
Added by Seth Mos over 7 years ago

More memory optimizations for the RRD upgrade. Ticket #1758 #2159

History

#1 Updated by Seth Mos almost 8 years ago

Caution, if the config is Upgraded on nanobsd platforms the converted RRD files are not immediately saved to the flash. This means that if the box is powercycled instead of rebooted after a conversion the converted RRD files are lost.

Since the conversion takes a really really long time on a Alix or similar the device is often power cycled while it assumed it hangs (which it isn't). This causes a upgraded config to be saved, so no other attempt will be made. This still leaves you with the old RRD files.

#2 Updated by Eddie Atherton almost 8 years ago

These errors showed up on the upgrade, and are associated with the upgrade process, not trying to display the graphs.

Also, during my tests, the box was never powercycled. It was always shut down, or rebooted, using the GUI.

Cheers.

#3 Updated by Seth Mos almost 8 years ago

That's good to know, we'll investigate. We're still not quite sure what's happening, some ipv6 builds seem to hit it more.

#4 Updated by Chris Buechler almost 8 years ago

  • Target version changed from 2.0 to 2.0.1

#5 Updated by Eddie Atherton almost 8 years ago

OK, I'm no expert on RRD, but have done a little more digging on this.

In 1.2.3, the Traffic and Packet RRDs were populated with 2 values, Ibytes, and Obytes, into 2 data points: "in" and "out".

Now, under 2.0, it appears that you are trying to collect 4 values, I4Pass, I4Block, O4Pass, and O4Block, into 4 data points: "inpass", "outpass", "inblock", and "outblock".

However, the RRD, in 2.0, is the "old" 1.2.3 one, hence the errors above, about trying to set 4 values, when only 2 are expected.

This would also explain the errors trying to display the graph:

Oct 9 22:46:02 pfHacom php: /status_rrd_graph_img.php: Failed to create graph with error code 1, the error is: ERROR: No DS called 'inpass' in '/var/db/rrd/wan-traffic.rrd'/usr/bin/nice -n20 /usr/local/bin/rrdtool graph /tmp/wan-traffic.rrd-8hour.png --start 1318196762 --end 1318225562 --vertical-label "bits/sec" --color SHADEA#eeeeee --color SHADEB#eeeeee --title "`hostname` - WAN :: Traffic - 8 hours - 1 minute average" --height 200 --width 620 DEF:wan-in_bytes_pass=/var/db/rrd/wan-traffic.rrd:inpass:AVERAGE DEF:wan-out_bytes_pass=/var/db/rrd/wan-traffic.rrd:outpass:AVERAGE DEF:wan-in_bytes_block=/var/db/rrd/wan-traffic.rrd:inblock:AVERAGE DEF:wan-out_bytes_block=/var/db/rrd/wan-traffic.rrd:outblock:AVERAGE CDEF:"wan-in_bits_pass=wan-in_bytes_pass,8,*" CDEF:"wan-out_bits_pass=wan-out_bytes_pass,8,*" CDEF:"wan-in_bits_block=wan-in_bytes_block,8,*" CDEF:"wan-out_bits_block=wan-out_bytes_block,8,*" CDEF:"wan-in_bytes=wan-in_bytes_pass,wan-in_bytes_block,+" CDEF:"wan-out_bytes=wan-out_bytes_pass,wan-out_bytes_bloc

Where the command is looking for the new data points: "inpass", "outpass", "inblock", and "outblock".

So, obviously, to upgrade from 1.2.3 -> 2.0, the traffic and packet RRDs must be updated to the "new" format.

Now, what I don't know, is if there is supposed to be a step in the upgrade, to perform the change or not. If there is, then somehow it was skipped.

If not, then it needs to be in place, for the 1.2.3 RRD to be carried forward to 2.0.

Cheers,
Eddie

#6 Updated by Seth Mos almost 8 years ago

In config step x we run the following commands. The only things I can think of is that these have failed.

function upgrade_054_to_055() {
global $config;
global $g;

<snip>

mwexec("$rrdtool tune {$rrddbpath}/{$database} -r in:inpass 2>&1");
mwexec("$rrdtool tune {$rrddbpath}/{$database} -r out:outpass 2>71");

That command renames a datasource in a RRD archive from in to inpass and from out to outpass.
We then create a new empty rrd file, dump the xml from that and then layer the old data over the new rrd so that we fill it with data.

It's a rather cumbersome procedure that can take many minutes on a low power alix.

If you can send the rd databases from 1.2.3 my way I can try a lab upgrade.

#7 Updated by Seth Mos almost 8 years ago

Just a FYI in case people want to upgrade their 1.2.3 databases on their 2.0 installs. Executing the following code on the command prompt page will call the required function.

Do not run this on already upgraded databases as that can and will cause data loss. Make sure to download the RRD databases before attempting this.
@
include("upgrade_config.inc");
include("rrd.inc");
include("shaper.inc");

upgrade_054_to_055();@

#8 Updated by Seth Mos almost 8 years ago

  • Status changed from New to Feedback

Ok, so the code now triggers properly on a nanobsd system but it causes a Out of Memory situation on a 128MB system (tested in a VM).

I've checked in another patch that I hope prevents the OOM condition on upgrade as this would leave a system dead in the water on upgrade.

I've just tested the upgrade and it works on a 128MB system without triggering a OOM. What is strange is that the graphs work as intended but the 3 month and 1 year graphs is empty. Upto and including the 1 month graph is ok.

The 4 year graph shows data again but it's shown from the left and not the right.

#9 Updated by Chris Buechler over 7 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF