Project

General

Profile

Bug #3469

rc.update_urltables can skip doing a required update

Added by B. Derman over 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Rules / NAT
Target version:
Start date:
02/19/2014
Due date:
% Done:

100%

Estimated time:
Affected Version:
2.1
Affected Architecture:
All

Description

/etc/rc.update_urltables often skips doing a required update ...

  • /etc/rc.update_urltables runs at 12:30 PM each day and processes url-table files after a "random" 5-60 second delay (I assume to help prevent thousands of servers from firing off at the same time)
  • the variable delay plus the actual file processing adds some number of seconds so timestamps on updated files in /var/db/aliastables/ are from 12:30:05 to 12:31:00 (or later if the server delivering the file is slow and/or the file is large)
  • when the "process_alias_urltable" subprogram (in /etc/inc/pfsense-utils.inc) checks to determine whether an update is required, it requires the /var/db/aliastables/ file's timestamp to be more than ('number of days set for "Update Freq." in the URL Table alias' times '86400 seconds' [1 day])
  • if the variable (5-60 second) delay in starting the update plus the processing time (in /etc/rc.update_urltables) was longer during the previous update than in the current update, the "Update Freq. * 86400 seconds" will be larger than the difference between the current date/time and the timestamp on the file from the previous update and no update will occur
  • a simple way to fix this is to bias the "should we update" check in /etc/inc/pfsense-utils.inc by the maximum delay introduced by /etc/rc.update_urltables, plus some allowance for file-processing time -- e.g.:

In /etc/inc/pfsense-utils.inc change line 2016 from
|| ((time() - filemtime($urltable_filename)) > ($freq * 86400))
to
|| ((time() - filemtime($urltable_filename)) > (($freq * 86400) - 90))

The 90-second bias allows another 30 sec. for any downloading/processing to complete, over and above the 60-second variable delay imposed by /etc/rc.update_urltables and it's still shorter than the 1-day minimum update frequency so it shouldn't break anything else.

Associated revisions

Revision 47f7842f (diff)
Added by Renato Botelho over 6 years ago

Fix #3469

Before downloading file to process urltable, there is a random wait
time between 5 and 60 seconds. Because of this, the difference between
file mtime and current time can be less than $freq * 86400 and it'll be
skipped. Add 90 seconds (60 of max random wait + 30 just to be sure) to
avoid skipping a file that should be updated

Revision e09da6c2 (diff)
Added by Renato Botelho over 6 years ago

Fix #3469

Before downloading file to process urltable, there is a random wait
time between 5 and 60 seconds. Because of this, the difference between
file mtime and current time can be less than $freq * 86400 and it'll be
skipped. Add 90 seconds (60 of max random wait + 30 just to be sure) to
avoid skipping a file that should be updated

History

#1 Updated by Renato Botelho over 6 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

#3 Updated by Chris Buechler about 6 years ago

  • Status changed from Feedback to Resolved
  • Target version set to 2.2

Also available in: Atom PDF