pfSense

/etc/rc.update_urltables often skips doing a required update ...

• /etc/rc.update_urltables runs at 12:30 PM each day and processes url-table files after a "random" 5-60 second delay (I assume to help prevent thousands of servers from firing off at the same time)

• the variable delay plus the actual file processing adds some number of seconds so timestamps on updated files in /var/db/aliastables/ are from 12:30:05 to 12:31:00 (or later if the server delivering the file is slow and/or the file is large)

• when the "process_alias_urltable" subprogram (in /etc/inc/pfsense-utils.inc) checks to determine whether an update is required, it requires the /var/db/aliastables/ file's timestamp to be more than ('number of days set for "Update Freq." in the URL Table alias' times '86400 seconds' [1 day])

• if the variable (5-60 second) delay in starting the update plus the processing time (in /etc/rc.update_urltables) was longer during the previous update than in the current update, the "Update Freq. * 86400 seconds" will be larger than the difference between the current date/time and the timestamp on the file from the previous update and no update will occur

• a simple way to fix this is to bias the "should we update" check in /etc/inc/pfsense-utils.inc by the maximum delay introduced by /etc/rc.update_urltables, plus some allowance for file-processing time -- e.g.:

In /etc/inc/pfsense-utils.inc change line 2016 from
|| ((time() - filemtime($urltable_filename)) > ($freq * 86400))
to
|| ((time() - filemtime($urltable_filename)) > (($freq * 86400) - 90))

The 90-second bias allows another 30 sec. for any downloading/processing to complete, over and above the 60-second variable delay imposed by /etc/rc.update_urltables and it's still shorter than the 1-day minimum update frequency so it shouldn't break anything else.