Commit f4e94c6f authored by Sean McGivern's avatar Sean McGivern

Make usage ping scheduling more robust

On GitLab.com, we see that cron jobs sometimes don't run. This appears to be
because the process that polls for current cron jobs only has a 60 second
validity period, so if (for whatever reason) it misses those 60 seconds, we have
to wait until the next execution.

For the usage ping, this is particularly problematic, as that's only scheduled
to run once a week.

Changing it to run for every minute in a 10 minute period should work around
this, if the above diagnosis is correct. The job itself obtains an exclusive
lease for 24 hours, so rescheduling in quick succession is safe.
parent 5843a43c
---
title: Make usage ping scheduling more robust
merge_request:
author:
type: fixed
......@@ -113,12 +113,14 @@ class Settings < Settingslogic
URI.parse(url_without_path).host
end
# Random cron time every Sunday to load balance usage pings
def cron_random_weekly_time
# Runs every minute in a random ten-minute period on Sundays, to balance the
# load on the server receiving these pings. The usage ping is safe to run
# multiple times because of a 24 hour exclusive lock.
def cron_for_usage_ping
hour = rand(24)
minute = rand(60)
minute = rand(6)
"#{minute} #{hour} * * 0"
"#{minute}0-#{minute}9 #{hour} * * 0"
end
end
end
......@@ -398,7 +400,7 @@ Settings.cron_jobs['stuck_import_jobs_worker'] ||= Settingslogic.new({})
Settings.cron_jobs['stuck_import_jobs_worker']['cron'] ||= '15 * * * *'
Settings.cron_jobs['stuck_import_jobs_worker']['job_class'] = 'StuckImportJobsWorker'
Settings.cron_jobs['gitlab_usage_ping_worker'] ||= Settingslogic.new({})
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_random_weekly_time)
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_for_usage_ping)
Settings.cron_jobs['gitlab_usage_ping_worker']['job_class'] = 'GitlabUsagePingWorker'
Settings.cron_jobs['schedule_update_user_activity_worker'] ||= Settingslogic.new({})
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment