Hangfire with azure kubernetes randomly running old code version

12/2/2019

I inherited a .Net core app hosted in azure kubernetes that runs a hangfire job daily (job A). Job A was taking several hours to complete, so I optimized it and reduced the run time to 20 minutes. I also added some logs and added a second job (job B) to be run after job A.

Everything was ok until one day I saw that hangfire run the old version of job A (several hours to complete and no logs written) and didn't run job B after that.

I did the following:

  • Deleted the recurring task and the server in hangfire
  • Rescaled the deployment in azure to 0 and then 1, which created the server in hangfire again.
  • Created the recurring task in hangfire again.

After that everything was back to normal until the same thing happened after a few days.

I've never used hangfire, azure or kubernetes before, so I'm a total loss as how this can be happening.

Edit: This is the hangfire startup:

public partial class Startup
{
    private void ConfigureHangFire(IServiceCollection services)
    {
        services.AddHangfire(x =>
            x.UseSqlServerStorage(
                Configuration.GetConnectionString("DefaultConnection")));
    }

    private void ConfigureHangFire(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.Use((context, next) =>
        {
            if (context.Request.Path.StartsWithSegments("/hangfire"))
            {
                context.Request.PathBase = new PathString(context.Request.Headers["X-Forwarded-Prefix"]);
            }
            return next();
        });

        app.UseHangfireServer();

        app.UseHangfireDashboard("/hangfire", new DashboardOptions()
        {
            AppPath = !env.IsDevelopment() && env.EnvironmentName != "Development.Docker" ? $"https://myurl.azure.com/jobssvc" : "http://localhost:60840/",
            Authorization = new[] {new HangFireAuthorizationFilter()}
        });

        BackgroundJob.Schedule(() => StartJobs(), new DateTimeOffset(ConfigurationMethods.GetYearInitDeleteOldInfoService, ConfigurationMethods.GetMonthInitDeleteOldInfoService, ConfigurationMethods.GetDayInitDeleteOldInfoService, ConfigurationMethods.GetHourInitDeleteOldInfoService2, ConfigurationMethods.GetMinuteInitDeleteOldInfoService, 0, 0, TimeSpan.Zero));

    }

    public class HangFireAuthorizationFilter : IDashboardAuthorizationFilter
    {
        public bool Authorize([NotNull] DashboardContext context)
        {
            //can add some more logic here...
            return true;
        }
    }

    public void StartJobs()
    {
        RecurringJob.AddOrUpdate(() => RunJobs(), Cron.Daily(0, 1));
    }

    public void RunJobs()
    {
        var parentId = BackgroundJob.Enqueue<IJobService>(job => job.JobA());            
        BackgroundJob.ContinueWith<IJobService>(parentId, job => job.JobB());            
    } 
}
-- StackUnderFlow
azure
hangfire
kubernetes

0 Answers