We have Azure Kubernetes clusters running (one per stage on DEV, TST, PRD) where we need several Python scripts to run periodically, which is what we are using APScheduler (3.6.0) for. The default in-memory jobstore is used.

Some days ago, however, I found out that APScheduler is behaving different than expected. It happens on all three stages:

• A specific job, which is set to run at minute 0 and minute 30 every hour, has stopped running between 21:30 and 00:00 since March 27, 2020.
• This same job runs at weird times, e.g. at ~:15 and :45, frequently.
• Jobs are getting skipped whilst they are scheduled to run every 5 minutes. The logging says 'Launching job', but then does not state the following, which it should: Job "x_job (trigger: cronmonth='', day='', day_of_week='', hour='', minute='0,5,10,15,20,25,30,35,40,45,50,55', next run at: 2020-08-18 10:35:00 UTC)" executed successfully. It occurs that at some point, two runs are triggered at the same time but not necessarily the next run.

Steps undertaken but not with desired result:

Increase the number of process_pool_workers and thread_pool_max_workers and set misfire_grace_time:

Executor thread_pool_max_workers: 50 process_pool_max_workers: 20 Job defaults job_defaults_coalesce: True job_defaults_max_instances: 3 misfire_grace_time: 120

• Set timezone='UTC' for scheduler and in the add_jobs. scheduler = BlockingScheduler(executors=executors, job_defaults=job_defaults, timezone='UTC') scheduler.add_job(launch_profile_job, CronTrigger.from_crontab(scheduler_config.profile_job), timezone='UTC')

I also have checked the resources of the cluster, but CPU and memory of the scheduler are not even close to their limits. We also have quite a low avg. active pod count of 25, and even it this would become an issue to our K8s cluster, autoscaling is enabled.

Does anyone here have a clue what might be going on?