Kubernetes merges deleted job and new job

9/5/2018

I'm having a google cloud function that is triggered when a file was uploaded to a bucket. The function connects to the kubernetes cluster (using this library), deletes any existing job in a given namespace matching a label purpose: 'address-migration' and creates a new job.

More specifically, this is my code:

const jobsListResponse = await clientInstance.api.batch.v1.jobs.get();
const foundJob = jobsListResponse.body.items.find(job => (
    job.metadata.namespace === 'microservice' &&
    job.metadata.labels && job.metadata.labels['purpose'] === 'address-migration'
));

if (foundJob !== undefined) {
    const deleteResponse = await clientInstance.api.batch.v1.namespace('microservice').job(foundJob.metadata.name).delete();
    console.log('Deleted old job', deleteResponse);
}
const jobManifest = require('./jobManifest.json');
jobManifest.metadata.name += (new Date()).getTime().toString();
jobManifest.spec.template.spec.containers[0].env.push({
    name: 'DP',
    value: targetLink
});
const createJobResponse = await clientInstance.api.batch.v1.namespace('microservice').jobs.post({body: jobManifest});
console.log('Created new job', createJobResponse);

When the function was executed for the first time and no job was there before, it created it and it ran through successfully.

However when the function was triggered a second time, the old job seems to be merged with the new job.

The new job now should have a name like address-migration-${timestamp}. But instead it is address-migration-15361359838821536136672927. The new timestamp was appended to the existing name (including the old timestamp) for strange reasons.

ALSO a new environment variable was merged into it as well. I now have two entries for DP in my manifest:

- name: DP
  value: gs://dpdat/dp2.dat
- name: DP
  value: gs://dpdat/dp3.dat

After all it seems that the job that I just deleted was merged with a job that I created straight afterwards.

-- AmazingTurtle
google-cloud-functions
kubernetes
node.js

1 Answer

9/5/2018

My bad. The google cloud function is stateful. I thought it would bring node up just like php in the old days for every request once. Nope, my changes to my require('./jobManifest.json') were stateful in memory. Once the function "cooled down" (no executions in a while) node exits as well resulting in a state reset. Explains why it worked sometimes and sometimes not.

Conclusion: Google clouds functions are kept warm for a given timespan. In memory changes may persist to following requests as well.

-- AmazingTurtle
Source: StackOverflow