I'm trying to setup our CI in a scalable way. Currently, we have a Windows Server VM on GCE and Jenkins runs jobs on it from time to time.

However, as we'll add more jobs, we would like to have a scalable system.

There are some constraints though:

• there's lots of data synced from Perforce (p4) for the Engine (custom UE4 Engine). And then it was built (takes a while). This is rarely updated.
• there's lots of data synced from p4 for our product. Currently, every time we build, we do a new p4 sync to just retrieve the delta (takes a few seconds) since the last time. And we have a persistent disk. Retrieving the full data would take 45 min - 1 hour (not counting the lost bandwidth).

So, we're trying to use GKE. We would need to somehow:

• Every night (or so), p4 sync the build machine and create an "image" from it now that it contains the latest data.
• Have GKE use that newly created image.
• When we start a job, have it create a container using that image. This would contain data at most 24 hours old.
• this container would p4 sync what has changed during the day, execute whatever it needs to execute, then disappear into the abyss. Whatever generated data would be transient and not affect other containers.
• if several jobs run in parallel, they don't read/write from/to the same place.

So, this is the big-picture idea. There might be better ideas.

Right now, I'm not sure if:

• this is doable
• how to tackle this (if doable)

Regards,

Samuel