We have several options how to correctly manage kubernetes delcaration files and dockerfiles. Services development may be considered as fully separate without any cross service communications for now.
Which approach is better and provides more flexibility? We are having 2 services and soon the counter of services with complex inner network configurations will increase to 8 and 3rd option not looking good at all.
2nd option is better, but I'm not quite sure if k8s separate repo is a good option. While local docker image might also create some difficulties for local development, as it not fully required for developer teams to interact with other services and spin up all services.
1st option looks good as it provides full responsibility dedication and solves devops only task, but in the future may lead to problems when team would need to spin up whole k8s cluster. But, even in this case, this repo might be pulled and executed in respect to minikube.
But neither of those options looking really good for me. Am I missing something?
I would recommend #3. All companies I have worked with so far, keep it that way.
Some Kubernetes native devtools like DevSpace (https://github.com/covexo/devspace) and Draft (https://github.com/Azure/draft) recommend putting Dockerfile and Kubernetes resource definitions (or Helm chart files) into the repository containing the code, too.
Using #3, developers will be able to reproduce production environments better for fixing bugs and CI/CD tools are usually set up to work with infrastructure-as-code like definitions contained in the same repo as the code.
There are exceptions, e.g. GitLab's Auto DevOps CI/CD tool. It is very much designed to work with Kubernetes and works with an external chart. However, they do that because they want to simplify the setup of a CI/CD pipeline to Kubernetes and want to abstract from the underlying Helm chart. They, however, also allow to define a Helm chart yourself and imply that it is located inside the same repository as the code.
#3 is favorable because you can be sure that you can always run repeatable builds when bundling code and "infrastructure" definitions. Let's say you want to go back in time and use an older version of your code, which version of the non-related other repo would you use? Having everything in one repo, will allow you to checkout any revision or branch and always be sure that you can build and instantiate your code.
Example: You change your dependency management tool and need to run another command to install the dependencies. This change will make you change your Dockerfile accordingly. Having both, code + k8s + Dockerfile all together, will make sure, you can still instantiate older versions that use the old dependency management tool.
In my opinion - the second option is the one that makes the most sense - you might want to change the Dockerfile when you change the code for the repository, so you can trigger the CI pipeline with a single push. On the CI here you also want to build the containers. Also the Dockerfile is a resource that you often want both DevOps and Dev teams to be able to access and often work on them together.
Additionally if you have CI integration, you would like to trigger the pipeline after each change to the Dockerfile, which would be considerably easier when it's in the same place with the code.
Then you also have all Kubernetes/Helm files in a different (single) repository so you can manage them and combine them when necessary for the different microservices (if you have more complex deployments). In the CI here, you just want linting for the charts/files.
I think it makes more sense to put the Dockerfiles and its deployment files(k8s YAMLs) all in the same repo of its service, this makes easy to build and deploy the service since the CI/CD tool could just pull de new code, build the image and then deploy. You could also go the Helm way and quit all the deployment files from the service repositories, and they could be all together in a single helm chart. The Helm path makes easy to see the application as a whole. You should use Telepresence if you dont want run all the services on the developer machine.