K
Q

Colocating related containers on nodes to avoid the cost of network accesses

12/5/2018

I'm still new to Kubernetes so please excuse if this is a silly question.

I'm architecting a system which includes:

  • an MQTT broker
  • a set of (containerized) microservices that publish and subscribe to it
  • a Redis cache that the microservices read and write to.

We will certainly need multiplicity of all of these components as we scale.

There is a natural division in the multiplicity of each of these things: they each pertain to a set of intersections in a city. A publishing or subscribing microservice will handle 1 or more intersections. The MQTT broker instance and the Redis instance each could be set up to handle n intersections.

I am wondering if it makes sense to try to avoid unnecessary network hops in Kubernetes by trying to divide things up by intersection and put all containers related to a given set of intersections on one node. Would this mean putting them all on a single pod, or is there another way?

(By the way, there will still be other publishers and subscribers that need to access the MQTT broker that are not intersection-specific.)

-- vmayer
kubernetes

Similar Questions

Kubernetes recreate pod if node becomes offline timeout
Differences between `Job` of k8s to the `units` of systemd?
Why am I getting "Structure needs cleaning" message on Ceph with Kubernetes?
Kubernetes go client: list events
autodiscover ASGs: InvalidClientTokenId:

1 Answer

12/5/2018

This is more of an opinion question.

Would this mean putting them all on a single pod, or is there another way?

I would certainly avoid putting them all in one Pod. In theory, you can put anything in a single pod, but the general practice is to add lightweight sidecars that handle a very specific function.

IMO an MQTT broker, a Redis datastore and a subscribe/publish app seem like a lot of to put in a single pod.

Possible Disadvantages:

  • Harder to debug because you may not know where the failure comes from.
  • A publish/subscriber is generally more of a stateless application and MQTT & Redis would stateful. Deployments are more recommended for stateless services and StatefulSets are recommended for stateful services.
  • Maybe networking latency. But you can use Node Affinity and Pod Affinity to mitigate that.

Possible Advantages:

  • All services sharing the same IP/Context.
  • Too much clutter in a pod.

It would be cleaner if you had:

Each one of these workload resources would create separate pods and you can scale independently up/down.

-- Rico
Source: StackOverflow