So I'm trying to wrap my head around what exactly a typical Kubernetes pod looks like. According to their docs, a pod:
"A pod (as in a pod of whales or pea pod) corresponds to a colocated group of applications running with a shared context."
Later in that same article:
"Pods can be used to host vertically integrated application stacks, but their primary motivation is to support co-located..."
OK, so you can organize a single pod as your entire vertical stack (from DB to web app). But apparently that's not typically how it's organized, so I assume that typically a "horizontal" organization is preferred (why??).
But to me, horizontal layering/stratification implies that you'll only have one container in a pod, because typically in each tier of service (web, app, cache, db, etc.) you'll have one type of component.
Let's take a concrete example. Say we have the following vertical stack of tiers:
These are fairly typical components in an app stack. If we went against Kubernetes' own advice, and created "vertically-aligned" pods, each pod would consist of 1 type of container for each tier (the web/app server, each microservice, each DB, etc.).
But how would a horizontally-aligned pod be organized? What containers would go in which pods?
A Pod is the basic scheduling unit in Kubernetes. It is the common case that a pod will only have a single container running in it, as most containers can be scheduled independently (i.e. they do not need to be co-located on the same machine).
With regards to your example, you could put most containers in individual pods, and use a Replication Controller to horizontally scale the number of replicas of each Pod (and therefore container) as needed. Along with your replication controller, you'll also want a Service to load balance between the replicas. Vertical tiers could be organized using labels on the pods/replication controllers/services, such as tier=message_broker.
Edit:
The reason it's not a good idea to put your entire stack in a single pod is it limits your flexibility: