I need to dig deeper into a scheduling issue with kubernetes 1.7:

• I have 18 nodes (of 119) with the label app-nodes=true

  kubectl get nodes -l app-nodes=true
  NAME             STATUS     AGE       VERSION
  app-001   Ready      53d       v1.7.0+coreos.0
  app-002   Ready      53d       v1.7.0+coreos.0
  app-003   Ready      53d       v1.7.0+coreos.0
  app-004   Ready      53d       v1.7.0+coreos.0
  app-005   Ready      53d       v1.7.0+coreos.0
  app-006   Ready      53d       v1.7.0+coreos.0
  app-007   Ready      53d       v1.7.0+coreos.0
  app-008   Ready      53d       v1.7.0+coreos.0
  app-009   Ready      53d       v1.7.0+coreos.0
  app-010   Ready      53d       v1.7.0+coreos.0
  app-011   Ready      53d       v1.7.0+coreos.0
  app-012   Ready      53d       v1.7.0+coreos.0
  app-013   Ready      53d       v1.7.0+coreos.0
  app-014   Ready      53d       v1.7.0+coreos.0
  app-015   Ready      53d       v1.7.0+coreos.0
  app-016   Ready      53d       v1.7.0+coreos.0
  app-017   Ready      53d       v1.7.0+coreos.0
  app-018   Ready      53d       v1.7.0+coreos.0

• I have a deployment w/18 replicas with NodeSelector app-nodes=true and HostPort: 4567

My deployment is stuck with 17/18 pods deployed. With the last one pending with the error:

No nodes are available that match all of the following predicates:: MatchNodeSelector (98), PodFitsHostPorts (18)

• All nodes - except node 11 - have iptable chain Docker with 4567 configured
• There are no dead/stuck pods hanging around in kubectl (taking up the port)
• The nodes are all Ready
• I have inspected deployment / pod / rs
• Have looked at events
• Deleted pods
• Scaled down and up.

With no additional information. I even tried going trough etcd to see if I could find stale data.

How does the kubernetes scheduler determine the Allocated HostPorts? How can I look inside the scheduler?