I have a problem where docker-compose containers aren't able to reach the internet. Manually created containers via the docker cli or kubelet work just fine.

This is on an AWS EC2 node created using Kops with Calico overlay (I think that may be unrelated, however).

Here's the docker-compose:

version: '2.1'
services:
  app:
    container_name: app
    image: "debian:jessie"
    command: ["sleep", "99999999"]
  app2:
    container_name: app2
    image: "debian:jessie"
    command: ["sleep", "99999999"]

This fails:

# docker exec -it app ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

docker-compose container<->container works (as expected):

# docker exec -it app ping app2
PING app2 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: icmp_seq=0 ttl=64 time=0.098 ms

Manually created container works fine:

# docker run -it -d --name app3 debian:jessie sh -c "sleep 99999999"
# docker exec -it app3 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=37 time=9.972 ms

So it seems like docker-compose containers can't reach the internet.

Here's the NetworkSettings from app3, which works:

"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "54168ea912b9caa842b208f36dac80a588ebdc63501a700379fb1b732a41d3ac",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {},
    "SandboxKey": "/var/run/docker/netns/54168ea912b9",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "cdddee0f3e25e7861a98ba6aff33652619a3970c061d0ed2a5dc5bd2b075b30d",
    "Gateway": "172.17.0.1",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "MacAddress": "02:42:ac:11:00:02",
    "Networks": {
        "bridge": {
            "IPAMConfig": null,
            "Links": null,
            "Aliases": null,
            "NetworkID": "46e8bc586d48c9a57e2886f7f35f7c2c8396f8084650fcc2bf1e74788df09e3f",
            "EndpointID": "cdddee0f3e25e7861a98ba6aff33652619a3970c061d0ed2a5dc5bd2b075b30d",
            "Gateway": "172.17.0.1",
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "02:42:ac:11:00:02"
        }
    }
}

From one of the docker-compose containers (fails):

  "NetworkSettings": {
    "Bridge": "",
    "SandboxID": "6b79a6b45f099c65f89adf59eb50eadff2362942f316b05cf20ae1959ca9b88b",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {},
    "SandboxKey": "/var/run/docker/netns/6b79a6b45f09",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
        "root_default": {
            "IPAMConfig": null,
            "Links": null,
            "Aliases": [
                "app2",
                "4f48647ba5bb"
            ],
            "NetworkID": "ffb540b2b9e2945908477a755a43d3505aea6ed94ef5fd944909a91fb104ce8e",
            "EndpointID": "48aff2f00bb4bd670b5178b459a353ac45f7d3efbfb013c1026064022e7c4e59",
            "Gateway": "172.19.0.1",
            "IPAddress": "172.19.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "02:42:ac:13:00:02"
        }
    }
}

So it seems like the major difference is that the docker-compose containers aren't created with an IPAddress or Gateway.

Some background info:

# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:17:57 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:17:57 2017
 OS/Arch:      linux/amd64

# docker-compose version
docker-compose version 1.15.0, build e12f3b9
docker-py version: 2.4.2
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

# ip route
default via 10.20.128.1 dev eth0 
10.20.128.0/20 dev eth0  proto kernel  scope link  src 10.20.140.184 
100.104.10.64/26 via 10.20.136.0 dev eth0  proto bird 
100.109.150.192/26 via 10.20.152.115 dev tunl0  proto bird onlink 
100.111.225.192 dev calic6f21d462fc  scope link 
blackhole 100.111.225.192/26  proto bird 
100.111.225.193 dev calief8dddb6a0d  scope link 
100.111.225.195 dev cali8ca1dd867c3  scope link 
100.111.225.196 dev cali34426885f86  scope link 
100.111.225.197 dev cali6cae60de42a  scope link 
100.111.225.231 dev calibd569acd2f3  scope link 
100.115.17.64/26 via 10.20.148.89 dev tunl0  proto bird onlink 
100.115.237.64/26 via 10.20.167.9 dev tunl0  proto bird onlink 
100.117.246.128/26 via 10.20.150.249 dev tunl0  proto bird onlink 
100.118.80.0/26 via 10.20.162.215 dev tunl0  proto bird onlink 
100.119.204.0/26 via 10.20.135.183 dev eth0  proto bird 
100.123.178.128/26 via 10.20.170.43 dev tunl0  proto bird onlink 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
172.18.0.0/16 dev br-bd6445b00ccf  proto kernel  scope link  src 172.18.0.1 
172.19.0.0/16 dev br-ffb540b2b9e2  proto kernel  scope link  src 172.19.0.1 

iptables are a bit long, so not posting for now (I would expect them to interfere with the non-docker-compose generated containers, so I think the iptables are unrelated).

Anyone know what's going on?