I'm running maven in a docker on a kubernetes cluster. While fetching dependencies, it sometimes hangs for up to 2 hours, logging several connection timeout errors before it finally succeed.

[11:34:31] :     [Step 1/2] [INFO] Downloading: http://repo1.maven.org/maven2/com/path/to/internal/artifact-20181129.112650-1.pom
[11:50:15] :     [Step 1/2] Nov 29, 2018 11:50:15 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[11:50:15] :     [Step 1/2] INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://repo1.maven.org:80: Connection timed out (Read failed)
[11:50:15] :     [Step 1/2] Nov 29, 2018 11:50:15 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[11:50:15] :     [Step 1/2] INFO: Retrying request to {}->http://repo1.maven.org:80
[12:05:59] :     [Step 1/2] Nov 29, 2018 12:05:59 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[12:05:59] :     [Step 1/2] INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://repo1.maven.org:80: Connection timed out (Read failed)
[12:05:59] :     [Step 1/2] Nov 29, 2018 12:05:59 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[12:05:59] :     [Step 1/2] INFO: Retrying request to {}->http://repo1.maven.org:80
[12:21:44] :     [Step 1/2] Nov 29, 2018 12:21:44 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[12:21:44] :     [Step 1/2] INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://repo1.maven.org:80: Connection timed out (Read failed)
[12:21:44] :     [Step 1/2] Nov 29, 2018 12:21:44 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
[12:21:44] :     [Step 1/2] INFO: Retrying request to {}->http://repo1.maven.org:80
[12:37:28] :     [Step 1/2] [INFO] Downloading: http://nexus/nexus/content/repositories/thirdparty/com/path/to/internal/artifact-20181129.112650-1.pom

This only happens for internal dependencies that should immediately fall back to our internal nexus maven repository which is also inside the kubernetes cluster.

A tcpdump during such hang shows only re-transmissions, without response from the central repo. other than that, the capture is filled with ARP queries until finally, the fallback happens and it queries our internal Nexus server.

wireshark screenshot

Here's the relevant section of my pom.xml:

<repositories>
<repository>
    <id>central</id>
    <name>Maven Repository Switchboard</name>
    <layout>default</layout>
    <url>http://repo1.maven.org/maven2</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>
<repository>
    <id>thirdparty</id>
    <name>internal third party repo</name>
    <url>${nexusUrl}/repositories/thirdparty</url>
</repository>
<repository>
    <id>releases</id>
    <name>internal releases repo</name>
    <url>${nexusUrl}/repositories/releases</url>
</repository>
<repository>
    <id>snapshots</id>
    <name>internal snapshots repo</name>
    <url>${nexusUrl}/repositories/snapshots</url>
</repository>
<repository>
    <releases>
        <enabled>true</enabled>
        <updatePolicy>always</updatePolicy>
        <checksumPolicy>fail</checksumPolicy>
    </releases>
    <id>license4j-runtime-library</id>
    <name>License4J Runtime Library Maven Repository</name>
    <url>https://www.license4j.com/maven/</url>
    <layout>default</layout>
</repository>

And here's my settings.xml:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
<localRepository/>
<interactiveMode/>
<usePluginRegistry/>
<offline/>
<pluginGroups/>
<servers>
    <server>
        <id>releases</id>
        <username>XXX</username>
        <password>YYY</password>
    </server>
    <server>
        <id>snapshots</id>
        <username>XXX</username>
        <password>YYY</password>
    </server>
</servers>
<mirrors/>
<proxies/>
<profiles/>
<activeProfiles/>

This happens every once in while and I can't find the root cause. I hope someone has ran into this somewhere.