Minikube: mount(2) system call failed: Connection refused

7/23/2019

Trying to mount a local drive to my minikube host (seems to be a dupe of this thread but no solution was provided...)

Using:

OSX 10.14.3 and minikube (using HyperVisor)

$ minikube mount --ip 192.168.64.5 --v=7 ~/Documents/projects/docker_storage/tf:/mnt/vda1/data/tf
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:58272
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
  Mounting host path /Users/XXXXXX/Documents/projects/docker_storage/tf into VM as /mnt/vda1/data/tf ...
  Mount options:
    ▪ Type:     9p
    ▪ UID:      docker
    ▪ GID:      docker
    ▪ Version:  9p2000.L
    ▪ MSize:    262144
    ▪ Mode:     755 (-rwxr-xr-x)
    ▪ Options:  map[]
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
  Userspace file server: (minikube) Calling .GetSSHUsername
ufs starting
  Userspace file server is shutdown

  mount failed: mount: /mnt/vda1/data/tf: mount(2) system call failed: Connection refused.
: Process exited with status 32

  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  https://github.com/kubernetes/minikube/issues/new

Expected the mount to map the from local:host however the connection is refused. Might be some firewall or proxy issues?

-- Nicholas Cilfone
kubernetes
macos
minikube

2 Answers

5/25/2020

I'll just leave this here since I had a similar issue on Fedora 32:

mount failed: mount with cmd /bin/bash -c "sudo mount -t 9p -o dfltgid=$(grep ^docker: /etc/group | cut -d: -f3),dfltuid=$(id -u docker),msize=262144,port=38811,trans=tcp,version=9p2000.L 192.168.39.1 /data/pvtheme" : /bin/bash -c "sudo mount -t 9p -o dfltgid=$(grep ^docker: /etc/group | cut -d: -f3),dfltuid=$(id -u docker),msize=262144,port=38811,trans=tcp,version=9p2000.L 192.168.39.1 /data/pvtheme": Process exited with status 32
stdout:

stderr:
mount: /data/pvtheme: mount(2) system call failed: Connection refused.

I solved thanks to this comment: https://github.com/kubernetes/minikube/issues/4726#issuecomment-510217223

It was a firewalld issue:

$ sudo firewall-cmd --permanent --zone=libvirt --add-rich-rule='rule family="ipv4" source address="192.168.39.0/24" accept'

$ sudo firewall-cmd --reload

After the changes, here's my firewalld configuration:

firewall-cmd --zone=libvirt --list-all
libvirt (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: crc virbr0 virbr1
  sources: 
  services: dhcp dhcpv6 dns ssh tftp
  ports: 
  protocols: icmp ipv6-icmp
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
    rule family="ipv4" source address="192.168.39.0/24" accept
    rule priority="32767" reject
-- Ricardo Zanini
Source: StackOverflow

7/26/2019

In brief, possible solution is to update minikube to the recent version. You may also need to delete minikube VM using minikube delete command and update other software like docker using brew upgrade && brew cask upgrade

update2: Another reason why you may have connection problem is because you are using wrong ufs server IP, which is usually host machine IP address in specific virtual subnet. It's usually different for different virtual environments. For --vm-driver=hyperkit it was 192.168.64.1, for --vm-driver=virtualbox it was 192.168.99.1, because I've changed virtualbox network default IP subnet (192.168.56.0/24). It also should work fine without specifying --ip argument at all. Please look at the details below.

The rest of the answer contain the output of my experiments using different vm-drivers, and minikube arguments.

So, sharing folders to minikube VM works pretty well on OSX v10.14.3 and minikube v0.35.0. I've tested it for --vm-driver=virtualbox and --vm-driver=hyperkit.

update: I've also tested it with most recent version of minikube (v1.2.0) with both options of vm-driver: hyperkit and virtualbox, and it works exactly in the same way as it was with older minikube and described below. Log messages are slightly different though.

Here is the results:

You have to install virtualbox to run minikube with virtualbox vm-driver:

$ brew cask install virtualbox
$ brew cask install virtualbox-extension-pack

--vm-driver=virtualbox :

$ minikube start   # --vm-driver=virtualbox is default

Sharing folder without --ip specified:

$ minikube mount --v=7 /Users/some_username/temp/minikube/mkshared/:/mnt/mkshared

  Mounting /Users/some_username/temp/minikube/mkshared/ into /mnt/mkshared on the minikube VM
  This daemon process needs to stay alive for the mount to be accessible ...
ufs starting
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 127.0.0.1 62225 <nil> <nil>}
About to run SSH command:
sudo umount /mnt/mkshared;
SSH cmd err, output: <nil>: 
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 127.0.0.1 62225 <nil> <nil>}
About to run SSH command:

sudo mkdir -p /mnt/mkshared || true;
sudo mount -t 9p -o trans=tcp,port=62422,dfltuid=1001,dfltgid=1001,version=9p2000.L,msize=262144 192.168.99.1 /mnt/mkshared;
sudo chmod 775 /mnt/mkshared || true;
2019/07/25 17:34:04 connected
2019/07/25 17:34:04 >>> 192.168.99.101:45904 Tversion tag 65535 msize 65536 version '9P2000.L'
2019/07/25 17:34:04 <<< 192.168.99.101:45904 Rversion tag 65535 msize 65536 version '9P2000'
2019/07/25 17:34:04 >>> 192.168.99.101:45904 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2019/07/25 17:34:04 <<< 192.168.99.101:45904 Rattach tag 1 aqid (759258 29c45c20 'd')
2019/07/25 17:34:04 >>> 192.168.99.101:45904 Tstat tag 1 fid 0
2019/07/25 17:34:04 <<< 192.168.99.101:45904 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29c45c20 'd') m d775 at 0 mt 1564068830 l 96 t 0 d 0 ext )
2019/07/25 17:34:04 >>> 192.168.99.101:45904 Tstat tag 1 fid 0
2019/07/25 17:34:04 <<< 192.168.99.101:45904 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29c45c20 'd') m d775 at 0 mt 1564068830 l 96 t 0 d 0 ext )
2019/07/25 17:34:04 >>> 192.168.99.101:45904 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2019/07/25 17:34:04 <<< 192.168.99.101:45904 Rwstat tag 1

I'm running these command in separate terminal window, because it's required to keep ufs server running while I'm using it.

host-vm$ minikube ssh
minikube-vm# mount 
...<some output skipped>
192.168.99.1 on /mnt/mkshared type 9p (rw,relatime,sync,dirsync,dfltuid=1001,dfltgid=1001,access=any,msize=65536,trans=tcp,noextend,port=62365)

Note that ufs server IP address is detected automatically. I'm using it In the next experiment as the value for the command line argument --ip.

Sharing folder with --ip specified:

$ minikube mount --ip 192.168.99.1 --v=7 /Users/some_username/temp/minikube/mkshared/:/mnt/mkshared

  Mounting /Users/some_username/temp/minikube/mkshared/ into /mnt/mkshared on the minikube VM
  This daemon process needs to stay alive for the mount to be accessible ...
ufs starting
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 127.0.0.1 62225 <nil> <nil>}
About to run SSH command:
sudo umount /mnt/mkshared;
SSH cmd err, output: <nil>: 
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 127.0.0.1 62225 <nil> <nil>}
About to run SSH command:

sudo mkdir -p /mnt/mkshared || true;
sudo mount -t 9p -o trans=tcp,port=62410,dfltuid=1001,dfltgid=1001,version=9p2000.L,msize=262144 192.168.99.1 /mnt/mkshared;
sudo chmod 775 /mnt/mkshared || true;
2019/07/25 17:31:37 connected
2019/07/25 17:31:37 >>> 192.168.99.101:41646 Tversion tag 65535 msize 65536 version '9P2000.L'
2019/07/25 17:31:37 <<< 192.168.99.101:41646 Rversion tag 65535 msize 65536 version '9P2000'
2019/07/25 17:31:37 >>> 192.168.99.101:41646 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2019/07/25 17:31:37 <<< 192.168.99.101:41646 Rattach tag 1 aqid (759258 29b8e141 'd')
2019/07/25 17:31:37 >>> 192.168.99.101:41646 Tstat tag 1 fid 0
2019/07/25 17:31:37 <<< 192.168.99.101:41646 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29b8e141 'd') m d775 at 0 mt 1564068077 l 96 t 0 d 0 ext )
2019/07/25 17:31:37 >>> 192.168.99.101:41646 Tstat tag 1 fid 0
2019/07/25 17:31:37 <<< 192.168.99.101:41646 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29b8e141 'd') m d775 at 0 mt 1564068077 l 96 t 0 d 0 ext )
2019/07/25 17:31:37 >>> 192.168.99.101:41646 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2019/07/25 17:31:37 <<< 192.168.99.101:41646 Rwstat tag 1

Again, I'm running these command in separate terminal window, because it's required to keep ufs server running while I'm using it.

host-vm$ minikube ssh
minikube-vm# mount 
...<some output skipped>
192.168.99.1 on /mnt/mkshared type 9p (rw,relatime,sync,dirsync,dfltuid=1001,dfltgid=1001,access=any,msize=65536,trans=tcp,noextend,port=62365)

Creating and editing files in the shared directory works fine, files are available in the host directory at the same moment after creating them inside the minikube VM. It works well in the opposite way also. It's required to have docker-machine-driver-hyperkit installed

$ brew install docker-machine-driver-hyperkit

--vm-driver=hyperkit:

$ minikube start --vm-driver=hyperkit

  minikube v0.35.0 on darwin (amd64)
  Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
  "minikube" IP address is 192.168.64.2
  Configuring Docker as the container runtime ...
✨  Preparing Kubernetes environment ...
  Pulling images required by Kubernetes v1.13.4 ...
  Launching Kubernetes v1.13.4 using kubeadm ... 
⌛  Waiting for pods: apiserver proxy etcd scheduler controller addon-manager dns
  Configuring cluster permissions ...
  Verifying component health .....
  kubectl is now configured to use "minikube"
  Done! Thank you for using minikube!

Sharing folder without --ip specified:

$ minikube mount --v=9 /Users/some_username/temp/minikube/mkshared/:/mnt/mkshared

Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:63027
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
  Mounting /Users/some_username/temp/minikube/mkshared/ into /mnt/mkshared on the minikube VM
  This daemon process needs to stay alive for the mount to be accessible ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
ufs starting
Plugin server listening at address 127.0.0.1:63034
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 192.168.64.2 22 <nil> <nil>}
About to run SSH command:
sudo umount /mnt/mkshared;
SSH cmd err, output: Process exited with status 32: umount: /mnt/mkshared: no mount point specified.

(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 192.168.64.2 22 <nil> <nil>}
About to run SSH command:

sudo mkdir -p /mnt/mkshared || true;
sudo mount -t 9p -o trans=tcp,port=63031,dfltuid=1001,dfltgid=1001,version=9p2000.L,msize=262144 192.168.64.1 /mnt/mkshared;
sudo chmod 775 /mnt/mkshared || true;
2019/07/26 10:51:15 connected
2019/07/26 10:51:15 >>> 192.168.64.2:47342 Tversion tag 65535 msize 65536 version '9P2000.L'
2019/07/26 10:51:15 <<< 192.168.64.2:47342 Rversion tag 65535 msize 65536 version '9P2000'
2019/07/26 10:51:15 >>> 192.168.64.2:47342 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2019/07/26 10:51:15 <<< 192.168.64.2:47342 Rattach tag 1 aqid (759258 29c66350 'd')
2019/07/26 10:51:15 >>> 192.168.64.2:47342 Tstat tag 1 fid 0
2019/07/26 10:51:15 <<< 192.168.64.2:47342 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29c66350 'd') m d775 at 0 mt 1564068963 l 96 t 0 d 0 ext )
2019/07/26 10:51:15 >>> 192.168.64.2:47342 Tstat tag 1 fid 0
2019/07/26 10:51:15 <<< 192.168.64.2:47342 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 29c66350 'd') m d775 at 0 mt 1564068963 l 96 t 0 d 0 ext )
2019/07/26 10:51:15 >>> 192.168.64.2:47342 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2019/07/26 10:51:15 <<< 192.168.64.2:47342 Rwstat tag 1
SSH cmd err, output: <nil>: 

Separate terminal window:

host-vm$ minikube ssh
minikube-vm# mount 
...<some output skipped>
192.168.64.1 on /mnt/mkshared type 9p (rw,relatime,sync,dirsync,dfltuid=1001,dfltgid=1001,access=any,msize=65536,trans=tcp,noextend,port=63031)

Note that ufs server IP address is detected automatically and it's different for hyperkit vm-driver. This time I'm also using the same IP address from the mount output in the next experiment as the value for the command line argument --ip.

Sharing folder with --ip specified:

$ minikube mount --ip=192.168.64.1 --v=9 /Users/some_username/temp/minikube/mkshared/:/mnt/mkshared

Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:63168
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
  Mounting /Users/some_username/temp/minikube/mkshared/ into /mnt/mkshared on the minikube VM
  This daemon process needs to stay alive for the mount to be accessible ...
ufs starting
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:63175
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 192.168.64.2 22 <nil> <nil>}
About to run SSH command:
sudo umount /mnt/mkshared;
SSH cmd err, output: <nil>: 
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 192.168.64.2 22 <nil> <nil>}
About to run SSH command:

sudo mkdir -p /mnt/mkshared || true;
sudo mount -t 9p -o trans=tcp,port=63172,dfltuid=1001,dfltgid=1001,version=9p2000.L,msize=262144 192.168.64.1 /mnt/mkshared;
sudo chmod 775 /mnt/mkshared || true;
2019/07/26 10:59:45 connected
2019/07/26 10:59:45 >>> 192.168.64.2:34050 Tversion tag 65535 msize 65536 version '9P2000.L'
2019/07/26 10:59:45 <<< 192.168.64.2:34050 Rversion tag 65535 msize 65536 version '9P2000'
2019/07/26 10:59:45 >>> 192.168.64.2:34050 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2019/07/26 10:59:45 <<< 192.168.64.2:34050 Rattach tag 1 aqid (759258 2d80dc96 'd')
2019/07/26 10:59:45 >>> 192.168.64.2:34050 Tstat tag 1 fid 0
2019/07/26 10:59:45 <<< 192.168.64.2:34050 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 2d80dc96 'd') m d775 at 0 mt 1564131515 l 96 t 0 d 0 ext )
2019/07/26 10:59:45 >>> 192.168.64.2:34050 Tstat tag 1 fid 0
2019/07/26 10:59:45 <<< 192.168.64.2:34050 Rstat tag 1 st ('mkshared' 'some_username' '1720083983' '' q (759258 2d80dc96 'd') m d775 at 0 mt 1564131515 l 96 t 0 d 0 ext )
2019/07/26 10:59:45 >>> 192.168.64.2:34050 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2019/07/26 10:59:45 <<< 192.168.64.2:34050 Rwstat tag 1
SSH cmd err, output: <nil>: 
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Twalk tag 1 fid 0 newfid 1 0:'somefile_from_minikube.txt' 
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rwalk tag 1 (75b56b 2d80dc2b '') 
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tstat tag 1 fid 1
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rstat tag 1 st ('somefile_from_minikube.txt' 'some_username' '1720083983' '' q (75b56b 2d80dc2b '') m 644 at 0 mt 1564131515 l 8140 t 0 d 0 ext )
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Twalk tag 1 fid 1 newfid 2 
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rwalk tag 1 
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Topen tag 1 fid 2 mode 0
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Ropen tag 1 qid (75b56b 2d80dc2b '') iounit 0
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tstat tag 1 fid 1
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rstat tag 1 st ('somefile_from_minikube.txt' 'some_username' '1720083983' '' q (75b56b 2d80dc2b '') m 644 at 0 mt 1564131515 l 8140 t 0 d 0 ext )
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tread tag 1 fid 2 offset 0 count 65512
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rread tag 1 count 8140
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tread tag 1 fid 2 offset 8140 count 65512
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rread tag 1 count 0
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tread tag 1 fid 2 offset 8140 count 65512
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rread tag 1 count 0
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tclunk tag 1 fid 2
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rclunk tag 1
2019/07/26 10:59:53 >>> 192.168.64.2:34050 Tclunk tag 1 fid 1
2019/07/26 10:59:53 <<< 192.168.64.2:34050 Rclunk tag 1

Separate terminal window:

$ minikube ssh
#mount 
...<some output skipped>
192.168.64.1 on /mnt/mkshared type 9p (rw,relatime,sync,dirsync,dfltuid=1001,dfltgid=1001,access=any,msize=65536,trans=tcp,noextend,port=63172)

Accessing existing content in the shared folder as well as creating new files works well, files are available in the host directory at the same moment after creating them inside the minikube VM.

-- VAS
Source: StackOverflow