K
Q

Try to use iSCSI volume in Kubernetes Cluster but got "wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program"

4/14/2021

Due to problems might caused by NFS ref, I tried to build iSCSI volume mount in K8S cluster, yet I got errors:

MountVolume.MountDevice failed for volume "iscsipd-rw" : mount failed: exit status 32

Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.20.100:3260-iqn.2020-09.com.xxxx:yyyy.testtarget-lun-1 --scope -- mount -t ext4 -o defaults /dev/disk/by-path/ip-192.168.20.100:3260-iscsi-iqn.2020-09.com.xxxx:yyyy.testtarget-lun-1 /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.20.100:3260-iqn.2020-09.com.xxxx:yyyy.testtarget-lun-1

mount: /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.20.100:3260-iqn.2020-09.com.xxxx:yyyy.testtarget-lun-1: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error.

At beginning I follow this document to create iSCSI initiator, due to errors caused by different situation, I've tried various settings multiple times. iSCSI initiator connection looked well

Command (m for help): p
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2CDE61DE-F57A-4C0B-AFB6-9DD7040A8BBD

Tue Apr 13 15:41:57 i@kt04:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   64G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    1G  0 part /boot
├─sda3                      8:3    0   49G  0 part
│ └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
└─sda4                      8:4    0   14G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
sdb                         8:16   0    1G  0 disk
sr0                        11:0    1 1024M  0 rom

Tue Apr 13 15:45:33 i@kt04:~$ sudo ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root  9 Apr 13 15:41 ip-192.168.20.100:3260-iscsi-iqn.2020-09.com.xxxx:yyyy.testtarget-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root  9 Apr 13 11:07 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 13 11:07 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 13 11:07 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Apr 13 11:07 pci-0000:00:10.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Apr 13 11:07 pci-0000:00:10.0-scsi-0:0:0:0-part4 -> ../../sda4
lrwxrwxrwx 1 root root  9 Apr 13 01:55 pci-0000:02:01.0-ata-1 -> ../../sr0

Tue Apr 13 15:46:18 i@kt04:~$ sudo iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.0-874
Target: iqn.2020-09.com.xxxx:yyyy.testtarget (non-flash)
        Current Portal: 192.168.20.100:3260,1
        Persistent Portal: 192.168.20.100:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2020-09.com.xxxx:yyyy.testtarget
                Iface IPaddress: 192.168.30.24
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 262144
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
                scsi33 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdb          State: running

Tue Apr 13 15:57:55 i@kt04:~$ sudo systemctl status open-iscsi
● open-iscsi.service - Login to default iSCSI targets
   Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2021-04-13 11:03:20 CST; 5h 6min ago
     Docs: man:iscsiadm(8)
           man:iscsid(8)
  Process: 1352 ExecStop=/lib/open-iscsi/logout-all.sh (code=exited, status=0/SUCCESS)
  Process: 1351 ExecStop=/bin/sync (code=exited, status=0/SUCCESS)
  Process: 1301 ExecStop=/lib/open-iscsi/umountiscsi.sh (code=exited, status=0/SUCCESS)
  Process: 1416 ExecStart=/lib/open-iscsi/activate-storage.sh (code=exited, status=0/SUCCESS)
  Process: 1383 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=0/SUCCESS)
 Main PID: 1416 (code=exited, status=0/SUCCESS)

Apr 13 11:03:20 kt04 systemd[1]: Starting Login to default iSCSI targets...
Apr 13 11:03:20 kt04 iscsiadm[1383]: Logging in to [iface: default, target: iqn.2020-09.com.xxxx:yyyy.testtarget, portal: 192.168.20.100,3260] (multiple)
Apr 13 11:03:20 kt04 iscsiadm[1383]: Login to [iface: default, target: iqn.2020-09.com.xxxx:yyyy.testtarget, portal: 192.168.20.100,3260] successful.
Apr 13 11:03:20 kt04 systemd[1]: Started Login to default iSCSI targets.

Tue Apr 13 16:09:28 i@kt04:~$ sudo systemctl status iscsid
● iscsid.service - iSCSI initiator daemon (iscsid)
   Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-13 11:03:20 CST; 5h 6min ago
     Docs: man:iscsid(8)
  Process: 1374 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
  Process: 1364 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
 Main PID: 1377 (iscsid)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/iscsid.service
           ├─1376 /sbin/iscsid
           └─1377 /sbin/iscsid

Apr 13 11:03:20 kt04 systemd[1]: Starting iSCSI initiator daemon (iscsid)...
Apr 13 11:03:20 kt04 iscsid[1374]: iSCSI logger with pid=1376 started!
Apr 13 11:03:20 kt04 systemd[1]: Started iSCSI initiator daemon (iscsid).
Apr 13 11:03:21 kt04 iscsid[1376]: iSCSI daemon with pid=1377 started!
Apr 13 11:03:21 kt04 iscsid[1376]: Connection2:0 to [target: iqn.2020-09.com.xxxx:yyyy.testtarget, portal: 192.168.20.100,3260] through [iface: default] is operational now

Tue Apr 13 16:21:21 i@kt04:~$ cat /proc/scsi/scsi
Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: NECVMWar Model: VMware SATA CD00 Rev: 1.00
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi32 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware   Model: Virtual disk     Rev: 2.0
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi33 Channel: 00 Id: 00 Lun: 01
  Vendor: SYNOLOGY Model: iSCSI Storage    Rev: 4.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05

I've tried "sdb" worked as raw disk like showed above, and also created a sdb1 partition mounted with ext4 filesystem another time (even created LVM once) which caused "mount failed: exit status 32/dev/sdb already mounted or mount point busy" error

pod yaml I used

apiVersion: v1
kind: Pod
metadata:
  name: iscsipd
spec:
  nodeName: kt04
  containers:
  - name: iscsipd-rw
    image: kubernetes/pause
    volumeMounts:
    - mountPath: "/mnt/iscsipd"
      name: iscsipd-rw
  restartPolicy: Always
  volumes:
  - name: iscsipd-rw
    iscsi:
      targetPortal: 192.168.20.100:3260
      iqn: iqn.2020-09.com.xxxx:yyyy.testtarget
      lun: 1
      fsType: ext4
      readOnly: false

The last try I use fdisk to create a sdb1 partition with ext4 but not mounted to /mnt which result as below, still got the same error "wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program"

Wed Apr 14 11:25:06 ice@kt04:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   64G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    1G  0 part /boot
├─sda3                      8:3    0   49G  0 part
│ └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
└─sda4                      8:4    0   14G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
sdb                         8:16   0    1G  0 disk
└─sdb1                      8:17   0 1023M  0 part
sr0                        11:0    1 1024M  0 rom

At NAS(use Synology RS1221) iSCSI configuration panel, it shows that Target is connected (Lun is Thick Provisioning)

k8s on bare-metal version: 1.19.6

iscsiadm version 2.0-874

open-iscsi version 2.0.874-5ubuntu2.10

Anyone can provide some suggestion that what I can try to make it work, or point out what I did wrong?

-- KiddHsieh
iscsi
kubernetes
persistent-volumes

Similar Questions

Using GKE ingress with https load balancer and IAP/security policy enabled
Unable to deploy Minio in kubernetes cluster using Helm
How to check kubelet configurations currently in use?
Limit access to Kubernetes secret by RBAC
how does kubernetes service decide which backend pod to route to

1 Answer

4/14/2021

problem solved. thanks to Long Wu Yuan on Slack#kubernetes-users.

information provided before problem solved:

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.9G     0  7.9G   0% /dev
tmpfs                              1.6G  3.5M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   10G   49G  18% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          976M  146M  764M  16% /boot

$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   64G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    1G  0 part /boot
├─sda3                      8:3    0   49G  0 part
│ └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
└─sda4                      8:4    0   14G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
sdb                         8:16   0    1G  0 disk
└─sdb1                      8:17   0 1023M  0 part
sr0                        11:0    1 1024M  0 rom

delete pod then

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.9G     0  7.9G   0% /dev
tmpfs                              1.6G  3.5M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   10G   49G  18% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          976M  146M  764M  16% /boot

execute sudo dd if=/dev/zero of=/dev/sdb bs=1M count=512 status=progress then

$ sudo dd if=/dev/zero of=/dev/sdb bs=1M count=512 status=progress
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 4.85588 s, 111 MB/s
$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   64G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    1G  0 part /boot
├─sda3                      8:3    0   49G  0 part
│ └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
└─sda4                      8:4    0   14G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
sdb                         8:16   0    1G  0 disk
sr0                        11:0    1 1024M  0 rom

then apply pod again, it WORKS！, df -h & lsblk information after pod working

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.9G     0  7.9G   0% /dev
tmpfs                              1.6G  3.7M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   10G   49G  17% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          976M  146M  764M  16% /boot

$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   64G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    1G  0 part /boot
├─sda3                      8:3    0   49G  0 part
│ └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
└─sda4                      8:4    0   14G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   63G  0 lvm  /
sdb                         8:16   0    1G  0 disk
sr0                        11:0    1 1024M  0 rom

As Long said, I should know better about the error message "bad superblock" and find solution or what the mis-configuration of my environment is for this iSCSI volume situation.

-- KiddHsieh
Source: StackOverflow