Bug 1866836
| Summary: | [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Matthew Heon <mheon> |
| Component: | Storage | Assignee: | Jan Safranek <jsafrane> |
| Storage sub component: | Kubernetes | QA Contact: | Qin Ping <piqin> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, chuffman, fbertina, hekumar, jsafrane, mstaeble, pehunt, tbarron |
| Version: | 4.4 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: |
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume
|
|
| Last Closed: | 2020-10-27 16:25:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Matthew Heon
2020-08-06 14:19:11 UTC
It looks like that formatting the volume took more than 6 minutes. Checked logs from https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1291011126674329600/artifacts/e2e-openstack/e2e.log and related kubelets: The volume is attached just fine: Aug 05 15:32:11.006695 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:32:11.006175 1410 operation_generator.go:560] MountVolume.WaitForAttach succeeded for volume "pvc-765b8c08-626f-4dd2-98a1-809f65908113" (UniqueName: "kubernetes.io/cinder/9afd1ed3-e5e6-4a3c-acbc-45e3ce2ac339") pod "exec-volume-test-cinder-dynamicpv-bjrc" (UID: "99934d91-3323-4d03-914e-02d5b53a8477") DevicePath "/dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a" The volume appears to be empty, so kubelet tries to format the volume: Aug 05 15:32:11.111733 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:32:11.111666 1410 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a] Format finishes after ~6 minutes: Aug 05 15:38:16.076925 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:38:16.076837 1410 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/9afd1ed3-e5e6-4a3c-acbc-45e3ce2ac339 The test already timed out at that time. This one shows similar symptoms: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1289556089884381184 Aug 01 14:43:06.627340 kig9wk0x-1a357-h74kn-worker-klxns hyperkube[1413]: I0801 14:43:06.627245 1413 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8] ... Aug 01 14:48:56.358364 kig9wk0x-1a357-h74kn-worker-klxns hyperkube[1413]: I0801 14:48:56.358263 1413 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8 /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/5dd0179e-4a5e-469d-8dfb-0c4e3ccb0657 (>5 minutes to format a volume) Trying to reduce the volume size to 1GiB upstream: https://github.com/kubernetes/kubernetes/pull/93923 Another instance when this happened: 557507:Aug 11 01:29:25.439930 9i40zjvx-1a357-f6vb5-worker-lqxjn hyperkube[1406]: I0811 01:29:25.439282 1406 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-8df196f4-3079-441d-8" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-8df196f4-3079-441d-8] 590773:Aug 11 01:35:57.653261 9i40zjvx-1a357-f6vb5-worker-lqxjn hyperkube[1406]: I0811 01:35:57.652410 1406 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-8df196f4-3079-441d-8 /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/8df196f4-3079-441d-817f-02ece43e0c82 From test run - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1292979717074325504 *** Bug 1857346 has been marked as a duplicate of this bug. *** I just noticed in bug #1857346: 1. This happens only in 4.3 and 4.4. Why? 2. Only with ext3 - mkfs.ext3 probably needs to read/write more stuff than ext4 / xfs. Should we skip ext3 in the tests? IMO it does not bring much value anyway. I removed ext2 / ext3 from all volume plugins upstream: https://github.com/kubernetes/kubernetes/pull/95009 Trying to catch 4.6.0 *** Bug 1883617 has been marked as a duplicate of this bug. *** Checked with: 4.6.0-0.nightly-2020-09-29-170625 These PRs are merged, ext2/3 e2e tests are disabled, but this PR missed GCEPD driver. I'll mark this bug as verified and open a new bug to track GCEPD driver. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |