Bug 1866836 - [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume
Summary: [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.6.0
Assignee: Jan Safranek
QA Contact: Qin Ping
URL:
Whiteboard:
: 1857346 1883617 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-06 14:19 UTC by Matthew Heon
Modified: 2020-10-27 16:25 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume
Last Closed: 2020-10-27 16:25:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 373 0 None closed Bug 1866836: UPSTREAM: 95009: Remove ext2 + ext3 tests 2021-02-04 02:35:07 UTC
Github openshift origin pull 25554 0 None closed Bug 1866836: Bump to current openshift/kubernetes master 2021-02-04 02:35:09 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:25:42 UTC

Description Matthew Heon 2020-08-06 14:19:11 UTC
[BUILD-WATCHER]

test:
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume 

is failing frequently in CI (~40%), see search results:
https://search.svc.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-storage%5C%5D+In-tree+Volumes+%5C%5BDriver%3A+cinder%5C%5D+%5C%5BTestpattern%3A+Dynamic+PV+%5C%28ext3%5C%29%5C%5D+volumes+%5C%5BTop+Level%5C%5D+%5C%5Bsig-storage%5C%5D+In-tree+Volumes+%5C%5BDriver%3A+cinder%5C%5D+%5C%5BTestpattern%3A+Dynamic+PV+%5C%28ext3%5C%29%5C%5D+volumes+should+allow+exec+of+files+on+the+volume


https://prow.ci.openshift.org/job-history/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4

https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1291011126674329600/build-log.txt


Also seeing identical failures in similar test:

[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (ext3)] volumes should allow exec of files on the volume

Comment 1 Jan Safranek 2020-08-10 15:58:13 UTC
It looks like that formatting the volume took more than 6 minutes. Checked logs from https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1291011126674329600/artifacts/e2e-openstack/e2e.log and related kubelets:

The volume is attached just fine:
Aug 05 15:32:11.006695 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:32:11.006175    1410 operation_generator.go:560] MountVolume.WaitForAttach succeeded for volume "pvc-765b8c08-626f-4dd2-98a1-809f65908113" (UniqueName: "kubernetes.io/cinder/9afd1ed3-e5e6-4a3c-acbc-45e3ce2ac339") pod "exec-volume-test-cinder-dynamicpv-bjrc" (UID: "99934d91-3323-4d03-914e-02d5b53a8477") DevicePath "/dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a"

The volume appears to be empty, so kubelet tries to format the volume:
Aug 05 15:32:11.111733 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:32:11.111666    1410 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a]

Format finishes after ~6 minutes:
Aug 05 15:38:16.076925 c9cg73pj-1a357-j6hh2-worker-49pk4 hyperkube[1410]: I0805 15:38:16.076837    1410 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-9afd1ed3-e5e6-4a3c-a /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/9afd1ed3-e5e6-4a3c-acbc-45e3ce2ac339

The test already timed out at that time.

Comment 2 Jan Safranek 2020-08-10 16:03:34 UTC
This one shows similar symptoms: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1289556089884381184

Aug 01 14:43:06.627340 kig9wk0x-1a357-h74kn-worker-klxns hyperkube[1413]: I0801 14:43:06.627245    1413 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8]

...

Aug 01 14:48:56.358364 kig9wk0x-1a357-h74kn-worker-klxns hyperkube[1413]: I0801 14:48:56.358263    1413 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-5dd0179e-4a5e-469d-8 /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/5dd0179e-4a5e-469d-8dfb-0c4e3ccb0657

(>5 minutes to format a volume)

Comment 3 Jan Safranek 2020-08-12 12:08:43 UTC
Trying to reduce the volume size to 1GiB upstream: https://github.com/kubernetes/kubernetes/pull/93923

Comment 4 Hemant Kumar 2020-08-12 19:49:03 UTC
Another instance when this happened:

557507:Aug 11 01:29:25.439930 9i40zjvx-1a357-f6vb5-worker-lqxjn hyperkube[1406]: I0811 01:29:25.439282    1406 mount_linux.go:322] Disk "/dev/disk/by-id/virtio-8df196f4-3079-441d-8" appears to be unformatted, attempting to format as type: "ext3" with options: [-F -m0 /dev/disk/by-id/virtio-8df196f4-3079-441d-8]
590773:Aug 11 01:35:57.653261 9i40zjvx-1a357-f6vb5-worker-lqxjn hyperkube[1406]: I0811 01:35:57.652410    1406 mount_linux.go:326] Disk successfully formatted (mkfs): ext3 - /dev/disk/by-id/virtio-8df196f4-3079-441d-8 /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/8df196f4-3079-441d-817f-02ece43e0c82

From test run - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.4/1292979717074325504

Comment 6 Jan Safranek 2020-08-21 12:22:55 UTC
*** Bug 1857346 has been marked as a duplicate of this bug. ***

Comment 7 Jan Safranek 2020-08-21 12:24:56 UTC
I just noticed in bug #1857346:

1. This happens only in 4.3 and 4.4. Why?
2. Only with ext3 - mkfs.ext3 probably needs to read/write more stuff than ext4 / xfs. Should we skip ext3 in the tests? IMO it does not bring much value anyway.

Comment 10 Jan Safranek 2020-09-24 08:37:43 UTC
I removed ext2 / ext3 from all volume plugins upstream: https://github.com/kubernetes/kubernetes/pull/95009
Trying to catch 4.6.0

Comment 15 Peter Hunt 2020-09-29 19:30:25 UTC
*** Bug 1883617 has been marked as a duplicate of this bug. ***

Comment 16 Qin Ping 2020-09-30 00:52:20 UTC
Checked with: 4.6.0-0.nightly-2020-09-29-170625

These PRs are merged, ext2/3 e2e tests are disabled, but this PR missed GCEPD driver.

I'll mark this bug as verified and open a new bug to track GCEPD driver.

Comment 19 errata-xmlrpc 2020-10-27 16:25:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.