Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1661026

Summary: Run iotests as part of build process
Product: Red Hat Enterprise Linux 8 Reporter: Karen Mezick <kmezick>
Component: qemu-kvmAssignee: Karen Mezick <kmezick>
Status: CLOSED CURRENTRELEASE QA Contact: CongLi <coli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0CC: areis, chayang, coli, ddepaula, jen, jsnow, juzhang, jwboyer, kmezick, mtessun, ngu, qzhang, rbalakri, virt-bugs, virt-maint, ymankad
Target Milestone: rcFlags: rule-engine: mirror+
Target Release: 8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-2.12.0-56.module+el8+2679+29eb8264 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1664463 (view as bug list) Environment:
Last Closed: 2019-06-14 01:59:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1664463, 1664855    

Description Karen Mezick 2018-12-19 21:39:45 UTC
Description of problem: 

We want to run iotests as part of the build process in order to catch potential issues earlier in the development cycle.

To accomplish this we will initially run 34 iotests with the -raw option and 83 iotests with the -qcow2 option. All of these tests are currently passing consistently. At a later date we will add additional iotests.

The following code will be added to qemu-kvm.spec.template. The numbers represent the iotest name:

./check -v -raw 001 002 004 005 008 009 010 011 012 021 025 032 033 048 052 063 077 086 101 106 120 140 143 145 150 159 160 162 170 171 175 184 221 226

./check -v -qcow2 001 002 004 005 008 009 010 011 012 017 018 019 020 021 024 025 027 028 029 032 033 034 035 037 038 042 046 047 048 050 052 053 058 062 063 066 068 069 072 073 074 086 087 089 090 095 098 102 103 105 107 108 110 111 120 127 133 134 138 140 141 143 144 145 150 154 156 158 159 162 170 177 179 182 184 188 190 195 204 209 217 226


Expected results:

build.log -->

...

TEST: tests/test-hmp... (pid=27900)
  /x86_64/hmp/pc-i440fx-rhel7.0.0:                                     qemu-system-x86_64: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
OK
  /x86_64/hmp/pc-q35-rhel7.6.0:                                        OK
  /x86_64/hmp/none:                                                    OK
  /x86_64/hmp/pc-i440fx-rhel7.5.0:                                     OK
  /x86_64/hmp/pc-i440fx-rhel7.6.0:                                     OK
  /x86_64/hmp/pc-i440fx-rhel7.1.0:                                     qemu-system-x86_64: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
OK
  /x86_64/hmp/pc-i440fx-rhel7.2.0:                                     OK
  /x86_64/hmp/pc-q35-rhel7.3.0:                                        OK
  /x86_64/hmp/pc-q35-rhel7.4.0:                                        OK
  /x86_64/hmp/pc-i440fx-rhel7.3.0:                                     OK
  /x86_64/hmp/pc-i440fx-rhel7.4.0:                                     OK
  /x86_64/hmp/pc-q35-rhel7.5.0:                                        OK
  /x86_64/hmp/none+2MB:                                                OK
PASS: tests/test-hmp
cd /builddir/build/BUILD/qemu-2.12.0/tests/decode && ./check.sh "/usr/libexec/platform-python -B" "/builddir/build/BUILD/qemu-2.12.0/scripts/decodetree.py"
+ pushd tests/qemu-iotests
~/build/BUILD/qemu-2.12.0/tests/qemu-iotests ~/build/BUILD/qemu-2.12.0
+ ./check -v -raw 001 002 004 005 008 009 010 011 012 021 025 032 033 048 052 063 077 086 101 106 120 140 143 145 150 159 160 162 170 171 175 184 221 226
./check: line 84: hostname: command not found
QEMU          -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-img" 
QEMU_IO       -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-io"  --cache writeback -f raw
QEMU_NBD      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-nbd" 
IMGFMT        -- raw
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64  4.14.0-49.el7a.noaead.x86_64
TEST_DIR      -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/socket_scm_helper
001        
002        
004        
005        
008        
009        
010        
011        
012        
021        
025        
032        
033        
048        
052        
063        
077        
086        
101        
106        
120        
140        
143        
145        
150        
159        
160        
162        
170        
171        
175        
184        
221        
226        
Passed all 34 tests
+ ./check -v -qcow2 001 002 004 005 008 009 010 011 012 017 018 019 020 021 024 025 027 028 029 032 033 034 035 037 038 042 046 047 048 050 052 053 058 062 063 066 068 069 072 073 074 086 087 089 090 095 098 102 103 105 107 108 110 111 120 127 130 133 134 138 140 141 143 144 145 150 154 156 158 159 162 170 177 179 182 184 188 190 195 204 209 217 226
./check: line 84: hostname: command not found
QEMU          -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-img" 
QEMU_IO       -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-io"  --cache writeback -f qcow2
QEMU_NBD      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-nbd" 
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64  4.14.0-49.el7a.noaead.x86_64
TEST_DIR      -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/socket_scm_helper
001        
002        
004        
005        
008        
009        
010        
011        
012        
017        
018        
019        
020        
021        
024        
025        
027        
028        
029        
032        
033        
034        
035        
037        
038        
042        
046        
047        
048        
050        
052        
053        
058        
062        
063        
066        
068        
069        
072        
073        
074        
086        
087        
089        
090        
095        
098        
102        
103        
105        
107        
108        
110        
111        
120        
127        
130        
133        
134        
138        
140        
141        
143        
144        
145        
150        
154        
156        
158        
159        
162        
170        
177        
179        
182        
184        
188        
190        
195        
204        
209        
217        
226        
Passed all 83 tests
~/build/BUILD/qemu-2.12.0
+ popd
+ exit 0


Additional info: 

Please note that the test has passed if there is no other text following the iotest number in the build.log file.

Comment 1 Ademar Reis 2018-12-20 16:52:17 UTC
Can't we use a blacklist instead of a whitelist? The reason is that it would be good to enable any new tests if a backport includes them.

Comment 2 Ademar Reis 2018-12-21 20:35:07 UTC
(In reply to Ademar Reis from comment #1)
> Can't we use a blacklist instead of a whitelist? The reason is that it would
> be good to enable any new tests if a backport includes them.

After discussing this f2f, I've learned that many of the tests are flaky (not stable enough) and odds are some of the new tests will also be flaky. So adding them automatically might cause destabilization of future builds, even if they pass at first.

Also discussed this with Yash, who raised the point that once we make the change to close this BZ, fine-tunning the list will require yet another BZ with an exception flag.

So my suggestion is to introduce the change to run the tests as part of the build process, but don't break the build if a test fail, at least not for now. This can be accomplished by adding the command with a "||:" at the end ("OR nop" in bash). This way we can monitor the official build logs to see if any other tests are failing before we commit to the change.

Comment 8 Danilo de Paula 2019-01-11 15:30:24 UTC
Fix included in qemu-kvm-2.12.0-56.module+el8+2679+29eb8264

Comment 10 CongLi 2019-01-30 10:52:53 UTC
Verified on qemu-kvm-2.12.0-60.module+el8+2725+0ab65287.x86_64.

Check build.log:

cd /builddir/build/BUILD/qemu-2.12.0/tests/decode && ./check.sh "/usr/libexec/platform-python -B" "/builddir/build/BUILD/qemu-2.12.0/scripts/decodetree.py"
~/build/BUILD/qemu-2.12.0/tests/qemu-iotests ~/build/BUILD/qemu-2.12.0
+ pushd tests/qemu-iotests
+ ./check -v -raw 001 002 004 005 008 009 010 011 012 021 025 032 033 048 052 063 077 086 101 106 120 140 143 145 150 159 160 162 170 171 175 184 221 226
./check: line 84: hostname: command not found
QEMU          -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-img" 
QEMU_IO       -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-io"  --cache writeback -f raw
QEMU_NBD      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-nbd" 
IMGFMT        -- raw
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64  4.14.0-49.el7a.noaead.x86_64
TEST_DIR      -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/socket_scm_helper
001        
002        
004        
005        
008        
009        
010        
011        
012        
021        
025        
032        
033        
048        
052        
063        
077        
086        
101        
106        
120        
140        
143        
145        
150        
159        
160        
162        
170        
171        
175        
184        
221        
226        
Passed all 34 tests
+ ./check -v -qcow2 001 002 004 005 008 009 010 011 012 017 018 019 020 021 024 025 027 028 029 032 033 034 035 037 038 042 046 047 048 050 052 053 058 062 063 066 068 069 072 073 074 086 087 089 090 095 098 102 103 105 107 108 110 111 120 127 133 134 138 140 141 143 144 145 150 154 156 158 159 162 170 177 179 182 184 188 190 195 204 209 217 226
QEMU          -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-img" 
QEMU_IO       -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-io"  --cache writeback -f qcow2
QEMU_NBD      -- "/builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/../../qemu-nbd" 
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64  4.14.0-49.el7a.noaead.x86_64
TEST_DIR      -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- /builddir/build/BUILD/qemu-2.12.0/tests/qemu-iotests/socket_scm_helper
001        ./check: line 84: hostname: command not found
002        
004        
005        
008        
009        
010        
011        
012        
017        
018        
019        
020        
021        
024        
025        
027        
028        
029        
032        
033        
034        
035        
037        
038        
042        
046        
047        
048        
050        
052        
053        
058        
062        
063        
066        
068        
069        
072        
073        
074        
086        
087        
089        
090        
095        
098        
102        
103        
105        
107        
108        
110        
111        
120        
127        
133        
134        
138        
140        
141        
143        
144        
145        
150        
154        
156        
158        
159        
162        
170        
177        
179        
182        
184        
188        
190        
195        
204        
209        
217        
226        
Passed all 82 tests
~/build/BUILD/qemu-2.12.0
+ popd
+ exit 0