Bug 1365721
| Summary: | Some nvml built-in tests failed with NVDIMM device in guest | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yumei Huang <yuhuang> |
| Component: | qemu-kvm-rhev | Assignee: | Stefan Hajnoczi <stefanha> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Yumei Huang <yuhuang> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.3 | CC: | chayang, jinzhao, jmoyer, juzhang, knoel, michen, pagupta, parmeshwr_prasad, qzhang, Robert.Hu, stefanha, virt-maint, xuelian.guo, yuhuang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-rhev-2.12.0-20.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-12-11 14:40:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Yumei Huang
2016-08-10 03:16:38 UTC
(In reply to Yumei Huang from comment #0) Thanks for posting these failed nvml test cases. They aren't obvious to me so I will bring them to the attention of Guangrong Xiao who is writing the patches upstream. Let's leave this BZ open to track these test failures but it doesn't need to block RHEL 7.3 since NVDIMM is not yet feature-complete. I reproduced the failures here too. NVDIMM is still under development. I'm moving this bug to RHEL 7.5 so it will be checked again in the future. Hi Yumei, Please rerun the NVDIMM test suite so we know the status for this release cycle. Eventually upstream will get all tests passing. We don't need to do anything except continue to track the progress for now. Thank you! Hi Stefan, I hit an error "PMEM_FS_DIR=/mnt/pmem does not point to a PMEM device" when running the test suite, and I did mount /dev/pmem0 to /mnt/pmem in guest. Details: qemu-kvm-rhev-2.10.0-9.el7 Cmdline: # /usr/libexec/qemu-kvm -m 8G,slots=240,maxmem=20G -smp 16 \ rhel75-64-virtio.qcow2 -M pc,nvdimm=on \ -object memory-backend-file,id=mem1,share=on,mem-path=/tmp/nv0,size=4G -device nvdimm,memdev=mem1,id=nv1 \ -netdev tap,id=hostnet1 -device virtio-net-pci,mac=42:ce:a9:d2:4d:d9,id=idlbq7eA,netdev=hostnet1 \ -no-user-config -nodefaults -numa node -numa node -usb -device usb-tablet,id=input0 -vga qxl -vnc :4 -monitor stdio In guest: # ll /dev/pmem0 brw-rw----. 1 root disk 259, 0 Nov 30 13:59 /dev/pmem0 # mkfs.xfs /dev/pmem0 # mount -o dax /dev/pmem0 /mnt/pmem/ # cat nvml/src/test/testconfig.sh ... PMEM_FS_DIR=/mnt/pmem ... # cd nvml/src/test # ./RUNTESTS ... error: PMEM_FS_DIR=/mnt/pmem does not point to a PMEM device RUNTESTS: stopping: blk_nblock/TEST0 failed, TEST=check FS=any BUILD=debug Hi, Can you please try with this option enabled? # If you don't have real PMEM or PMEM emulation set up, but still want to test # PMEM codepaths uncomment this line. It will set PMEM_IS_PMEM_FORCE to 1 for # tests that require pmem. # PMEM_FS_DIR_FORCE_PMEM=1 (In reply to pagupta from comment #10) > Hi, > > Can you please try with this option enabled? > > > # If you don't have real PMEM or PMEM emulation set up, but still want to > test > # PMEM codepaths uncomment this line. It will set PMEM_IS_PMEM_FORCE to 1 for > # tests that require pmem. > # > > PMEM_FS_DIR_FORCE_PMEM=1 With this option enabled, all tests passed except "obj_ctl_prefault" and "ex_libpmemobj_cpp". # ./RUNTESTS ex_libpmemobj_cpp ex_libpmemobj_cpp/TEST0: SETUP (check/pmem/debug) ../unittest/unittest.sh: line 727: ../../examples/libpmemobj/cpp/queue: No such file or directory ex_libpmemobj_cpp/TEST0 failed with exit code 127. RUNTESTS: stopping: ex_libpmemobj_cpp/TEST0 failed, TEST=check FS=pmem BUILD=debug # ./RUNTESTS obj_ctl_prefault obj_ctl_prefault/TEST0: SETUP (check/pmem/debug) obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: PASS [00.173 s] obj_ctl_prefault/TEST0: SETUP (check/pmem/nondebug) obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: PASS [00.068 s] obj_ctl_prefault/TEST0: SETUP (check/pmem/static-debug) obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: PASS [00.188 s] obj_ctl_prefault/TEST0: SETUP (check/pmem/static-nondebug) obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: PASS [00.062 s] obj_ctl_prefault/TEST0: SETUP (check/non-pmem/debug) obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault obj_ctl_prefault/TEST0: START: obj_ctl_prefault RUNTESTS: stopping: obj_ctl_prefault/TEST0 failed, TEST=check FS=non-pmem BUILD=debug I guess you have : libpmemobj-devel, libpmemobj packages and other dependencies? Pankaj (In reply to pagupta from comment #12) > I guess you have : libpmemobj-devel, libpmemobj packages and other > dependencies? > > Pankaj Yes, I have installed libpmemobj-devel, libpmemobj packages in guest. I don't think the failed cases are because of dependencies. Some cases skipped instead of failed when lack of dependencies or configuration , e.g. out_err_mt/TEST1: SKIP valgrind-devel package (ver 3.7 or later) required pmempool_check/TEST20: SKIP DEVICE_DAX_PATH does not specify enough dax devices (min: 1) pmem_movnt_align/TEST4: SKIP not compiled with support for Valgrind pmemcheck And I don't understand why need to enable PMEM_FS_DIR_FORCE_PMEM for the test. Is it trying to find out if the test suite works well when no pmem device? Thanks, Yumei Huang Hello Yumei, Thanks for testing this. It looks like latest NVML tests only considering device DAX as pmem device. As we are not using device DAX? so we are getting this error. See below commit: https://github.com/pmem/nvml/commit/9831acece3d1883afc764f31ca6f754b6ee44a1d#diff-573f7ef9c3a0387876f4d5e345241b8e > > Yes, I have installed libpmemobj-devel, libpmemobj packages in guest. > > I don't think the failed cases are because of dependencies. Some cases > skipped instead of failed when lack of dependencies or configuration , e.g. > > out_err_mt/TEST1: SKIP valgrind-devel package (ver 3.7 or later) required > > pmempool_check/TEST20: SKIP DEVICE_DAX_PATH does not specify enough dax > devices (min: 1) > > pmem_movnt_align/TEST4: SKIP not compiled with support for Valgrind pmemcheck > > > And I don't understand why need to enable PMEM_FS_DIR_FORCE_PMEM for the > test. Is it trying to find out if the test suite works well when no pmem > device? Reason for this is filesystem DAX does not guarantee persistent data until userspace does explicit msync/fsync (before MAP_SYNC). This document explains some of the details: http://pmem.io/nvml/libpmem/ The case is even different for KVM as we are testing 'fake DAX' range here which is just mmaped file at hostside that require an explicit flush from guest. Still we don't have that feature upstream and we don't have device DAX if there is no hardware. So, better option is to test with 'PMEM_FS_DIR_FORCE_PMEM'. Thanks, Pankaj (In reply to Yumei Huang from comment #13) > And I don't understand why need to enable PMEM_FS_DIR_FORCE_PMEM for the > test. Is it trying to find out if the test suite works well when no pmem > device? The NVML code really wants to know if it can flush to persistence safely from userspace (w/o calling msync). Right now, file systems still require fsync/msync to ensure that metadata is flushed to the storage. There's a patch set that recently went in upstream to allow applications to specify a MAP_SYNC parameter to mmap. That will allow nvml to flush to persistence without calling msync (and at that point in time, you won't have to use the force flag for the tests). The reason device dax is considered "pmem" is that there is no file system, so flush from userspace is safe. Thanks Pankaj and Jeff for your explanation. It helps QE understand better. We will enable option 'PMEM_FS_DIR_FORCE_PMEM' for current test. Thanks! *** Bug 1539541 has been marked as a duplicate of this bug. *** Reproduced on RHEL7.6 Alpha Reproduced on RHEL7.6 Beta.
Note : run ./RUNTESTS
error reproduced on Native and VM.
obj_ctl_prefault/TEST0 failed, TEST=check FS=non-pmem BUILD=debug
No reproduced on RHEL7.6 RC. Based on latest comments, this has been fixed in RHEL-7.6 (comment #10, comment #20 and others). Can you please retest? Tested against qemu-kvm-rhev-2.12.0-20.el7, with PMEM_FS_DIR_FORCE_PMEM=1, all tests passed or skipped, no error. |