Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1899530

Summary: NVDIMM: Windows VMs cannot start NVDIMM device.
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Nisim Simsolo <nsimsolo>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
qemu-kvm sub component: Devices QA Contact: Yumei Huang <yuhuang>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: unspecified CC: ahadas, areis, bugs, chayang, coli, jinzhao, juzhang, lijin, mzamazal, nsimsolo, stefanha, virt-maint, vrozenfe, yvugenfi
Version: 8.3Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-07 07:27:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
VM screenshot
none
vdsm.log
none
engine.log
none
qemu log
none
Windows 16 VM domxml
none
MS document - DSM interface associated with virtual NVDIMM ACPI Namespace Devices none

Description Nisim Simsolo 2020-11-19 13:41:24 UTC
Description of problem:
- When running Windows VM (2016 server and 2019 server) with NVDIMM device, the VM is recognizing NVDIMM devices as "storage class memory bus" with exclamation mark stating that: "This device cannot start. (Code 10)
                    Invalid data for the target was specified"

- This issue is true for all NVDIMM namespace modes (raw, sector, fsdax and devdax).
- selinux was set to permissive when Windows VM ran.
- when updating Windows driver of NVDIMM device the update states that the best driver for this device is already installed.
- both Windows VMs (2016 and 2019 servers) are running with virtio-win drivers.

Version-Release number of selected component (if applicable):
ovirt-engine-4.4.3.8-0.1.el8ev
vdsm-4.40.35-1.el8ev.x86_64
libvirt-daemon-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64
qemu-kvm-5.1.0-13.module+el8.3.0+8382+afc3bbea.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Run Windows VM with hostdev NVDIMM in dax mode.
2. Open VM console -> device manager
3.

Actual results:
NVDIMM device is listed as "storage class memory bus" with ! stating that this device cannot start.

Expected results:


Additional info:
VM screenshot, qemu log, 
vdsm.log (2020-11-19 08:08:05,528-0500 INFO  (jsonrpc/6) [api.virt] START create(vmParams={'vmId': '3c707284-cd4a-451e-bd3d-ec1fd5b366d6')
and engine.log (2020-11-19 15:08:05,638+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-133716) [] EVENT_ID: USER_STARTED_VM(153), VM 1_NVDIMM_Win16 was started)
attached.

Comment 1 Nisim Simsolo 2020-11-19 13:44:03 UTC
Created attachment 1730938 [details]
VM screenshot

Comment 2 Nisim Simsolo 2020-11-19 13:48:40 UTC
Created attachment 1730943 [details]
vdsm.log

Comment 3 Nisim Simsolo 2020-11-19 13:50:40 UTC
Created attachment 1730944 [details]
engine.log

Comment 4 Nisim Simsolo 2020-11-19 13:51:18 UTC
Created attachment 1730945 [details]
qemu log

Comment 5 Nisim Simsolo 2020-11-19 13:51:56 UTC
Created attachment 1730946 [details]
Windows 16 VM domxml

Comment 6 Yvugenfi@redhat.com 2020-11-23 13:52:56 UTC
Code 10, means that driver returned the error during its initialisation.
The error is: "Invalid data for the target was specified" which is 0xC014000F STATUS_ACPI_INVALID_DATA.
From other MS driver samples I can see that this is an error that the driver will return when it gets some unexpected or invalid data from ACPI tables.

I am continuing to investigate the issue, for now I can reproduce it without actual NVDIMM on the host as well.

Best regards,
Yan.

Comment 7 Yvugenfi@redhat.com 2020-11-23 13:56:28 UTC
Created attachment 1732553 [details]
MS document - DSM interface associated with virtual NVDIMM ACPI Namespace Devices

Attaching Microsoft document - "_DSM interface associated with virtual NVDIMM ACPI Namespace Devices"

Comment 8 Vadim Rozenfeld 2020-11-24 11:50:28 UTC
I've seen this problem as well when scmbus (storage class memeoru bus driver) was failing to create FDO objects due to 
"Missing NFIT table" problem.

btw, it will be interesting to dump acpi tables on a real NVDIMM ready system like DELL 640 to use them as a reference. 

Vadim.

Comment 11 Yumei Huang 2021-03-08 03:54:31 UTC
It's a known issue since NVDIMM is supported in virtualization. I was told "Intel and Microsoft are working to solve the issue but the fix will be in a future RHEL release".

Cc Stefan, who might have a better understanding of the issue and the progress.

Comment 13 Stefan Hajnoczi 2021-03-17 11:37:46 UTC
(In reply to Yumei Huang from comment #11)
> It's a known issue since NVDIMM is supported in virtualization. I was told
> "Intel and Microsoft are working to solve the issue but the fix will be in a
> future RHEL release".
> 
> Cc Stefan, who might have a better understanding of the issue and the
> progress.

Intel and Microsoft developed two separate ACPI NVDIMM interfaces.

The Intel equivalent to the ACPI _DSM document that Yan posted is https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf

Maybe Intel offers NVDIMM drivers for Windows - they might work with QEMU's NVDIMM device?

I don't know the status of efforts to standardize.

I'm also not sure whether QEMU could offer both (either at the same time for maximum compatibility or by leaving the choice to the user if they are mutually exclusive).

I'll send an email to folks who might know.

Comment 14 Stefan Hajnoczi 2021-03-17 12:56:34 UTC
Here is the mailing list discussion. I'm hoping to get input from folks who know the status of these two ACPI _DSM interfaces:
https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg06227.html

Comment 15 Stefan Hajnoczi 2021-03-31 10:18:11 UTC
The outcome of this discussion is that the Windows native drivers support NVDIMM-N while the device emulated by QEMU is NVDIMM-P. They are not compatible.

It's not clear to me whether NVDIMM-P drivers are available for Windows and work with QEMU's emulated device. I suggest checking that.

If there are no drivers then it may be possible for QEMU to emulate NVDIMM-N or the Hyper-V NVDIMM interface instead. Unfortunately this is a guest-visible change and may cause compatibility problems with non-Windows guests that lack support for those devices.

Comment 18 RHEL Program Management 2021-09-07 07:27:05 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.