Bug 2055955

Summary:	[virtual network][vdpa]reboot guest qemu output error info: qemu-kvm: vhost VQ 2 ring restore failed: -1: Operation not permitted (1)
Product:	Red Hat Enterprise Linux 8	Reporter:	Lei Yang <leiyang>
Component:	qemu-kvm	Assignee:	lulu <lulu>
qemu-kvm sub component:	Networking	QA Contact:	Lei Yang <leiyang>
Status:	CLOSED MIGRATED	Docs Contact:
Severity:	low
Priority:	low	CC:	aadam, chayang, jasowang, jinzhao, juzhang, lulu, lvivier, virt-maint, yalzhang, yama, yicui, yihyu
Version:	8.6	Keywords:	MigratedToJIRA, Reopened, Triaged
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-09-22 15:52:17 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lei Yang 2022-02-18 02:50:32 UTC

Description of problem:
Boot a guest with vhost-vdpa device, reboot guest output error info: 
qemu-kvm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)

Version-Release number of selected component (if applicable):
kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64
qemu-kvm-6.2.0-7.module+el8.6.0+14253+42d4e2d6.x86_64
iproute-5.15.0-2.el8.x86_64
openvswitch2.16-2.16.0-51.el8fdp.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot a guest with vhost-vdpa device
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35,memory-backend=mem-machine_mem \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x2 \
-m 29696 \
-object memory-backend-ram,size=29696M,id=mem-machine_mem  \
-smp 32,maxcpus=32,cores=16,threads=1,dies=1,sockets=2  \
-cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel860-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-port-3,addr=0x0  \
-netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
-vnc :1  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=8 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \

2.Reboot guest qemu output error info:
qemu-kvm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)


Actual results:
qemu-kvm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)

Expected results:
qemu no error info output

Additional info:
1. kernel-4.18.0-366.el8.x86_64 works well

Comment 1 jason wang 2022-02-23 04:23:11 UTC

What vdpa device are you using?

This is expected if you're using vp_vdpa but not mlx5.

Thanks

Comment 2 Lei Yang 2022-02-23 05:29:59 UTC

(In reply to jason wang from comment #1)

Hello Jason

> What vdpa device are you using?

I am used "Mellanox + vhost-vdpa"
-device virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-port-3,addr=0x0  \
-netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \

The device info:
# lspci |grep Mell
3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

# lspci -vvv -s 0000:3b:00.0 |grep driver
	Kernel driver in use: mlx5_core

Best Regards
Lei

> 
> This is expected if you're using vp_vdpa but not mlx5.
> 
> Thanks

Comment 3 jason wang 2022-02-23 05:51:28 UTC

(In reply to Lei Yang from comment #2)
> (In reply to jason wang from comment #1)
> 
> Hello Jason
> 
> > What vdpa device are you using?
> 
> I am used "Mellanox + vhost-vdpa"
> -device
> virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> port-3,addr=0x0  \
> -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> 
> The device info:
> # lspci |grep Mell
> 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> Dx]
> 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> Dx]
> 
> # lspci -vvv -s 0000:3b:00.0 |grep driver
> 	Kernel driver in use: mlx5_core
> 
> Best Regards
> Lei

Thanks.

Is this a regression? Can we reproduce this with upstream qemu + kernel?


> 
> > 
> > This is expected if you're using vp_vdpa but not mlx5.
> > 
> > Thanks

Comment 4 Lei Yang 2022-02-23 06:14:57 UTC

(In reply to jason wang from comment #3)
> (In reply to Lei Yang from comment #2)
> > (In reply to jason wang from comment #1)
> > 
> > Hello Jason
> > 
> > > What vdpa device are you using?
> > 
> > I am used "Mellanox + vhost-vdpa"
> > -device
> > virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> > port-3,addr=0x0  \
> > -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> > 
> > The device info:
> > # lspci |grep Mell
> > 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > Dx]
> > 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > Dx]
> > 
> > # lspci -vvv -s 0000:3b:00.0 |grep driver
> > 	Kernel driver in use: mlx5_core
> > 
> > Best Regards
> > Lei
> 
> Thanks.
> 
> Is this a regression? Can we reproduce this with upstream qemu + kernel?

I'm not sure if this is a regression bug. Because it cannot be reproduced on the downstream kernel-4.18.0-366.el8.x86_64. This problem is found in bug 2052849's scratch build kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64.
Based on the above info. Does I need to reproduce this with upstream qemu + kernel?

Thanks
Lei
> 
> > 
> > > 
> > > This is expected if you're using vp_vdpa but not mlx5.
> > > 
> > > Thanks

Comment 5 jason wang 2022-02-23 06:17:49 UTC

(In reply to Lei Yang from comment #4)
> (In reply to jason wang from comment #3)
> > (In reply to Lei Yang from comment #2)
> > > (In reply to jason wang from comment #1)
> > > 
> > > Hello Jason
> > > 
> > > > What vdpa device are you using?
> > > 
> > > I am used "Mellanox + vhost-vdpa"
> > > -device
> > > virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> > > port-3,addr=0x0  \
> > > -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> > > 
> > > The device info:
> > > # lspci |grep Mell
> > > 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > Dx]
> > > 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > Dx]
> > > 
> > > # lspci -vvv -s 0000:3b:00.0 |grep driver
> > > 	Kernel driver in use: mlx5_core
> > > 
> > > Best Regards
> > > Lei
> > 
> > Thanks.
> > 
> > Is this a regression? Can we reproduce this with upstream qemu + kernel?
> 
> I'm not sure if this is a regression bug. Because it cannot be reproduced on
> the downstream kernel-4.18.0-366.el8.x86_64. This problem is found in bug
> 2052849's scratch build kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64.
> Based on the above info. Does I need to reproduce this with upstream qemu +
> kernel?

I think we need to check the recent RHEL kernel to see if it can still reproduced.

If not, we can close this bug.

Thanks

> 
> Thanks
> Lei
> > 
> > > 
> > > > 
> > > > This is expected if you're using vp_vdpa but not mlx5.
> > > > 
> > > > Thanks

Comment 6 jason wang 2022-02-23 09:10:32 UTC

(In reply to jason wang from comment #5)
> (In reply to Lei Yang from comment #4)
> > (In reply to jason wang from comment #3)
> > > (In reply to Lei Yang from comment #2)
> > > > (In reply to jason wang from comment #1)
> > > > 
> > > > Hello Jason
> > > > 
> > > > > What vdpa device are you using?
> > > > 
> > > > I am used "Mellanox + vhost-vdpa"
> > > > -device
> > > > virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> > > > port-3,addr=0x0  \
> > > > -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> > > > 
> > > > The device info:
> > > > # lspci |grep Mell
> > > > 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > Dx]
> > > > 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > Dx]
> > > > 
> > > > # lspci -vvv -s 0000:3b:00.0 |grep driver
> > > > 	Kernel driver in use: mlx5_core
> > > > 
> > > > Best Regards
> > > > Lei
> > > 
> > > Thanks.
> > > 
> > > Is this a regression? Can we reproduce this with upstream qemu + kernel?
> > 
> > I'm not sure if this is a regression bug. Because it cannot be reproduced on
> > the downstream kernel-4.18.0-366.el8.x86_64. This problem is found in bug
> > 2052849's scratch build kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64.
> > Based on the above info. Does I need to reproduce this with upstream qemu +
> > kernel?
> 
> I think we need to check the recent RHEL kernel to see if it can still
> reproduced.
> 
> If not, we can close this bug.

To clarify, I think we need to:

1) test recent rhel8 (e.g 369)
2) if it can't be reproduced, close this bug
3) if it can be reproduced, if possible, do a bisection to find the first bad commit

Thanks

> 
> Thanks
> 
> > 
> > Thanks
> > Lei
> > > 
> > > > 
> > > > > 
> > > > > This is expected if you're using vp_vdpa but not mlx5.
> > > > > 
> > > > > Thanks

Comment 7 jason wang 2022-02-23 09:37:34 UTC

(In reply to jason wang from comment #6)
> (In reply to jason wang from comment #5)
> > (In reply to Lei Yang from comment #4)
> > > (In reply to jason wang from comment #3)
> > > > (In reply to Lei Yang from comment #2)
> > > > > (In reply to jason wang from comment #1)
> > > > > 
> > > > > Hello Jason
> > > > > 
> > > > > > What vdpa device are you using?
> > > > > 
> > > > > I am used "Mellanox + vhost-vdpa"
> > > > > -device
> > > > > virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> > > > > port-3,addr=0x0  \
> > > > > -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> > > > > 
> > > > > The device info:
> > > > > # lspci |grep Mell
> > > > > 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > > Dx]
> > > > > 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > > Dx]
> > > > > 
> > > > > # lspci -vvv -s 0000:3b:00.0 |grep driver
> > > > > 	Kernel driver in use: mlx5_core
> > > > > 
> > > > > Best Regards
> > > > > Lei
> > > > 
> > > > Thanks.
> > > > 
> > > > Is this a regression? Can we reproduce this with upstream qemu + kernel?
> > > 
> > > I'm not sure if this is a regression bug. Because it cannot be reproduced on
> > > the downstream kernel-4.18.0-366.el8.x86_64. This problem is found in bug
> > > 2052849's scratch build kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64.
> > > Based on the above info. Does I need to reproduce this with upstream qemu +
> > > kernel?
> > 
> > I think we need to check the recent RHEL kernel to see if it can still
> > reproduced.
> > 
> > If not, we can close this bug.
> 
> To clarify, I think we need to:
> 
> 1) test recent rhel8 (e.g 369)

Speak too fast. 369 won't contain Cindy's latest sync.

According to https://gitlab.com/redhat/rhel/src/kernel/rhel-8/-/merge_requests/2258

It might be 370.

Let's try upstream directly.

Thanks

> 2) if it can't be reproduced, close this bug
> 3) if it can be reproduced, if possible, do a bisection to find the first
> bad commit
> 
> Thanks
> 
> > 
> > Thanks
> > 
> > > 
> > > Thanks
> > > Lei
> > > > 
> > > > > 
> > > > > > 
> > > > > > This is expected if you're using vp_vdpa but not mlx5.
> > > > > > 
> > > > > > Thanks

Comment 8 Lei Yang 2022-02-24 03:19:31 UTC

(In reply to jason wang from comment #7)
> (In reply to jason wang from comment #6)
> > (In reply to jason wang from comment #5)
> > > (In reply to Lei Yang from comment #4)
> > > > (In reply to jason wang from comment #3)
> > > > > (In reply to Lei Yang from comment #2)
> > > > > > (In reply to jason wang from comment #1)
> > > > > > 
> > > > > > Hello Jason
> > > > > > 
> > > > > > > What vdpa device are you using?
> > > > > > 
> > > > > > I am used "Mellanox + vhost-vdpa"
> > > > > > -device
> > > > > > virtio-net-pci,id=net0,netdev=hostnet0,mac=00:11:22:33:44:00,bus=pcie-root-
> > > > > > port-3,addr=0x0  \
> > > > > > -netdev vhost-vdpa,id=hostnet0,vhostdev=/dev/vhost-vdpa-0 \
> > > > > > 
> > > > > > The device info:
> > > > > > # lspci |grep Mell
> > > > > > 3b:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > > > Dx]
> > > > > > 3b:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6
> > > > > > Dx]
> > > > > > 
> > > > > > # lspci -vvv -s 0000:3b:00.0 |grep driver
> > > > > > 	Kernel driver in use: mlx5_core
> > > > > > 
> > > > > > Best Regards
> > > > > > Lei
> > > > > 
> > > > > Thanks.
> > > > > 
> > > > > Is this a regression? Can we reproduce this with upstream qemu + kernel?
> > > > 
> > > > I'm not sure if this is a regression bug. Because it cannot be reproduced on
> > > > the downstream kernel-4.18.0-366.el8.x86_64. This problem is found in bug
> > > > 2052849's scratch build kernel-4.18.0-367.el8.mr2208_220211_1216.x86_64.
> > > > Based on the above info. Does I need to reproduce this with upstream qemu +
> > > > kernel?
> > > 
> > > I think we need to check the recent RHEL kernel to see if it can still
> > > reproduced.
> > > 
> > > If not, we can close this bug.
> > 
> > To clarify, I think we need to:
> > 
> > 1) test recent rhel8 (e.g 369)
> 
> Speak too fast. 369 won't contain Cindy's latest sync.
> 
> According to
> https://gitlab.com/redhat/rhel/src/kernel/rhel-8/-/merge_requests/2258
> 
Hello Jason

> It might be 370.

Yes, you're right, kernel-4.18.0-370.el8.x86_64 contain Cindy's latest sync.
And through my tests, this bug can only be reproduced on kernel-4.18.0-370.el8.x86_64. The problem is not reproduced on kernel-4.18.0-369.el8.x86_64.

Just simple comparison of the error message, this may be caused by this commit： vdpa/mlx5: Fix tracking of current number of VQs

Best regards
Lei
> 
> Let's try upstream directly.
> 
> Thanks
> 
> > 2) if it can't be reproduced, close this bug
> > 3) if it can be reproduced, if possible, do a bisection to find the first
> > bad commit
> > 
> > Thanks
> > 
> > > 
> > > Thanks
> > > 
> > > > 
> > > > Thanks
> > > > Lei
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > This is expected if you're using vp_vdpa but not mlx5.
> > > > > > > 
> > > > > > > Thanks

Comment 9 Lei Yang 2022-02-24 04:58:38 UTC

Based on Comment 8, add keywords "Regression".

Comment 10 jason wang 2022-03-02 09:43:27 UTC

Hi Lei:

So this is an known issue of current vhost-vDPA:

Qemu needs to synchronize indices after stop, but vDPA doesn't implement the stop. So we use reset as a workaround. But after reset the virtqueue indices is lost.

This can be fixed by:

1) implementing stop, Eugenio is working on that
2) workaround in mlx5

1) seems better.

So what we need to check is if networking still work after stop/cont. If it still work we can safely defer it to next release since it is required only for migration.

Thanks

Comment 11 Lei Yang 2022-03-02 11:24:53 UTC

(In reply to jason wang from comment #10)
> Hi Lei:
> 
> So this is an known issue of current vhost-vDPA:
> 
> Qemu needs to synchronize indices after stop, but vDPA doesn't implement the
> stop. So we use reset as a workaround. But after reset the virtqueue indices
> is lost.
> 
> This can be fixed by:
> 
> 1) implementing stop, Eugenio is working on that
> 2) workaround in mlx5
> 
> 1) seems better.
> 
> So what we need to check is if networking still work after stop/cont. If it
> still work we can safely defer it to next release since it is required only
> for migration.

Yep, After repeatedly disable/enable, the network can still work.

Test step:
1. # for i in $(seq 100); do modprobe -r virtio_net ; sleep 0.1; modprobe virtio_net ; sleep 0.1 ; done
2. # ping host from guest succeed.

> 
> Thanks

Comment 12 jason wang 2022-03-02 13:51:35 UTC

(In reply to Lei Yang from comment #11)
> (In reply to jason wang from comment #10)
> > Hi Lei:
> > 
> > So this is an known issue of current vhost-vDPA:
> > 
> > Qemu needs to synchronize indices after stop, but vDPA doesn't implement the
> > stop. So we use reset as a workaround. But after reset the virtqueue indices
> > is lost.
> > 
> > This can be fixed by:
> > 
> > 1) implementing stop, Eugenio is working on that
> > 2) workaround in mlx5
> > 
> > 1) seems better.
> > 
> > So what we need to check is if networking still work after stop/cont. If it
> > still work we can safely defer it to next release since it is required only
> > for migration.
> 
> Yep, After repeatedly disable/enable, the network can still work.
> 
> Test step:
> 1. # for i in $(seq 100); do modprobe -r virtio_net ; sleep 0.1; modprobe
> virtio_net ; sleep 0.1 ; done
> 2. # ping host from guest succeed.
> 
> > 
> > Thanks

Thanks, so let's move it to 8.7/9.1.

Thanks

Comment 13 jason wang 2022-03-02 13:52:21 UTC

Note that it's not an regression, it's introduced by the cvq/mq support, so we probably can remove the "regression tag".

Thanks

Comment 14 Lei Yang 2023-04-10 10:52:23 UTC

Hi All


Hit a similar issue when QE test on the qemu-kvm-8.0.0-rc1, qemu output error messages "qemu-kvm: vhost VQ 2 ring restore failed: -1: Operation not permitted (1)". It also occurs after reboot or shutdown vm. So maybe they are the same issues. Let's track this issue with the current bug first and wait for what happens in downstream official qemu-kvm 8.0.

Thanks
Lei

Comment 15 Lei Yang 2023-04-23 01:10:16 UTC

(In reply to Lei Yang from comment #14)
> Hi All
> 
> 
> Hit a similar issue when QE test on the qemu-kvm-8.0.0-rc1, qemu output
> error messages "qemu-kvm: vhost VQ 2 ring restore failed: -1: Operation not
> permitted (1)". It also occurs after reboot or shutdown vm. So maybe they
> are the same issues. Let's track this issue with the current bug first and
> wait for what happens in downstream official qemu-kvm 8.0.
> 
> Thanks
> Lei

Test on the official downstream version:qemu-kvm-8.0.0-1.el9.x86_64 , it also hit this problem. But from the QE perspective, they should be the same problem. Please correct me if I'm wrong.

Comment 17 RHEL Program Management 2023-08-18 07:28:20 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 19 RHEL Program Management 2023-09-22 15:50:54 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 20 RHEL Program Management 2023-09-22 15:52:17 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.