Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 890460

Summary: [HCK][SVVP] Can not do kernel debug via -serial
Product: Red Hat Enterprise Linux 6 Reporter: Mike Cao <bcao>
Component: qemu-kvmAssignee: Yvugenfi <yvugenfi>
Status: CLOSED CANTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: acathrow, areis, bcao, bsarathy, drjones, juzhang, michen, mkenneth, rhod, virt-maint, vrozenfe, yvugenfi
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-23 09:54:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screendump for job failed
none
Error w/ no symbols found none

Description Mike Cao 2012-12-27 05:09:01 UTC
Description of problem:


Version-Release number of selected component (if applicable):
2.6.32-338.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.331.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Setting up the host VM
CLI:/usr/libexec/qemu-kvm --nodefaults -M rhel6.4.0 -m 2G -smp 4 -cpu cpu64-rhel6,+x2apic -usbdevice tablet -drive file=win2012-debug2.raw,format=raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device ide-drive,drive=drive-virtio0,id=virtio-blk-pci0,bootindex=1 -netdev tap,sndbuf=0,id=hostnet0,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet0,mac=00:11:22:46:13:28,bus=pci.0,addr=0x4 -boot c -uuid fc722bfa-c0cf-4f2d-afd3-6f3bac0bdd0a -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/win2012_debug2,server,nowait -mon chardev=111a,mode=readline -name win2k3-32-blk-48 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -monitor stdio -vnc :22 -vga cirrus -serial tcp:0:5999,server
2.Start Target VM:
CLI:/usr/libexec/qemu-kvm --nodefaults -M rhel6.4.0 -m 2G -smp 4 -cpu cpu64-rhel6,+x2apic -usbdevice tablet -drive file=win2012-debug1.raw,format=raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device ide-drive,drive=drive-virtio0,id=virtio-blk-pci0,bootindex=1 -netdev tap,sndbuf=0,id=hostnet0,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet0,mac=00:52:22:46:13:28,bus=pci.0,addr=0x4 -boot c -uuid 8742c6e5-4ef3-4bd3-b748-27a0fa8f19db -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/win2012_debug1,server,nowait -mon chardev=111a,mode=readline -name win2k3-32-blk-48 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -monitor stdio -vnc :11 -vga cirrus -serial tcp:0:5999
3.install windbg in the host VM
4.use bcdedit to start debug mode in Target VM.
  
Actual results:
Can not do kernel debug debug ,the target VM will hang after opening debug mode

Expected results:
no hang ,we can see kernel info in the host VM

Additional info:
either this bug or https://bugzilla.redhat.com/show_bug.cgi?id=874406 should be fixed in RHEL6.4.0

Comment 2 Yvugenfi@redhat.com 2012-12-30 15:47:42 UTC
What version of WinDbg was used?

I used version 6.12.0002.633 (this is old version, I will try to reproduce with 6.2.xxx)

And following bcdedit commands:
bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200
bcdedit /debug ON

I cannot reproduce the issue.
One more thing - first connection can take a lot of time if you configured Microsoft symbols server in WinDbg.

Best regards,
Yan.

Comment 3 Mike Cao 2012-12-31 03:16:14 UTC
(In reply to comment #2)
> What version of WinDbg was used?
> 
> I used version 6.12.0002.633 (this is old version, I will try to reproduce
> with 6.2.xxx)

6.2 (the default one in HCK controller)
> 
> And following bcdedit commands:
> bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200
> bcdedit /debug ON

Which guest need do above commands ,the  guest need to be debug ,right ?

BTW ,I install the windbg in the guests booted  with -serial tcp:0:5888,server
in another guest I did not find com1 which I want to debug (start it with -serial tcp:0:5888)

The guest which start with -serial tcp:0:5888 hang more than 2 hours ,and I did not find symbols download on the host which cli with -serial tcp:0:5888,server

Best Regards,
Mike 

> 
> I cannot reproduce the issue.
> One more thing - first connection can take a lot of time if you configured
> Microsoft symbols server in WinDbg.
> 
> Best regards,
> Yan.

Comment 4 Yvugenfi@redhat.com 2012-12-31 09:13:13 UTC
(In reply to comment #3)
> (In reply to comment #2)

> BTW ,I install the windbg in the guests booted  with -serial
> tcp:0:5888,server
> in another guest I did not find com1 which I want to debug (start it with
> -serial tcp:0:5888)
> 

You will not see the COM1 in device manager on target guest. The debugger is using it and OS doesn't display it. So this is OK.

Comment 5 RHEL Program Management 2013-01-05 06:47:19 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 8 Andrew Jones 2013-01-11 14:35:43 UTC
(In reply to comment #2)
> I cannot reproduce the issue.

I also didn't see this hang when I was debugging a win2012 guest using windbg from another. So I'm not sure we have an issue here.

Comment 9 Mike Cao 2013-01-12 04:04:40 UTC
(In reply to comment #8)
> (In reply to comment #2)
> > I cannot reproduce the issue.
> 
> I also didn't see this hang when I was debugging a win2012 guest using
> windbg from another. So I'm not sure we have an issue here.

Which version windbg you are using ?
I still can reproduce it ,and according to SVVP Test job logs ,it shows guest did not connect to debug host successfully.

Mike

Comment 11 Mike Cao 2013-01-14 09:09:36 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> 
> > BTW ,I install the windbg in the guests booted  with -serial
> > tcp:0:5888,server
> > in another guest I did not find com1 which I want to debug (start it with
> > -serial tcp:0:5888)
> > 
> 
> You will not see the COM1 in device manager on target guest. The debugger is
> using it and OS doesn't display it. So this is OK.

Hi, Yan

I still think this is the point .I tried the following steps on a "testing environment" 

1.I start debug host w/ -chardev socket,id=debug,host=0.0.0.0,port=5888,server,nowait -device isa-serial,chardev=debug,id=serial_debug
2. start SUT host w/ chardev socket,id=debug,host=10.66.9.83,port=5888 -device isa-serial,chardev=debug,id=serial_debug 

I find that COM1 can be found in the guest ,then make the job pass 

But what confused me is that I tried to add those paramters on my *SVVP environment*,But COM1 will not show in the device manager ,which cause the the job failed 

Any idea how to make COM1 visible to the SUT ?

thanks,
Mike

Comment 12 Vadim Rozenfeld 2013-01-14 09:26:17 UTC
As Yan said already, Windows will not enumerate serial port, if it was assigned to kernel mode debugger. From that point kdcom owns the serial port, and nobody else. You will not see it in the device manager dialog.
Vadim.

Comment 15 Yvugenfi@redhat.com 2013-01-15 09:59:34 UTC
(In reply to comment #11)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > (In reply to comment #2)
> > 
> > > BTW ,I install the windbg in the guests booted  with -serial
> > > tcp:0:5888,server
> > > in another guest I did not find com1 which I want to debug (start it with
> > > -serial tcp:0:5888)
> > > 
> > 
> > You will not see the COM1 in device manager on target guest. The debugger is
> > using it and OS doesn't display it. So this is OK.
> 
> Hi, Yan
> 
> I still think this is the point .I tried the following steps on a "testing
> environment" 
> 
> 1.I start debug host w/ -chardev
> socket,id=debug,host=0.0.0.0,port=5888,server,nowait -device
> isa-serial,chardev=debug,id=serial_debug
> 2. start SUT host w/ chardev socket,id=debug,host=10.66.9.83,port=5888
> -device isa-serial,chardev=debug,id=serial_debug 
> 
> I find that COM1 can be found in the guest ,then make the job pass 
> 
> But what confused me is that I tried to add those paramters on my *SVVP
> environment*,But COM1 will not show in the device manager ,which cause the
> the job failed 
> 
> Any idea how to make COM1 visible to the SUT ?
> 
> thanks,
> Mike

Hi Mike,

So I have a question - do you configure the guest for the debugging before the test (all the "bcdedit -debug on" and etc commands) or you let the test do it?

Best regards,
Yan.

Comment 16 Mike Cao 2013-01-15 10:11:21 UTC
(In reply to comment #15)
> (In reply to comment #11)
> > (In reply to comment #4)
> > > (In reply to comment #3)
> > > > (In reply to comment #2)
> > > 
> > > > BTW ,I install the windbg in the guests booted  with -serial
> > > > tcp:0:5888,server
> > > > in another guest I did not find com1 which I want to debug (start it with
> > > > -serial tcp:0:5888)
> > > > 
> > > 
> > > You will not see the COM1 in device manager on target guest. The debugger is
> > > using it and OS doesn't display it. So this is OK.
> > 
> > Hi, Yan
> > 
> > I still think this is the point .I tried the following steps on a "testing
> > environment" 
> > 
> > 1.I start debug host w/ -chardev
> > socket,id=debug,host=0.0.0.0,port=5888,server,nowait -device
> > isa-serial,chardev=debug,id=serial_debug
> > 2. start SUT host w/ chardev socket,id=debug,host=10.66.9.83,port=5888
> > -device isa-serial,chardev=debug,id=serial_debug 
> > 
> > I find that COM1 can be found in the guest ,then make the job pass 
> > 
> > But what confused me is that I tried to add those paramters on my *SVVP
> > environment*,But COM1 will not show in the device manager ,which cause the
> > the job failed 
> > 
> > Any idea how to make COM1 visible to the SUT ?
> > 
> > thanks,
> > Mike
> 
> Hi Mike,
> 
> So I have a question - do you configure the guest for the debugging before
> the test (all the "bcdedit -debug on" and etc commands) or you let the test
> do it?
> 
> Best regards,
> Yan.

Hi, Yan

For manual test ,yes 

for WHQL test ,No ,there is a pre-config called bcdedit configuration .I think it will be set in this step.
Seems Vadim think it may fail due to guest can not connect to symbol server ,referring to the screendump

Comment 17 Mike Cao 2013-01-15 10:13:24 UTC
Created attachment 678700 [details]
screendump for  job failed

Comment 18 Mike Cao 2013-01-15 10:31:06 UTC
This is the log for job passed
Runtime Index:   1527248882 
Machine:   M-12-E2 
Process Name:   C:\WLK\JobsWorkingDir\Tasks\WTTJobRunC35F0C03-3009-40C5-B027-30D14DBE1B85\te.exe 
Process ID:   2908 
Thread ID:   2420 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Create the debug client: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Query IDebugControl interface: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Attach to the kernel debugger, most failures here are due to invalid connection parameters: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Set output callbacks: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Set event callbacks: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:01.622 PM SUCCEEDED(hr): Get execution status: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:01.622 PM SUCCEEDED(hr): Send break-in request: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:38.622 PM SUCCEEDED(hr): Get execution status: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:38.622 PM AreEqual(Status, (ULONG)DEBUG_STATUS_GO): Resume the target WexContext Verify 
 
End Test 1/14/2013 3:01:38.513 PM KernelDebugLogoTest::KernelDebugLogoTest1 
Result:   Pass

Comment 19 Mike Cao 2013-01-15 10:33:25 UTC
This is the log for job failed 

This is the log for job passed
Runtime Index:   1527248882 
Machine:   M-12-E2 
Process Name:   C:\WLK\JobsWorkingDir\Tasks\WTTJobRunC35F0C03-3009-40C5-B027-30D14DBE1B85\te.exe 
Process ID:   2908 
Thread ID:   2420 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Create the debug client: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Query IDebugControl interface: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Attach to the kernel debugger, most failures here are due to invalid connection parameters: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Set output callbacks: 00000000 WexContext Verify 
 
Message 1/14/2013 3:00:31.622 PM SUCCEEDED(hr): Set event callbacks: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:01.622 PM SUCCEEDED(hr): Get execution status: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:01.622 PM SUCCEEDED(hr): Send break-in request: 00000000 WexContext Verify 
 
Message 1/14/2013 3:01:38.622 PM The Test timed out waiting for the kernel debugger to break in 
WexContext Verify

sdktools\debuggers\kdit\testbin\kdit.cpp

0x0
Error 0x00000000

KernelDebugLogoTest::KernelDebugLogoTest1
Fail

Comment 20 Mike Cao 2013-01-15 11:19:28 UTC
Created attachment 678725 [details]
Error w/ no symbols found

Seems Debug Compability Test job failed due to no symbols file found. 
Referring to the screendump

Comment 21 Yvugenfi@redhat.com 2013-01-16 09:17:42 UTC
(In reply to comment #20)
> Created attachment 678725 [details]
> Error w/ no symbols found
> 
> Seems Debug Compability Test job failed due to no symbols file found. 
> Referring to the screendump

If this is a case - then try to do the following.

Define environment variable _NT_SYMBOL_PATH and set it to: SRV*C:\symbols\websymbols*http://msdl.microsoft.com/download/symbols

Before that also create C:\symbols\websymbols directory to be a placeholder for the downloaded symbols.

Run the test twice if it fails in a first time as initial download of the symbols might take considerable time.

To define environment variable: Right click on "Computer" icon -> Properties -> Advanced system settings -> Environment variables -> click "New..." under System variables.

Comment 25 Mike Cao 2013-01-17 09:33:45 UTC
After changing CLI to -chardev .. -device isa-serial ,and make Target VM and debug VM to the same host .the original issue still existed ,but can not 100% reproduce.

For the Debug Capability Test job ,seems using serial protocol ,ntkrnlmp.pdb is must .After download this file from kernel debug manually ,the job can pass successfully.

For comment #22 ,I think I download the wrong symbols ,windows server 2012 may do windows update itself during daily testing in workgroup environment 

Based on above ,this seems not to be a testblocker anymore 
Remove Testblocker flag.

Comment 26 Ademar Reis 2013-01-17 21:56:46 UTC
(In reply to comment #25)
> Based on above ,this seems not to be a testblocker anymore 
> Remove Testblocker flag.

If it's not a testblocker, we can defer it to 6.5, because the feature itself is not important to most customers. Thanks.

Comment 27 Yvugenfi@redhat.com 2013-01-27 16:17:48 UTC
(In reply to comment #25)
> After changing CLI to -chardev .. -device isa-serial ,and make Target VM and
> debug VM to the same host .the original issue still existed ,but can not
> 100% reproduce.
> 
> For the Debug Capability Test job ,seems using serial protocol ,ntkrnlmp.pdb
> is must .After download this file from kernel debug manually ,the job can
> pass successfully.
> 
> For comment #22 ,I think I download the wrong symbols ,windows server 2012
> may do windows update itself during daily testing in workgroup environment 
> 
> Based on above ,this seems not to be a testblocker anymore 
> Remove Testblocker flag.

As the feature itself works outside the test environment, can we document and close?

Thanks,
Yan.

Comment 28 Yvugenfi@redhat.com 2013-01-27 16:17:59 UTC
(In reply to comment #25)
> After changing CLI to -chardev .. -device isa-serial ,and make Target VM and
> debug VM to the same host .the original issue still existed ,but can not
> 100% reproduce.
> 
> For the Debug Capability Test job ,seems using serial protocol ,ntkrnlmp.pdb
> is must .After download this file from kernel debug manually ,the job can
> pass successfully.
> 
> For comment #22 ,I think I download the wrong symbols ,windows server 2012
> may do windows update itself during daily testing in workgroup environment 
> 
> Based on above ,this seems not to be a testblocker anymore 
> Remove Testblocker flag.

As the feature itself works outside the test environment, can we document and close?

Thanks,
Yan.

Comment 29 Mike Cao 2013-01-28 08:50:57 UTC
(In reply to comment #28)
> (In reply to comment #25)
> > After changing CLI to -chardev .. -device isa-serial ,and make Target VM and
> > debug VM to the same host .the original issue still existed ,but can not
> > 100% reproduce.
> > 
> > For the Debug Capability Test job ,seems using serial protocol ,ntkrnlmp.pdb
> > is must .After download this file from kernel debug manually ,the job can
> > pass successfully.
> > 
> > For comment #22 ,I think I download the wrong symbols ,windows server 2012
> > may do windows update itself during daily testing in workgroup environment 
> > 
> > Based on above ,this seems not to be a testblocker anymore 
> > Remove Testblocker flag.
> 
> As the feature itself works outside the test environment, can we document
> and close?

I am afraid not .Comment #25 is just a workaround and it is not 100% work.We hit this issue again today .it is really consume lots of QE's resources.

Mike