Bug 450918 - vmware - Console graphic problem when mouse is moved
vmware - Console graphic problem when mouse is moved
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.7
i686 Linux
high Severity medium
: rc
: ---
Assigned To: Vivek Goyal
: Regression
Depends On:
Blocks: 240187
  Show dependency treegraph
 
Reported: 2008-06-11 14:07 EDT by Brett Lackey
Modified: 2008-07-24 15:30 EDT (History)
16 users (show)

See Also:
Fixed In Version: RHSA-2008-0665
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-07-24 15:30:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
A screenshot of the messages displayed on the console when using the mouse. (123.21 KB, image/jpeg)
2008-06-11 14:07 EDT, Brett Lackey
no flags Details
dmesg (kernel-smp-2.6.9-75.EL.TEST.bz450918.1) (41.00 KB, text/plain)
2008-07-01 15:41 EDT, James Laska
no flags Details
dmesg for TEST.bz450918.kernel (111.05 KB, application/octet-stream)
2008-07-01 15:58 EDT, Wei Li
no flags Details
dmesg (kernel-smp-2.6.9-75.EL.TEST.bz450918.2) (109.54 KB, text/plain)
2008-07-01 20:23 EDT, James Laska
no flags Details
dmesg output for kernel.TEST.bz450918.2 (51.97 KB, text/plain)
2008-07-01 20:24 EDT, Wei Li
no flags Details
dmesg output for TEST.bz450918.3 kernel. (23.97 KB, text/plain)
2008-07-02 19:13 EDT, Wei Li
no flags Details
dmesg output for TEST.bz450918.5 kernel. (23.44 KB, text/plain)
2008-07-03 14:24 EDT, Wei Li
no flags Details

  None (edit)
Description Brett Lackey 2008-06-11 14:07:35 EDT
Description of problem:

When loading a system with AS RHEL 4.7 beta snapshot 1 (32bit or 64bit) in 
VMware Server, the graphic installation goes away and is replaced with DOS 
based text messages such as:

atkbd.c: Unknown key pressed (raw set 2, code 0x115 on isa0060/serio1).
atkbd.c: Use 'setkeycodes 15 <keycode>' to make it known.

This happens during the install of the OS, or any time after the system is 
loaded (loaded with keyboard and mousenot being touched) in the VMware console.


Version-Release number of selected component (if applicable):
[root@linuxbuild-vm64-3 ~]# uname -a
Linux linuxbuild-vm64-3 2.6.9-71.ELsmp #1 SMP Tue May 27 16:42:27 EDT 2008 
x86_64 x86_64 x86_64 GNU/Linux

RHEL4-U7-re20080528.1-x86_64-AS-disc1-ftp.iso

[root@linuxbuild-vm64-3 ~]# rpm -qa | grep -i release
redhat-release-4AS-7.3

[root@linuxbuild-vm64-3 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 7 Beta)



How reproducible:

Install the OS in a VMware Server Console session.


Steps to Reproduce:
1. Create RedHat Virtual System with VMware Server.
2. Install RedHat OS.
3. Move mouse while doing installation.
  
Actual results:

See attached file.

Expected results:

Expected esults are no problems using a mouse in the graphic console session.

Additional info:
Comment 1 Brett Lackey 2008-06-11 14:07:35 EDT
Created attachment 308971 [details]
A screenshot of the messages displayed on the console when using the mouse.
Comment 2 Wei Li 2008-06-19 23:15:43 EDT
The same problem has been encountered on various VMware products, such as
VMware Workstation 6.x and Vmware Server 3.x, when install RHEL4.7Beta AS/WS/ES
VMs.  RHEL4.7Beta WS has 2.6.9-70.EL kernel.

Without touching the mouse, users may install the RHEL4.7Beta guest OS 
in the text mode and login into the text console without X running.  
As long as moving the mouse, the messages of "atkbd.c: Unknown key pressed ..."
are displayed on the console and the keyboard no longer works properly.
Comment 4 Kurt Daverman 2008-06-23 12:54:16 EDT
RHEL 4.7 beta does not follow ps/2 mouse protocol, and starts sending invalid
commands almost immediately. Most systems eventually setup Intellimouse that
supports scroll wheel by sending set sample rate 200, 100, 80 (mouse commands f3
c8 f3 64 f3 50)

mouse command sequence from RHEL 4.7:
-------------------------------------
disable (f5)
reset (ff)
read id (f2) (which returns 00)
unknown command ed ?
unknown command 00 ?
unknown command ed ?
unknown command 00 ?
set rate 0 (f3 00)
enable (f4)
(nothing more)

mouse command sequence from RHEL 5.0:
-------------------------------------
disable (f5)
reset (ff)
read id (f2) (returns 00)
read id (f2) (returns 00)
set defaults (f6)
set rate 10 (f3 0a)
set resolution 0 (e8 00)
set rate 20 (f3 14)
set rate 60 (f3 3c)
set rate 40 (f3 28)
set rate 20 (f3 14)
set rate 20 (f3 14)
set rate 60 (f3 3c)
set rate 40 (f3 28)
set rate 20 (f3 14)
set rate 20 (f3 14)
read id (f2) (returns 00)
set resolution 0 (e8 00)
set resolution 0 (e8 00)
set resolution 0 (e8 00)
set resolution 0 (e8 00)
status request (e9) returns (00 00 14)
set defaults (f6)
set resolution 0 (e8 00)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
status request (e9) returns (00 00 64)
set resolution 0 (e8 00)
set scaling 2:1 (e7)
set scaling 2:1 (e7)
set scaling 2:1 (e7)
status request (e9) returns (10 00 64)
set resolution 3 (e8 03)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
status request (e9) returns (00 03 64)
set resolution 0 (e8 00)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
set scaling 1:1 (e6)
status request (e9) returns (00 00 64)
unknown command e1 ?
reset (ff)
set rate 200 (f3 c8)
set rate 100 (f3 64)
set rate 80 (f3 50)
read id (f2) (returns 03) (Intellimouse supported - 4-byte packets)
...

Sure looks like something basic changed in RHEL 4.7 ps/2 mouse support.
Comment 8 Peter Martuccelli 2008-06-25 14:06:18 EDT
Stratus:  Did this problem occur when installing earlier RHEL 4.7 Beta or
development builds?  Does RHEL 4.6 install correctly?  
Comment 9 Brett Lackey 2008-06-25 14:14:40 EDT
This problem was encountered install RHEL AS 4.0 Update 7 beta using snapshots 
one and two on both 32-bit and 64-bit systems.  This problem does not occur 
using RHEL AS 4.0 Update 6.
Comment 10 Peter Martuccelli 2008-06-26 09:54:55 EDT
Thanks for the update.   Did Stratus test any of the RHEL 4.7 development builds
, e.g. pre-Beta release timeframe?  I will need to narrow down the kernel
revision information as I do not have vmware server available to test on, the
problem will need to be resolved by inspection.
Comment 11 Kurt Daverman 2008-06-26 11:27:43 EDT
Note the command sequence sent to the mouse appears to be keyboard commands:

RHEL 4.7 command sequence sent to mouse
---------------------------------------
disable (f5)   // same for mouse and keyboard
reset (ff)     // same for mouse and keyboard
read id (f2) (which returns 00)  // same for mouse and keyboard
unknown command ed  // unknown mouse - set leds for keyboard
unknown command 00  // unknown mouse - led data for keyboard
unknown command ed  // unknown mouse - set leds for keyboard
unknown command 00  // unknown mouse - led data for keyboard
set rate 0 (f3 00)  // same for mouse and keyboard
enable (f4)  // same for mouse and keyboard 
(nothing more)

So was there some simplification/consolidation in keyboard and mouse initialization?
Comment 12 Tim Burke 2008-06-30 10:54:04 EDT
In an attempt to differentiate whether this is a kernel issue, vs X input
methods, etc... can you try a RHEL4.6 install then drop in the 4.7 kernel?  Or
similarly, try a RHEL4.7 beta install and swap in the 4.6 kernel.
Comment 13 Brett Lackey 2008-06-30 14:25:04 EDT
This appears to be a kernel problem.  I took a virtual RHEL AS 4.6 system 
running a 2.6.9-67.0.7ELsmp kernel, and installed RPM's kernel-smp-2.6.9-
72.EL.x86_64.rpm and kernel-smp-devel-2.6.9-72.EL.x86_64.rpm.  The exact same 
problem is encountered.  Even if I moved the mouse during the text mode booting 
process, I would experience problems.  So this does not appear to be an X 
problem, but rather a kernel problem.
Comment 14 Brett Lackey 2008-06-30 17:23:52 EDT
As an additional test, I took a virtual system running RHEL 4.7 beta snapshot 1 
and installed the U6 kernel of 2.6.9-67.0.7ELsmp (kernel and kernel-devel).  I 
rebooted the virtual system on the U6 kernel and everything worked fine.  There 
were no mouse or console graphic problems with the U6 kernel running on the U7 
beta release.  This helps to prove that the problem seems to be kernel related.
Comment 17 Vivek Goyal 2008-06-30 20:57:40 EDT
Kurt/Brett,

I have built a test kernel with one of the patches removed. That's the only
patch which touches the  
mouse/keyboard related code. We suspect it might have introduced the regression.
To confirm, can you please try following test kernels and see if it resolves the
problem.

Test kernels are available at following site.

http://people.redhat.com/vgoyal/.vmware/

Comment 18 Wei Li 2008-06-30 23:36:16 EDT
I applied the test kernel from the comment#17 in a RHEL4.7-WSbeta VM.
Boot the VM with this kernel, the mouse problem is gone. 
Comment 19 Wei Li 2008-06-30 23:36:29 EDT
I applied the test kernel from the comment#17 in a RHEL4.7-WSbeta VM.
Boot the VM with this kernel, the mouse problem is gone. 
Comment 20 Vivek Goyal 2008-07-01 08:32:05 EDT
Thanks Wei. Good to know that problem is gone after reverting one patch. We will
dive deeper to find out why it is happening.
Comment 21 James Laska 2008-07-01 09:07:49 EDT
+1 on comment#19 ... booting with kernel-smp-2.6.9-76.EL.vgoyal.vmware.i686.rpm
does not cause the console keypress output when moving the mouse around on the
guest display.
Comment 22 David Milburn 2008-07-01 10:04:17 EDT
The messages from Comment #1 are from atkbd_interrupt which would be called
from serio_interrupt based upon what port was passed in from i8042_interrupt.
And in this case it seems i8042_interrupt is treating the mouse movements
as keyboard data. I will need to build a debug kernel to determine how the
system is getting into this state.
Comment 23 Peter Martuccelli 2008-07-01 10:04:33 EDT
Tested the kernel on Jan's setup with no failures to report.
Comment 25 Kurt Daverman 2008-07-01 12:08:59 EDT
With regard to comment#22, I think this occurs because the ps/2 mouse is being
initialized as a keyboard as shown in comment #11.
Comment 26 Kurt Daverman 2008-07-01 12:21:29 EDT
We have found a VMware problem that is related to this issue.  When the invalid
"keyboard" commands are sent to the PS/2 mouse, the mouse responds with ACK
(0xfa) when it should respond with NACK (0xfe).  The ACK allows the invalid
command sequence to continue.  Nevertheless, keyboard commands should not be
routed to the mouse.
Comment 27 David Milburn 2008-07-01 14:55:33 EDT
When removing the polling timer, we did change some of the init code,
I have added some debug code to the kernel-smp-2.6.9-75.EL.TEST.bz450918.1 kernel,
would you please supply the dmesg output after booting up? 

http://people.redhat.com/dmilburn/
Comment 28 James Laska 2008-07-01 15:41:18 EDT
Created attachment 310711 [details]
dmesg (kernel-smp-2.6.9-75.EL.TEST.bz450918.1)
Comment 29 Wei Li 2008-07-01 15:58:43 EDT
Created attachment 310714 [details]
dmesg for TEST.bz450918.kernel
Comment 30 David Milburn 2008-07-01 17:11:16 EDT
This looks odd which is consistent with Comment#25

input: AT Raw Set 2 keyboard on isa0060/serio1
input: AT Translated Set 2 keyboard on isa0060/serio0

James, would you please attach the dmesg output from Comment#21? Thank you
Comment 31 David Milburn 2008-07-01 17:59:25 EDT
James or Wei,

Would you please repeat the test on kernel-smp-2.6.9-75.EL.TEST.bz450918.2 and
supply the dmesg output? Thank you.

http://people.redhat.com/dmilburn/
Comment 32 James Laska 2008-07-01 20:23:34 EDT
Created attachment 310736 [details]
dmesg (kernel-smp-2.6.9-75.EL.TEST.bz450918.2)
Comment 33 Wei Li 2008-07-01 20:24:03 EDT
Created attachment 310738 [details]
dmesg output for kernel.TEST.bz450918.2
Comment 34 David Milburn 2008-07-02 09:51:46 EDT
I thought we may not be hitting all the init code since we moved it from
i8042_open into i8042_init, but that doesn't look to be the case.

The serio driver tries to match each port with the approriate driver, and 
it looks like the atkbd driver is claiming serio1 as a keyboard device.
I will need to add some debug in the atkbd driver's init and probe code to
furthur debug.

input: AT Raw Set 2 keyboard on isa0060/serio1 <===
Comment 37 David Milburn 2008-07-02 14:52:43 EDT
Please retest with kernel-smp-2.6.9-75.EL.TEST.bz450918.3 and supply the dmesg
output, this should give us a better idea on why the atkbd driver is claiming
serio1, thanks.

http://people.redhat.com/dmilburn/
Comment 38 Wei Li 2008-07-02 15:15:21 EDT
(In reply to comment #37)
> Please retest with kernel-smp-2.6.9-75.EL.TEST.bz450918.3 and supply the dmesg

Boot the VM with this kernel, it doesn't response any keypress or mouse movement.
I can't login to get dmesg.  
Comment 39 Wei Li 2008-07-02 19:13:47 EDT
Created attachment 310867 [details]
dmesg output for TEST.bz450918.3 kernel.
Comment 40 David Milburn 2008-07-02 19:33:35 EDT
Wei, looks like we were updating the BZ at the same time, I thought since you
were having problems with the .3 kernel that there maybe too many debug
statements. After Comment #38, I went ahead and built another .4 kernel with
less debugging, though it still looks at the atkbd commands/responses. Also,
I didn't see (input: AT Raw Set 2 keyboard on isa0060/serio1) in Comment #39,
can you please boot the .4 kernel and see if that works better?

http://people.redhat.com/dmilburn/
Comment 41 Wei Li 2008-07-02 19:53:39 EDT
David,

For the comment#39, the keyboard/mouse still didn't work. I managed 
to get the content of dmesg buffer via some other way:)

I'll try your .4 kernel.
Comment 42 Wei Li 2008-07-02 20:34:34 EDT
The keyboard/mouse still doesn't work with .4 kernel as with .3 kernel.
I didn't see "input: AT Raw Set 2 keyboard on isa0060/serio1" or
"input: AT Translated Set 2 keyboard on isa0060/serio0" either in 'messages'.
Comment 43 Wei Li 2008-07-02 20:37:34 EDT
In dmesg for .3 kernel, there is the error

drivers/input/serio/i8042.c: f2 -> i8042 (kbd-data) [4614]
ATKBD_PROBE Get ID command FAILED set LEDS -1515913216
Comment 44 David Milburn 2008-07-03 08:38:59 EDT
Wei, I think the debugging is causing commands to fail, I was specifically
looking for the get ID command, but according to the output sendbyte would
have failed, plus it looks like that was for serio0. So, we really need to
get back to where you can login normally, I will need to narrow down the debug
some more.
Comment 46 RHEL Product and Program Management 2008-07-03 11:19:16 EDT
This bugzilla has Keywords: Regression.  

Since no regressions are allowed between releases, 
it is also being proposed as a blocker for this release.  

Please resolve ASAP.
Comment 47 David Milburn 2008-07-03 11:54:52 EDT
Wei, I have another test kernel (.5) available with minimal debug, you should be
able to login normally and get the dmesg output, this should show the atkbd probe
concerning the get id command for serio0 and serio1. Thanks.

http://people.redhat.com/dmilburn/
Comment 51 Wei Li 2008-07-03 14:24:54 EDT
Created attachment 310945 [details]
dmesg output for TEST.bz450918.5 kernel.

The keyboard works with .5 kernel. I got dmesg output and attach it here.
Comment 52 David Milburn 2008-07-03 15:25:35 EDT
Ok, thanks, the keyboard driver is sending the ATKBD_CMD_GETID to serio1
(before it sends the cmd the driver invalidates the params with 0xa5). And
the command does fail, but, the keyboard driver still treats serio1 as a
keyboard, tries to set the LEDS, returns 0 to atkbd_connect which sets it up as
a keyboard device

"input: AT Raw Set 2 keyboard on isa0060/serio1"

The upstream atkbd driver behaves the same way on the GETID failure, it is also
odd that the patch backed out in Comment#17 didn't change any of the atkbd code.

====Comment #51 dmesg ouput=========================
ATKBD_CONNECT: atkbd->write 1
ATKBD_PROBE serio isa0060/serio1 atkbd_reset 0
ATKBD_COMMAND sending ATKBD_CMD_GETID
ATKBD_COMMAND, ATKBD_CMD_GETID ret 0xa5
ATKBD_PROBE Get ID command FAILED set LEDS     <==ATKBD_CMD_FAILED in probe  
input: AT Raw Set 2 keyboard on isa0060/serio1 <==Still treats as keyboard
ATKBD_CONNECT
ATKBD_CONNECT: TYPE 0x6000000
ATKBD_CONNECT: atkbd->write 1
ATKBD_PROBE serio isa0060/serio0 atkbd_reset 0
ATKBD_COMMAND sending ATKBD_CMD_GETID
ATKBD_COMMAND, ATKBD_CMD_GETID ret 0xab   <==cmd succeeds, ret 0xab
ATKBD_COMMAND, KEYBOARD FOUND 0xab
ATKBD_PROBE serio isa0060/serio0 param[0] 0xab FOUND KEYBOARD
ATKBD_PROBE atkbd->id 43841 atkbd->translated 1
input: AT Translated Set 2 keyboard on isa0060/serio0
=============================================================

Normally, as long as the GETID command succeeds and returns
anything but 0xac or 0xab then the keyboard driver probe would know for sure
that the device isn't a keyboard.

=====dmesg from a x86_64 box====================
ATKBD_PROBE serio isa0060/serio1 atkbd_reset 0
ATKBD_COMMAND sending ATKBD_CMD_GETID
ATKBD_COMMAND, ATKBD_CMD_GETID ret 0x0
ATKBD_COMMAND, NOT A KEYBOARD, ret 0x0  <===succeed, not 0xac or 0xab in param
ATKBD_PROBE serio isa0060/serio1 param[0] 0x0 NOT A KEYBOARD
ATKBD_CONNECT
ATKBD_CONNECT: TYPE 0x6000000
ATKBD_CONNECT: atkbd->write 1
ATKBD_PROBE serio isa0060/serio0 atkbd_reset 0
ATKBD_COMMAND sending ATKBD_CMD_GETID
ATKBD_COMMAND, ATKBD_CMD_GETID ret 0xab <==Succeeded, ret 0xab
ATKBD_COMMAND, KEYBOARD FOUND 0xab
ATKBD_PROBE serio isa0060/serio0 param[0] 0xab FOUND KEYBOARD
ATKBD_PROBE atkbd->id 43841 atkbd->translated 1
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImPS/2 Logitech Wheel Mouse on isa0060/serio1
Comment 53 Vivek Goyal 2008-07-07 14:52:44 EDT
Committed in 77.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/

For the time being this problem has been resolved by reverting the problematic
patch which was committed in 68.12 with following changelog entry.

-i8042: remove polling timer support (David Milburn) [340561]

Revert git commit id is commit 7913eba6ad6a49aa98d0b4ccf38945ca08d0697a

Note: bz 340561 is little misleading here. Above patch had actually solved bz 246233
Comment 56 errata-xmlrpc 2008-07-24 15:30:20 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2008-0665.html

Note You need to log in before you can comment on or make changes to this bug.