Description of problem:

This bug is a follow-up on bug #492529 regarding key accelerators in OpenMotif.

The problem is that in X11R6 CapsLock is seen as a modifier, so the state on a KeyPress event with CapsLock enabled is different from the state without.

The solution is to grab the key with both states, CapsLock on and off (the same goes for other modifiers like NumLock or ScrollLock).

This solution works fine except when using VNC. Within VNC, enabling CapsLock defeats the key accelerators.

The reason for this is that VNC emulates CapsLock with Shift, so the accelerator wil lnot work with CapsLock enabled in VNC.

Another problem is that some applications (for example NEdit) will have different meaning with or without shift (for example, in NEdit, Ctrl+v is "paste" whereas Ctrl+Shift+v is "paste column").

Version-Release number of selected component (if applicable):

vnc-server-4.1.2-14.el5_3.1

How reproducible:

Always

Steps to Reproduce:
1. Make sure the latest openmotif-2.3.1-2.el5_4.1 from RHBA-2010-0132 or later is installed

2. Save and build the attached "accel.c" file

  $ cc -o accel accel.c -L/usr/lib -lXm -lXt -lX11

3. Run the program locally

  $ ./accel

4. Enable CapLock

5. Press Ctrl+w, notice the window closes as expected

6. Redo step 3. 4. and 5. via VNC
  
Actual results:

In VNC the window does not close

Expected results:

The window closes as expected even in VNC

Additional info:

Attaching 2 xev captures, one from locally, the other one via VNC.

Locally, when CapsLock is enabled, the state field of the event reports the modifier:

  KeyPress event, serial 31, synthetic NO, window 0x4200001,
      root 0x10a, subw 0x0, time 13172032, (698,1049), root:(1582,1199),
      state 0x12, keycode 38 (keysym 0x41, A), same_screen YES,
      XLookupString gives 1 bytes: (41) "A"
      XmbLookupString gives 1 bytes: (41) "A"
      XFilterEvent returns: False

  KeyRelease event, serial 31, synthetic NO, window 0x4200001,
      root 0x10a, subw 0x0, time 13172168, (698,1049), root:(1582,1199),
      state 0x12, keycode 38 (keysym 0x41, A), same_screen YES,
      XLookupString gives 1 bytes: (41) "A"
      XFilterEvent returns: False

Note the state field in the KeyPress event is 0x12, this is expected because NumLock and CapsLock as enabled during the test.

Now the same test in VNC gives:

  KeyPress event, serial 29, synthetic NO, window 0x1c00001,
      root 0x3a, subw 0x0, time 2583228944, (443,730), root:(447,752),
      state 0x0, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
      XLookupString gives 0 bytes: 
      XmbLookupString gives 0 bytes: 
      XFilterEvent returns: False

  KeyPress event, serial 29, synthetic NO, window 0x1c00001,
      root 0x3a, subw 0x0, time 2583228944, (443,730), root:(447,752),
      state 0x1, keycode 38 (keysym 0x41, A), same_screen YES,
      XLookupString gives 1 bytes: (41) "A"
      XmbLookupString gives 1 bytes: (41) "A"
      XFilterEvent returns: False

  KeyRelease event, serial 29, synthetic NO, window 0x1c00001,
      root 0x3a, subw 0x0, time 2583228944, (443,730), root:(447,752),
      state 0x1, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
      XLookupString gives 0 bytes: 
      XFilterEvent returns: False

  KeyRelease event, serial 29, synthetic NO, window 0x1c00001,
      root 0x3a, subw 0x0, time 2583229036, (443,730), root:(447,752),
      state 0x0, keycode 38 (keysym 0x61, a), same_screen YES,
      XLookupString gives 1 bytes: (61) "a"
      XFilterEvent returns: False

Note that there additional KeyPress for Shift_L (which was not pressed intentionally) and the state field is now 0x1, ie CapsLock is not reported, instead it's emulated by a fake Shift.

This behaviour /might/ be intentional in vncserver because this is what the documentation for the RFB protocols specifies in paragraph 6.4 "CLIENT TO SERVER MESSAGES":

    http://www.realvnc.com/docs/rfbproto.pdf

    ...
    The difference between upper and lower case keysyms is significant. This is
    unlike some of the keyboard processing in the X Window System which treats
    them as the same. For example, a server receiving an uppercase ’A’ keysym
    without any shift presses should interpret it as an uppercase ’A’. Again
    this may involve an internal “fake” shift press.
    ...

    ...
    Servers should ignore “lock” keysyms such as CapsLock and NumLock where
    possible. Instead they should interpret each character-based keysym
    according to its case.
    ... 

Unfortunately, this is causing troubles for applications and toolkit which deal with modifiers in their accelerators.

Some other VNC server, like "x11vnc" do implement a workaround for this problem:

    http://linux.die.net/man/1/x11vnc

    ...
    -capslock
  
    When in -modtweak (the default) or -xkb mode, if a keysym in the range A-Z
    comes in check the X server to see if the Caps_Lock is set. If it is do not
    artificially press Shift to generate the keysym. This will enable the
    CapsLock key to behave correctly in some circumstances: namely *both* the
    VNC viewer machine and the x11vnc X server are in the CapsLock on state. If
    one side has CapsLock on and the other off and the keyboard is not behaving
    as you think it should you should correct the CapsLock states (hint:
    pressing CapsLock inside and outside of the viewer can help toggle them
    both to the correct state). However, for best results do not use this
    option, but rather *only* enable CapsLock on the VNC viewer side (i.e. by
    pressing CapsLock outside of the viewer window, also -skip_lockkeys below).
    Also try -nomodtweak for a possible workaround.
    ...

Unfortunately, vncserver we have in Red Hat Enterprise Linux do not have such an option.