Bug 121876 - FC2T3 install boot fails on VMware Workstation
Summary: FC2T3 install boot fails on VMware Workstation
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel   
(Show other bugs)
Version: rawhide
Hardware: i686
OS: Linux
Target Milestone: ---
Assignee: Arjan van de Ven
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2004-04-28 20:29 UTC by Phil Schaffner
Modified: 2007-11-30 22:10 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-05-09 20:59:03 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description Phil Schaffner 2004-04-28 20:29:34 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040124

Description of problem:
Attempting to install T3 on VMware Workstation 4.5 on FC1 host.
Booting from ISO #1 gets to:

running install...
running /sbin/loader

Then dies with the VMware Error:
*** VMware Workstation internal monitor error ***NOT_IMPLEMENTED at
Please report this problem by selecting menu item Help > VMware on
theWeb > Request Support, or by going to the Web
Please provide us with the log file
(/raid/prs/vmware/FC2T2/vmware.log) and the core file
If the problem is repeatable, please set 'Logging level' to 'Debug'
inthe Misc panel of the configuration editor. Then reproduce the
incidentand file it according to the instructions.
We will respond on the basis of your support entitlement.
We appreciate your feedback,  -- the VMware Workstation team.

Not entirely unexpected as VMware (on FC1 host) has not been able to
runFC2 test kernels as guest OS since 2.6.5-1.322 (just before the
reiserfs fix) without this problem.  Reported to VMware, but this
configuration is (obviously ;-) unsupported. 

Version-Release number of selected component (if applicable):
Fedora Core 2 Test 3, anaconda-9.92-8

How reproducible:

Steps to Reproduce:
1. Boot from boot.iso or CD #1

Actual Results:  Boot fails.

Expected Results:  Successfull FC installation.

Additional info:

Comment 1 Jeremy Katz 2004-04-29 02:01:14 UTC
This is a VMWare bug.  There's not much I can do about it.   

Comment 2 Petr Vandrovec 2004-05-04 16:03:48 UTC
No, it is your bug, and VMware can only workaround around it. You are
returing with SYSEXIT to segment with limit 0xBFFFFFFF while Intel
documentation explicitly states that segment limit for
sysexit/sysenter must be 0xFFFFFFFF (and hardware always sets limit to
0xFFFFFFFF regardless what GDT says).

This warning is there to only warn you that you wanted to load segment
with limit 0xBFFFFFFF to the processor, but it is not going to happen:
sysexit ALWAYS loads limit 0xFFFFFFFF.

So if you wanted this to do stack no-exec patch, you MUST disable
sysenter, otherwise stack is executable after first sysenter based
syscall in the process occurs, until first interrupt (or taskswitch)
occurs (as interrupts use iret they'll restore proper (shortened)
segment limit on CS).

And if you set limit on CS to 0xBFFFFFFF for some other reason than
stack no-exec (and it does not matter for you that sometime that limit
is 0xFFFFFFFF), then I'd like to know that reason...

Comment 3 Phil Schaffner 2004-05-04 17:47:07 UTC
Looks like this may warrant reconsideration from Red Hat.

Comment 4 Phil Schaffner 2004-05-06 16:04:02 UTC
A work-around that seems to work for some, not for others:

On Mon, 2004-05-03 at 11:19 -1000, Warren Togami wrote:
> Running FC2 Test3 and higher in VMWare is broken because VMWare
needs to 
> implement vdso.  For now you can use "vdso=0" boot option.  It works
> me when running FC2 Test3 within VMWare 4.5.1 hosted on FC2 Test3

This works for me to run latest FC2/rawhide as a guest OS when the
host is FC1 using the vmware-any-any-update65.tar.gz from

When host if FC2/rawhide with vmware-any-any-update65, and guest is
FC2T3 install attempt, the host reboots during the guest installer boot.

Comment 5 Arjan van de Ven 2004-05-07 06:49:01 UTC
I would like to ask Petr where he gets that information, the docs I
have state explicitly that sysexit must return to a segment with
limits *UPTO* 4Gb

(we have an open interpretation request with intel on this one)

Comment 6 Petr Vandrovec 2004-05-07 10:33:20 UTC
I've used http://folk.uio.no/botnen/intel/vt/reference/vc312.htm, as
it is first reference Google finds when you ask for "sysexit segment
limit". See also

And main problem is not that documentation states it, but that it
works that way. 

Just try loading CS's MSR with 0xFFC4, and you'll find that your
system still works until first interrupt (iret will not work, as iret
consults descriptor tables) - and although you have no LDT, you'll
find that CS contains 0xFFC4 (in case of sysenter) or 0xFFD7 (in case
of sysexit), with 4GB limit, executable, non-writeable. So CS's MSR is
used only as a base for visible portion of CS/SS, and invisible
portion is always loaded with base=0, limit=4G-1, big, CS with
readable executable code segment, and SS with writable expand-up.

P4 even works if you'll load values 0xFFE8-0xFFFF to the CS's base,
although then your CS or SS is NULL selector (as Dan Arai found, P3
behaves erratically in this situation, but it is not problem we are
hitting here).

Yes, sysexit can return to the segment with 0 base and limit *upto*
4GB... Only problem is that you'll have set limit to 4GB then, which
does not matter for correct code, but it is killer for stack no-exec...

Comment 7 Ingo Molnar 2004-05-07 12:51:51 UTC
you are right, the limit is set to 4GB on certain CPUs (but not on all
SEP-capable x86 CPUs - e.g. the limit worked fine on my primary
testbox). I've fixed the exec-shield patch and uploaded a new version.
(Arjan updated the Fedora kernel.)

Comment 8 Ingo Molnar 2004-05-07 12:53:46 UTC
we enabled sysenter/sysexit (via vdso=1) for Fedora only a couple of
weeks ago, so this bug had minimal exposure so far. The separate
exec-shield patches i have never enabled the vdso.

Comment 9 Phil Schaffner 2004-05-09 20:59:03 UTC
Seems to be fixed with kernel-2.6.5-1.356 :-)

Currently running FC2 rawhide install with this kernel, VMware
4.5.1/vmware-any-any-update65 on a FC2T3+updates host.

Comment 10 Edward Chow 2004-05-10 17:23:55 UTC
Got quite a few line similar to below when install kernel-2.6.5-1.356
with rpm -ivh kernel-2.6.5-1.356.i686.rpm

/etc/security/selinux/file_contexts:  invalid context 
system_u:object_r:staff_home_spamassassin_t on line number 1856

Should I get/install mkinitrd-
or generate that from kernel source?

Can you be more specific about how you install FC2 rawhide?

Note You need to log in before you can comment on or make changes to this bug.