Bug 1015558

Summary: Kernel 3.11.1 and higher does not boot on VMware VM with BusLogic driver (32-bit)
Product: [Fedora] Fedora Reporter: Bojan Smojver <bojan>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 19CC: gansalmon, itamar, jonathan, kernel-maint, lonelywoolf, madhu.chinakonda, marcelo.barbosa, mstevens, nettrash
Target Milestone: ---Keywords: Patch, Reopened
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-3.11.7-100.fc18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-13 02:15:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Hang with 32-bit kernel 3.11.3
none
Hang with 64-bit kernel 3.11.2
none
dmesg of 3.10.11-200.fc19.i686 none

Description Bojan Smojver 2013-10-04 13:50:37 UTC
Created attachment 807664 [details]
Hang with 32-bit kernel 3.11.3

Description of problem:
No kernels from 3.11.1 to 3.11.3 will boot inside this VMware VM. No LVM volumes are found. Switching back to 3.10 boots immediately.

I am suspecting that latest changes to BusLogic driver may be the problem. See kernel commit 391e2f25601e34a7d7e5dc155e487bc58dffd8c6.

Version-Release number of selected component (if applicable):
kernel-3.11.3-201.fc19

How reproducible:
Always.

Steps to Reproduce:
1. yum -y update
2. reboot
3. Hang.

Actual results:
Kernel hangs on boot. LVM volumes cannot be found.

Expected results:
Boots fine with 3.10, so it should boot fine with 3.11?

Additional info:
I have even tried a 64-bit 3.11.2 kernel, because the CPUs support it. Same result - hang.

Comment 1 Bojan Smojver 2013-10-04 13:51:14 UTC
Created attachment 807665 [details]
Hang with 64-bit kernel 3.11.2

Comment 2 Morten Stevens 2013-10-04 15:01:32 UTC
This is probably an upstream bug!

Please report it to:
https://bugzilla.kernel.org

Comment 3 lonelywoolf 2013-10-05 01:48:53 UTC
do you shure with upstream? )

Comment 4 lonelywoolf 2013-10-06 11:19:37 UTC
I'm confirming bug with vmware ESXi 5.5 in 32-bit vm installation

Comment 5 Bojan Smojver 2013-10-06 12:15:52 UTC
(In reply to lonelywoolf from comment #4)
> I'm confirming bug with vmware ESXi 5.5 in 32-bit vm installation

Thank you kindly for your help.

Comment 6 Bojan Smojver 2013-10-08 21:40:04 UTC
(In reply to lonelywoolf from comment #4)
> I'm confirming bug with vmware ESXi 5.5 in 32-bit vm installation

Just one question: do you see this problem just with BusLogic controller or in general?

Comment 7 Josh Boyer 2013-10-10 20:07:20 UTC
What version of VMWare?

Are you using out-of-tree VMWare modules?

Can you attach the output dmesg after a working boot, and save the output of dmesg somewhere when it fails (or the output of a serial console)?

Can you bisect with kernels in koji to see which point in the 3.11 development series it started failing?

Did you report this upstream as suggested in comment #2?

Comment 8 Bojan Smojver 2013-10-10 21:09:01 UTC
(In reply to Josh Boyer from comment #7)
> What version of VMWare?

I do not know. I have asked the VMware admin, but I was not given that information yet.

> Are you using out-of-tree VMWare modules?

No. Everything is vanilla Fedora.

> Can you attach the output dmesg after a working boot, and save the output of
> dmesg somewhere when it fails (or the output of a serial console)?

I will do the first one. For the second one, I'll have to ask the VMware admin to do it. May take a few days.

> Can you bisect with kernels in koji to see which point in the 3.11
> development series it started failing?

I can try this, but it may take weeks.
 
> Did you report this upstream as suggested in comment #2?

No, because I am not yet convinced it is an upstream bug. I have googled far and wide and could not find anything resembling this. The closest was some problem in one of 3.11 RCs where some crypto module was not included, due to dependency problems caused by recent changes.

Comment 9 Bojan Smojver 2013-10-10 21:14:02 UTC
What I also did try is switching SCSI controller to LSI Logic SAS. I rebuilt initramfs with --add-driver=mptsas (I checked with lsinitrd and indeed a number of modules did get included: mptsas, mptscsih, mptbase). The VMware admin then switched the VM to that controller and rebooted. Same result.

Now, whether I screwed something up and the test was invalid or whether this really is BusLogic unrelated, I cannot tell for sure.

Comment 10 Bojan Smojver 2013-10-10 23:54:07 UTC
Created attachment 810827 [details]
dmesg of 3.10.11-200.fc19.i686

Successful boot.

Comment 11 lonelywoolf 2013-10-13 05:50:21 UTC
(In reply to Bojan Smojver from comment #6)
> (In reply to lonelywoolf from comment #4)
> > I'm confirming bug with vmware ESXi 5.5 in 32-bit vm installation
> 
> Just one question: do you see this problem just with BusLogic controller or
> in general?

Only with BusLogic

Comment 12 lonelywoolf 2013-10-13 05:51:37 UTC
I will test all kernels tomorrow on my installation.

Comment 13 Bojan Smojver 2013-10-13 21:16:33 UTC
(In reply to lonelywoolf from comment #11)

> > Just one question: do you see this problem just with BusLogic controller or
> > in general?
> 
> Only with BusLogic

Thanks for that info. My test with LSI must have been bogus then.

Comment 14 Bojan Smojver 2013-10-13 22:12:44 UTC
Interesting link:

http://www.spinics.net/lists/linux-scsi/msg69044.html

Includes a patch, which, AFAICT, is not included in the current head or stable.

Comment 15 Bojan Smojver 2013-10-14 22:38:15 UTC
(In reply to Bojan Smojver from comment #14)
> Interesting link:
> 
> http://www.spinics.net/lists/linux-scsi/msg69044.html
> 
> Includes a patch, which, AFAICT, is not included in the current head or
> stable.

It is this commit that broke the driver to require the above patch: 	839cb99e8f748391059d10388c8aea48a88c142c. Not the one I mentioned in comment #0.

Josh,

Could you please produce a scratch build with the patch included? Or just wind back the driver in a scratch build so that we can see whether this fixes the issue?

Comment 16 Bojan Smojver 2013-10-15 13:17:36 UTC
Kernel 3.11.4 now running on my VM with mptspi driver. The problem with my previous test was that I added mptsas driver, but the virtual controller is actually LSI parallel SCSI on that VM.

So, most definitely a BusLogic only problem.

Comment 17 Josh Boyer 2013-10-15 16:30:33 UTC
(In reply to Bojan Smojver from comment #15)
> (In reply to Bojan Smojver from comment #14)
> > Interesting link:
> > 
> > http://www.spinics.net/lists/linux-scsi/msg69044.html
> > 
> > Includes a patch, which, AFAICT, is not included in the current head or
> > stable.
> 
> It is this commit that broke the driver to require the above patch: 
> 839cb99e8f748391059d10388c8aea48a88c142c. Not the one I mentioned in comment
> #0.
> 
> Josh,
> 
> Could you please produce a scratch build with the patch included? Or just
> wind back the driver in a scratch build so that we can see whether this
> fixes the issue?

Scratch build with the patch:

http://koji.fedoraproject.org/koji/taskinfo?taskID=6062997

Thanks for chasing things down on this one.  Let me know if that works.

Comment 18 Bojan Smojver 2013-10-15 20:50:50 UTC
(In reply to Josh Boyer from comment #17)
 
> Scratch build with the patch:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=6062997
> 
> Thanks for chasing things down on this one.  Let me know if that works.

Thanks for that. Unless someone beats me to it, I'll give it a go. It make take a few days.

Comment 19 Bojan Smojver 2013-10-17 13:12:33 UTC
(In reply to Josh Boyer from comment #17)
 
> Scratch build with the patch:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=6062997
> 
> Thanks for chasing things down on this one.  Let me know if that works.

Yep, that's the fix.

Comment 20 Josh Boyer 2013-10-17 14:00:05 UTC
Great, thanks for testing.  I'll get the patch included soon.

Comment 21 Josh Boyer 2013-10-17 14:17:35 UTC
Patch added in Fedora git.

Comment 22 Fedora Update System 2013-10-20 19:14:09 UTC
kernel-3.11.6-200.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.11.6-200.fc19

Comment 23 Fedora Update System 2013-10-20 19:15:38 UTC
kernel-3.11.6-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.6-100.fc18

Comment 24 Fedora Update System 2013-10-21 12:07:56 UTC
kernel-3.11.6-300.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.11.6-300.fc20

Comment 25 Fedora Update System 2013-10-21 18:27:58 UTC
Package kernel-3.11.6-300.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.6-300.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-19611/kernel-3.11.6-300.fc20
then log in and leave karma (feedback).

Comment 26 Fedora Update System 2013-10-22 05:39:27 UTC
kernel-3.11.6-300.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2013-10-23 03:36:05 UTC
kernel-3.11.6-200.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 28 Fedora Update System 2013-11-02 19:24:51 UTC
kernel-3.11.6-101.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.6-101.fc18

Comment 29 Fedora Update System 2013-11-03 04:36:01 UTC
Package kernel-3.11.6-101.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.6-101.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20545/kernel-3.11.6-101.fc18
then log in and leave karma (feedback).

Comment 30 Fedora Update System 2013-11-04 20:20:48 UTC
kernel-3.11.7-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.7-100.fc18

Comment 31 Fedora Update System 2013-11-13 02:15:35 UTC
kernel-3.11.7-100.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 32 HiThere 2013-11-18 04:19:49 UTC
Hello
Few days ago I updated my Fedora 18, now I cant compile Vmware 9.2 Need help with this.

Fedora 18 (3.11.7-100.fc18.x86_64) + Vmware 9.2 
Log:http://pastebin.com/kttLFAeY

Thank you.