Bug 1370061

Summary: Kernel 4.7.2-200 panics after boot (bug in TPROXY code)
Product: [Fedora] Fedora Reporter: Bojan Smojver <bojan>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 24CC: gansalmon, ichavero, itamar, jonathan, kernel-maint, labbott, madhu.chinakonda, mchehab, xavier268
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-4.7.2-201.fc24 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-02 20:51:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
XS35GT V2 kernel panic photo of the screen
none
Double free on ThinkPad T450s with kernel-4.7.2-200.rhbz1370061.fc24.x86_64
none
OOPS on the i686 VM after boot with the scratch kernel
none
OOPS on the i686 VM after boot with the scratch kernel
none
Panic during boot 4.7.2-101 on VirtualBox. Messed up display.
none
Kernel 7.2-101.fc23 panic boot on VirtualBox (flash screen capture video) none

Description Bojan Smojver 2016-08-25 08:22:16 UTC
Created attachment 1193909 [details]
XS35GT V2 kernel panic photo of the screen

Description of problem:
Machine comes up, but after a minute or two, it panics. The only thing I know is what's in the photo (see attached). Reverting to kernel 4.6.7-300 immediately makes the machine work.

Version-Release number of selected component (if applicable):
kernel-4.7.2-300

How reproducible:
Always.

Steps to Reproduce:
1. Boot Shuttle XS35GT V2 (Atom, x86_64).

Actual results:
Kernel panic.

Expected results:
Kernel 4.6 works fine.

Additional info:

Comment 1 Bojan Smojver 2016-08-25 08:23:34 UTC
Sorry, version is 4.7.2-200, of course.

Comment 2 Bojan Smojver 2016-08-25 10:27:13 UTC
Totally Wild guess (I'm running squid in tproxy mode on this box too):

https://patchwork.ozlabs.org/patch/660174/

Comment 3 Laura Abbott 2016-08-25 14:36:04 UTC
Thanks for the pointer to the patch, can you try this scratch build with the patch suggested? http://koji.fedoraproject.org/koji/taskinfo?taskID=15377542

Comment 4 Bojan Smojver 2016-08-25 22:25:48 UTC
OK, shall do.

There is also this:

http://lkml.iu.edu/hypermail/linux/kernel/1607.3/01867.html

I had an unusable i686 VM running on VMware ESX with 4.7.2 kernel as well. Both that VM and the server crash reported here run rpc.gssd. Just sayin'...

Comment 5 Bojan Smojver 2016-08-25 23:09:01 UTC
(In reply to Laura Abbott from comment #3)
> Thanks for the pointer to the patch, can you try this scratch build with the
> patch suggested? http://koji.fedoraproject.org/koji/taskinfo?taskID=15377542

Booted that server up with the scratch build kernel and used squid. Didn't crash yet. Will keep you posted.

Comment 6 Bojan Smojver 2016-08-26 00:48:45 UTC
Created attachment 1194174 [details]
Double free on ThinkPad T450s with kernel-4.7.2-200.rhbz1370061.fc24.x86_64

The scratch kernel won't boot on ThinkPad T450s.

Comment 7 Josh Boyer 2016-08-26 00:50:11 UTC
(In reply to Bojan Smojver from comment #6)
> Created attachment 1194174 [details]
> Double free on ThinkPad T450s with kernel-4.7.2-200.rhbz1370061.fc24.x86_64
> 
> The scratch kernel won't boot on ThinkPad T450s.

That's a known problem with grub2 and unsigned kernels (all scratch builds are unsigned).  Nothing to do with the kernel itself.

Comment 8 Bojan Smojver 2016-08-26 00:58:48 UTC
(In reply to Josh Boyer from comment #7)

> That's a known problem with grub2 and unsigned kernels (all scratch builds
> are unsigned).  Nothing to do with the kernel itself.

OK. Is that with EUFI boot only or does it affect BIOS machines as well? Because I installed the i686 scratch build onto my remote VM (VMware ESX) and that hung it as well. AFAIK, that 32-bit "machine" is BIOS based, not EUFI.

PS. The x86_64 server that did boot with it is not EUFI.

Comment 9 Bojan Smojver 2016-08-26 03:57:53 UTC
(In reply to Bojan Smojver from comment #8)

> OK. Is that with EUFI boot only or does it affect BIOS machines as well?
> Because I installed the i686 scratch build onto my remote VM (VMware ESX)
> and that hung it as well. AFAIK, that 32-bit "machine" is BIOS based, not
> EUFI.
> 
> PS. The x86_64 server that did boot with it is not EUFI.

And answering my own question, I see this grub2 problem is related to secure boot, which (to the best of my knowledge), my i686 VM isn't using. So, that scratch build is still no good on that VM.

I don't have access to the console there, so I cannot provide any more diagnostics.

Comment 10 Bojan Smojver 2016-08-26 12:20:41 UTC
Created attachment 1194330 [details]
OOPS on the i686 VM after boot with the scratch kernel

This kernel hung the i686 box on first boot, just like the 4.7.2 that's currently in testing. No idea whether OOPS is related to the hang or not.

Comment 11 Bojan Smojver 2016-08-26 12:25:38 UTC
Created attachment 1194331 [details]
OOPS on the i686 VM after boot with the scratch kernel

Correct image.

Comment 12 Bojan Smojver 2016-08-26 13:13:05 UTC
(In reply to Bojan Smojver from comment #10)

> This kernel hung the i686 box on first boot, just like the 4.7.2 that's
> currently in testing. No idea whether OOPS is related to the hang or not.

It came up on second try, but then hung the box on reboot. So, something is not quite right there in a rather unpredictable way.

Comment 13 Laura Abbott 2016-08-26 15:25:13 UTC
Let's keep this bugzilla to one issue please. The squid proxy issue looks to be fixed, the i686 issue looks to be separate. Please file a separate bugzilla for the i686 issue. I pulled the squid fix into f24/f23. I'm going to do another build today.

Comment 14 Bojan Smojver 2016-08-27 11:23:41 UTC
(In reply to Laura Abbott from comment #13)
> Let's keep this bugzilla to one issue please. The squid proxy issue looks to
> be fixed, the i686 issue looks to be separate. Please file a separate
> bugzilla for the i686 issue.

See bug #1370795.

Comment 15 Fedora Update System 2016-08-29 16:19:48 UTC
kernel-4.7.2-201.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-2e5ebfed6d

Comment 16 Fedora Update System 2016-08-29 16:22:21 UTC
kernel-4.7.2-101.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f1adaaadc6

Comment 17 Fedora Update System 2016-08-31 12:58:02 UTC
kernel-4.7.2-201.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-2e5ebfed6d

Comment 18 Fedora Update System 2016-08-31 12:58:29 UTC
kernel-4.7.2-101.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f1adaaadc6

Comment 19 Fedora Update System 2016-09-02 20:50:46 UTC
kernel-4.7.2-201.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2016-09-02 23:19:19 UTC
kernel-4.7.2-101.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 xavier268 2016-09-04 16:18:22 UTC
Created attachment 1197643 [details]
Panic during boot 4.7.2-101 on VirtualBox. Messed up display.

Comment 22 xavier268 2016-09-04 17:19:37 UTC
Created attachment 1197657 [details]
Kernel 7.2-101.fc23 panic boot on VirtualBox (flash screen capture video)

Kernel panic while booting kernel 4.7.2-101 fc23 on VirtualBox (64 bits) running on Windows10-i7 host. Reverting to Kernel 4.6.x works perfectly.

Comment 23 xavier268 2016-09-05 09:41:53 UTC
(In reply to Fedora Update System from comment #20)
> kernel-4.7.2-101.fc23 has been pushed to the Fedora 23 stable repository. If
> problems still persist, please make note of it in this bug report.

Pb persists on fed23 4.7.2-101. See uploaded screenshots.

Comment 24 xavier268 2016-09-05 15:14:18 UTC
Same issue with a fresh install of fed24-live-Xfce-x64 on VirtualBox 5.2.26 (Windows10 host on i7. Initial install is ok, using kernel 4.5.5. Updating to 4.7.2.201fc24. First reboot is ok. Install Oracle VBox additions (larger screen, etc ...). After reboot, similar kernel panic as was happening on F23. After stopping VM, then restarting with 4.5.5. kernel, everything  works fine. A new attempt to boot with 4.7.2.201 generates same panic again.

Comment 25 xavier268 2016-09-05 15:48:41 UTC
Just guessing from observation, if that can help - this seems to be triggered by the installation of the oravle VirtualBox extensions. Without installin the VB Extensions, it f24 kernel 4.7.2.201 now boots fine ... ? Will try to signal the Oracle VBox guys as well.

Comment 26 Bojan Smojver 2016-09-05 22:57:28 UTC
Xavier, unless you are running squid with TPROXY enabled, you should open a different bug for the trouble you're having.

Comment 27 xavier268 2016-09-06 09:17:27 UTC
@Bojan, Understood. I confirm I am not using squid. New Bug 1373424 opened. Xavier