Bug 1684833 - Kernel compiled with gcc version lower than that of gcc package
Summary: Kernel compiled with gcc version lower than that of gcc package
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 29
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1689592 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-02 22:26 UTC by IFo Hancroft
Modified: 2021-08-05 06:54 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-09 20:37:08 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
make.log of the failed dkms run during Nvidia driver installation (1.52 KB, text/plain)
2019-03-18 19:15 UTC, Michael Riss
no flags Details
nvidia installer log file with the failed dkms run (10.27 KB, text/plain)
2019-03-18 19:17 UTC, Michael Riss
no flags Details

Description IFo Hancroft 2019-03-02 22:26:37 UTC
In the latest update, the kernel 4.20.13-200 is compiled with gcc version lower than that of the gcc package in this update.
The version the kernel is compiled with is 8.2.x I believe.
The version of gcc in the gcc package in the update is gcc version 8.3.1 20190223 (Red Hat 8.3.1-2) (GCC)

Comment 1 Michael Riss 2019-03-05 15:07:28 UTC
We also ran into this issue. When trying to build the nvidia kernel module (version 418.43) with dkms for kernel 4.20.13-200.fc29.x86_64 we get the following error:


---

DKMS make.log for nvidia-418.43 for kernel 4.20.13-200.fc29.x86_64 (x86_64)
Tue Mar  5 15:37:06 CET 2019
make[1]: Entering directory '/usr/src/kernels/4.20.13-200.fc29.x86_64'
make[2]: Entering directory '/usr/src/kernels/4.20.13-200.fc29.x86_64'
  SYMLINK /var/lib/dkms/nvidia/418.43/build/nvidia/nv-kernel.o
  SYMLINK /var/lib/dkms/nvidia/418.43/build/nvidia-modeset/nv-modeset-kernel.o

Compiler version check failed:

The major and minor number of the compiler used to
compile the kernel:

gcc version 8.2.1 20181215 (Red Hat 8.2.1-6) (GCC)

does not match the compiler used here:

cc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


It is recommended to set the CC environment variable
to the compiler that was used to compile the kernel.

The compiler version check can be disabled by setting
the IGNORE_CC_MISMATCH environment variable to "1".
However, mixing compiler versions between the kernel
and kernel modules can result in subtle bugs that are
difficult to diagnose.

*** Failed CC version check. Bailing out! ***

make[3]: *** [/var/lib/dkms/nvidia/418.43/build/Kbuild:181: cc_version_check] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/usr/src/kernels/4.20.13-200.fc29.x86_64/Makefile:1566: _module_/var/lib/dkms/nvidia/418.43/build] Error 2
make[2]: Leaving directory '/usr/src/kernels/4.20.13-200.fc29.x86_64'
make[1]: *** [Makefile:152: sub-make] Error 2
make[1]: Leaving directory '/usr/src/kernels/4.20.13-200.fc29.x86_64'
make: *** [Makefile:79: modules] Error 2

---


Would it be possible to recompile the current kernel with the new gcc and release it as an update?


PS. Setting IGNORE_CC_MISMATCH=1 as a stopgap solution does not work as some part of the build process explicitly resets IGNORE_CC_MISMATCH to "" - in case someone thinks about doing that.

Comment 2 Laura Abbott 2019-03-05 19:36:02 UTC
4.20.14 is being built as we speak which should have the correct compiler https://koji.fedoraproject.org/koji/taskinfo?taskID=33199212

I also went ahead and pushed a change to disable stackleak to avoid this problem in the future until we properly package gcc plugins (the change won't be in 4.20.14 but should be in .15)

Comment 3 Michael Riss 2019-03-11 23:07:54 UTC
4.20.14-200.fc29.x86_64 works fine now, thanks!

Comment 4 Michael Riss 2019-03-16 22:10:26 UTC
The same issue also came up on Fedora 28. I opened a separate bug for it (https://bugzilla.redhat.com/show_bug.cgi?id=1689592). 

Not sure whether this was the right thing to do. How do you guys like to handle issues that affect multiple versions of the distribution? Separate bug reports or one report referencing several distribution versions?

Comment 5 Laura Abbott 2019-03-18 15:20:37 UTC
Please keep everything to one bug for now. That should have been fixed for both F28 and F29. Please post full DKMS logs for 4.20.15.

Comment 6 Laura Abbott 2019-03-18 17:26:43 UTC
*** Bug 1689592 has been marked as a duplicate of this bug. ***

Comment 7 Michael Riss 2019-03-18 19:13:11 UTC
As said, I encounter the problem when trying to compile the nvidia kernel module with the dkms system.
Installed are:

- kernel 4.20.15-100.fc28.x86_64
- gcc-8.3.1-2.fc28

Then I try to run the Nvidia shell installer 'NVIDIA-Linux-x86_64-418.43.run', choose that I want to use
the dkms kernel module and get the following error message in the installer:


---

  ERROR: Failed to run `/usr/sbin/dkms build -m nvidia -v 418.43 -k            
         4.20.15-100.fc28.x86_64`:
         Kernel preparation unnecessary for this kernel.  Skipping...          
 
         Building module:       
         cleaning build area...                                                
         'make' -j4 NV_EXCLUDE_BUILD_MODULES=''
         KERNEL_UNAME=4.20.15-100.fc28.x86_64 IGNORE_CC_MISMATCH=''            
         modules...(bad exit status: 2)
         Error! Bad return status for module build on kernel:
         4.20.15-100.fc28.x86_64 (x86_64)
         Consult /var/lib/dkms/nvidia/418.43/build/make.log for more           
         information.

---


and


---

  ERROR: Failed to install the kernel module through DKMS. No kernel module
         was installed; please try installing again without DKMS, or check the
         DKMS logs for more information.

---


and


---

  ERROR: Installation has failed.  Please see the file
         '/var/log/nvidia-installer.log' for details.  You may find            
         suggestions on fixing installation problems in the README available   
         on the Linux driver download page at www.nvidia.com.

---


I will attach /var/lib/dkms/nvidia/418.43/build/make.log and /var/log/nvidia-installer.log.

But again, it seems to be the same situation as with F29, recompiling the kernel with the current system compiler should solve
the issue. In fact, I did it privately on our systems, I rebuilt the rpms and reinstalled them - and the problem was gone.

Comment 8 Michael Riss 2019-03-18 19:15:25 UTC
Created attachment 1545425 [details]
make.log of the failed dkms run during Nvidia driver installation

Comment 9 Michael Riss 2019-03-18 19:17:13 UTC
Created attachment 1545426 [details]
nvidia installer log file with the failed dkms run

Comment 10 Michael Riss 2019-03-22 11:52:49 UTC
The problem persists in 4.20.16-100.fc28.x86_64, it's still compiled with the old compiler, gcc version 8.2.1 20181215 (Red Hat 8.2.1-6). It should be compiled with gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2).

Comment 11 Michael Riss 2019-03-22 13:26:19 UTC
But the new kernel in testing (https://bodhi.fedoraproject.org/updates/FEDORA-2019-0a771739b5) is compiled with gcc 8.3.1 and works fine for us. Thanks!

Comment 12 Toomas Kaevand 2020-04-03 13:26:29 UTC
This problem has reappeared with Fedora 31 science kernel 5.5.11-200.fc31.x86_64. Kernel has compiled with GCC 9.2 but system has updated to GCC 9.3 (gcc 9.3.1 20200317 (Red Hat 9.3.1-1)). Also kernel 5.5.13-200.fc31.x86_64 is affected by this problem.

Comment 13 Joaquin Alvarez 2020-04-08 20:29:00 UTC
(In reply to Toomas Kaevand from comment #12)
> This problem has reappeared with Fedora 31 science kernel
> 5.5.11-200.fc31.x86_64. Kernel has compiled with GCC 9.2 but system has
> updated to GCC 9.3 (gcc 9.3.1 20200317 (Red Hat 9.3.1-1)). Also kernel
> 5.5.13-200.fc31.x86_64 is affected by this problem.

Yes, i can confirm that. I have a GTX 960 and Fedora 30. The old driver (430.50) causes xorg crash after kernel 5.5 upgrade. When you try upgrade the driver, dkms fail due the Kernel compiled with GCC 9.2, a older version.

I have a workarround tested by me. You can add the parameter --no-cc-version-check in the NVIDIA installer.

Comment 14 naaa 2020-08-03 23:45:47 UTC
> I have a workarround tested by me. You can add the parameter --no-cc-version-check in the NVIDIA installer.

It needs to be pointed out further that this workaround or anything like that isn't good. The check that causes it to fail states that it can result in problems. Use it at your own roll of the dice. As noted above:

"mixing compiler versions between the kernel
and kernel modules can result in subtle bugs that are
difficult to diagnose."

It will only be good to have matching versions unless someone can fix mismatched versions.

This is happening for me with Fedora 32 and trying to boot kernel 5.11 with the current newest Nvidia driver.

Comment 15 Marc 2020-08-05 11:10:08 UTC
happening again on kernel 5.7.11. 

Compiler version check failed:
The major and minor number of the compiler used to
compile the kernel:
gcc version 10.1.1 20200507 (Red Hat 10.1.1-1) (GCC), GNU ld version 2.34-4.fc32
does not match the compiler used here:
cc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)
Copyright (C) 2020 Free Software Foundation, Inc.

Comment 16 Marc 2020-08-05 11:12:34 UTC
sorry, that is Fedora 32, nvidia driver 440.100

Comment 17 leigh scott 2020-08-05 17:28:07 UTC
(In reply to Marc from comment #16)
> sorry, that is Fedora 32, nvidia driver 440.100

You could add 

IGNORE_CC_MISMATCH=1

to the nvidia dkms file to work around the issue or use the updates-testing kernel

https://bodhi.fedoraproject.org/updates/FEDORA-2020-faee35ba77

Comment 18 Marc 2020-08-05 18:57:00 UTC
I think it might be a better idea to sync kernel distribution with gcc?

generic warning in /var/lib/dkms/nvidia/440.100/build/make.log:

Compiler version check failed:

The major and minor number of the compiler used to
compile the kernel:

gcc version 10.1.1 20200507 (Red Hat 10.1.1-1) (GCC), GNU ld version 2.34-4.fc32

does not match the compiler used here:

cc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


It is recommended to set the CC environment variable
to the compiler that was used to compile the kernel.

The compiler version check can be disabled by setting
the IGNORE_CC_MISMATCH environment variable to "1".
However, mixing compiler versions between the kernel
and kernel modules can result in subtle bugs that are
difficult to diagnose.

Comment 19 naaa 2020-08-05 23:01:12 UTC
The issue with 5.7.11 is fixed with 5.7.12. You can go to fedora updates https://bodhi.fedoraproject.org/ and search for kernel then use the dnf text for the kernel to install it now or wait until its pushed out.

It seems that they get it wrong at times so it will probably continue to happen randomly. I really don't know what they can do about that, if they can do a code test on that or not.

Comment 20 davidwriter 2021-04-29 19:56:45 UTC
The latest Fedora 33 kernel:

DKMS make.log for nvidia-460.39 for kernel 5.11.16-200.fc33.x86_64 (x86_64)
Thu 29 Apr 2021 01:38:24 AM PDT
make[1]: Entering directory '/usr/src/kernels/5.11.16-200.fc33.x86_64'
scripts/Makefile.lib:8: 'always' is deprecated. Please use 'always-y' instead
  SYMLINK /var/lib/dkms/nvidia/460.39/build/nvidia/nv-kernel.o

Compiler version check failed:

The major and minor number of the compiler used to
compile the kernel:

gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9), GNU ld version 2.35-18.fc33

does not match the compiler used here:

cc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)

Comment 21 Marc 2021-08-05 06:54:03 UTC
saw this on a clean install of F34 and dkms as well. Now again on kernel 5.13.6, where a gcc update came in a few days earlier as the kernel

Compiler version check failed:

The major and minor number of the compiler used to
compile the kernel:

gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34

does not match the compiler used here:

cc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Note You need to log in before you can comment on or make changes to this bug.