Bug 130754

Summary: Needed: Fedora Core kernel compilation tutorial
Product: [Fedora] Fedora Documentation Reporter: Paul W. Frields <stickster>
Component: docs-requestsAssignee: Kevin Hobbs <kevin.hobbs.1>
Status: CLOSED WONTFIX QA Contact: Paul W. Frields <stickster>
Severity: medium Docs Contact:
Priority: medium    
Version: develCC: feliciano.matias, herrold, kevin.hobbs.1, k.georgiou, shishz
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
URL: http://www.redhat.com/archives/fedora-docs-list/2004-August/msg00229.html
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-11-27 14:46:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 129807    
Attachments:
Description Flags
A new kernel compilation tutorial
none
A newer cleaner kernel compilation tutorial
none
a kernel compilation tutorial
none
A snip from the kernel build output none

Description Paul W. Frields 2004-08-24 13:00:49 UTC
Description of problem:

Fedora Core needs a kernel compilation guide. Although releases 1 and
2 follow roughly the same guidelines as earlier Red Hat Linux products
(with the exception of the renaming of kernel-source to
kernel-sourcecode in Core 2), current development status is that
kernel-sourcecode will not be produced any more, in favor of simply
using the .src.rpm, which brings kernel building in line with building
other RPMs.

The release of Core 3 will likely prompt a large number of repetitive
threads on fedora-list (and other venues) asking how to complete this
process, since many Fedora Core users tend to complile their own
kernels. Arjan produced a quick guide to the changes for
FedoraNews.org (see references below), which we should use as source
material.

*** NOTE: Arjan's material on FedoraNews.org is under a Creative
Commons License (BY/NC/SA 1.0), which means we can't use it verbatim.
I'd like to suggest that future large-scale explanations by Fedora
Project members and developers be run by the Fedora Docs Project
first, so we have an option to FDL these documents with the author's
permission before it falls under a license that makes it harder for us
to use the material. CC material *can* be released under an FDL as
long as the author gives permission for us to do so. ***

I'm including Arjan in the CC: list so that he knows we (the Fedora
Docs Project) are going to try and coordinate an official tutorial or
guide with this information, and might tap him with a couple questions
while preparing it.


Additional info:

List of references:
http://www.redhat.com/archives/fedora-docs-list/2004-August/msg00229.html
  (Top of thread in fedora-docs-list introducing idea, 5 msgs total)
http://fedoranews.org/colin/fnu/issue14.shtml
  (FedoraNews.org article with contributions from Arjan van de Ven)

Comment 1 Kevin Hobbs 2004-09-27 23:36:11 UTC
I would like to participate in any way I can.  I have a custom build I
want to try and I figure it dosn't take any extra effort to keep a
journal. 

Comment 2 Tammy Fox 2004-09-30 15:04:01 UTC
Since Kevin has started this and posted a draft to the list, I'm
moving it to the docs in progress tracker bug. ;-) Thanks for working
on this!

Comment 3 Kevin Hobbs 2004-10-05 17:12:40 UTC
Created attachment 104791 [details]
A new kernel compilation tutorial

Comment 4 Kevin Hobbs 2004-10-17 16:51:45 UTC
Created attachment 105353 [details]
A newer cleaner kernel compilation tutorial

This version has a much simpler rpmmacros and thus simpler commands through the
rest of document.

Comment 5 Paul W. Frields 2004-10-18 12:51:43 UTC
Kevin,

What you've got is a good start. A couple things to keep in mind as
you move through this process. We've exchanged a few e-mails already,
so I'll just post this in Bugzilla for easy reference. Also, these
things may be helpful for other new FDP contributors.

1. Please remember to set the MIME type properly on Bugzilla
attachments, since otherwise most browsers will simply download
garbage into the existing window. The best way to do this is change
the attachment type to auto-detect. If you are posting a tarball, do a
"make clean" before you make your tarball, and remove unnecessary
backup files. The only things that need to be in the tarball are your
XML source files and the Makefile.

2. When last we conversed, you were going to get your steps
technically checked by Arjan. Have you done so?

3. The overall style of your tutorial is much improved, but the
Introduction still needs a little love. Try and replicate the
Introduction structure used in a couple of other tutorials, such as my
usb-hotplug-tutorial or mirror-tutorial. (You can search those terms
in Bugzilla or simply find them in the "all fedora-docs bugs" list.)

4. You should probably use a <procedure> for this document instead of
separate <section>s. You'll probably want to cut down to three actual
<section>s, the Introduction, the procedure, and your bibliography or
source list. As a side effect, the rendering for the actual procedure
will be on one page, which makes sense for such a short tutorial. The
XML would look like this:

  <procedure>
    <step>
      <title>Step 1</title>
      <para>This is step 1.</para>
    </step>
    <step>
      <para>Step 2 has many parts.</para>
      <substeps>
        <step>
          <title>Step 2A</title>
          <para>This is Step 2A.</para>
        </step>
        <step>
          <title>Step 2B</title>
          <para>This is Step 2B.</para>
        </step>
      </substeps>
    </step>
  </procedure>

Glad you're still working on it and improving it!

Comment 6 Kevin Hobbs 2004-10-19 13:31:30 UTC
1. I'll be more carefull about the MIME type in the future. I seem to
recal that it was auto, but oh well. I'll clean up before making the
tarball.  I thought including the renderd html was a good idea but I
suppose I can just add a link.

2. I have now e-mailed Arjan.  His comments to the boiled bown
instructions were:

> mkdir ~/rpmbuild ~/rpmbuild/SOURCES ~/rpmbuild/RPMS ~/rpmbuild/BUILD
> rpm -ihv kernel-XXX.src.rpm
> rpmbuild -bp --target=ARCH ~/rpmbuild/SPECS/kernel-XXX.spec

ARCH->noarch 
noarch ought to work just fine there

> cd ~/rpmbuild/BUILD/kernel-XXX/linux-XXX/

at this point you can already custom build...

> make menuconfig
> cp .config ~/rpmbuild/SOURCES/kernel-XXX-ARCH.config
> rpmbuild -bb --target=ARCH ~/rpmbuild/SPECS/kernel-XXX.spec
> sudo rpm -ihv ~/rpmbuild/RPMS/ARCH/kernel-XXX.NAME.ARCH.rpm

....and these steps are if you want your modified config to be there
in rpm
format.

3. OK I'll take a look.

4. Sounds good I'll try that out.

I'm also concerned with my use of XXX, ARCH, and NAME.  The ARCH
confused one of my coworkers who tried out the tutorail this weekend.
 I considered using shellisms like $(arch) but I didn't think they
would be more clear to the reader.

Comment 7 Paul W. Frields 2004-10-19 18:04:24 UTC
To simplify your directions, make ARCH consistently "i686," "x86_64,"
or whatever is appropriate, rather than mixing "noarch" in there. (I'm
uncertain whether "noarch" should be used during the rpmbuild -bp
step; there's no compelling reason to make it different from the -bb
step, though.)

I can think of three ways to handle the "ARCH" question. First, in
your directions you could simply include a step before the rpmbuild
-bp step:

    export ARCH="i686"

Then you can simply write the rest of your steps using $ARCH.

However, DocBook XML allows you to use the tag <replaceable> around
ARCH. It will render in a way that visually cues the user that they
should substitute their own value. So as a second alternative you
could code the XML as (ignore the line break):

<screen>
  <userinput>rpmbuild -bb --target=<replaceable>ARCH</replaceable>
~/rpmbuild/SPECS/kernel-2.6.spec</userinput>
</screen>

A third way to do this -- and I think *this* is the best way -- is to
eliminate the term "ARCH" entirely and use uname to generate the right
arch for the user:

  rpmbuild -bb --target=`uname -p` ~/rpmbuild/SPECS/kernel-2.6.spec

This is the most elegant solution... it doesn't require an extra step,
the user types it verbatim without having to substitute or guess, and
you can drop `uname -p` (remember those are backquotes, not single
quotes) anywhere you are currently using ARCH in your text.

Comment 8 FĂ©liciano Matias 2004-10-26 12:11:10 UTC
Chapter : 7. Reconfigure the kernel
make menuconfig
=> ncurses-devel require
make xmenu
=> qt-devel require
make gmenu
=> gtk2-devel require



Comment 9 John Reiser 2004-10-29 16:02:28 UTC
Chapter 2, rpmmacros: mkdir ~/rpmbuild/SRPMS

Chapter 5, Disable SMP Build: This optional action saves build time
when the SMP kernel will not be used.  This step is not needed for
correctness.

Chapter 6, Apply Patches: If you modify any files within the kernel
source tree, then in order to carry your modifications forward into
the actual rebuild (Chapter 8, Build the Kernel), you must make your
changes into an actual patch.  Use "diff -Nru", put the output into a
new file in ~/rpmbuild/SOURCES, give the file a PatchNNN number in the
~/rpmbuild/SPECS/kernel-XXX.spec file, insert a "%patchNNN -p1"
command there.

Chapter 7: Reconfigure the Kernel: Change the "%define release" string
in the ~/rpmbuild/SPECS/kernel-XXX.spec file, by appending some text
that will identify your changes.  When your new kernel is running,
then this string will appear in the output from "uname -a" and "cat
/proc/version", and will distinguish your kernel from the plain
distributed kernel.

Chapter 8: Build the Kernel:  Use "-ba" instead of "-bb".  This will
build the ~/rpmbuild/SRPMS/kernel-XXX.src.rpm file in addition to the
~/rpmbuild/RPMS/i686/kernel-XXX.i686.rpm file.  This will help you
comply with the GPL if you want to distribute your kernel to someone
else, because the .src.rpm contains the _complete_ _reproducible_
recipe (with all sources and patches, both original and yours) to
build _your_ kernel.  This requires 50MB more diskspace and a few more
minutes of time, but is well worth it.  Saving your new .src.rpm file
allows you (or anyone else) to reproduce your new kernel.

Chapter 8: Build the Kernel:  Note that one of the first things that
rpmbuild does, is throw away any existing ~/rpmbuild/BUILD/kernel-XXX
tree!  As a result, all of your changes to the kernel source tree must
be properly contained in .patch files within the ~/rpmbuild/SOURCES
directory, and managed with "PatchNNN" and "%patchNNN -p1" in the
.spec file.

Chapter 8: Build the Kernel:  Do this step using a gcc that is
compatible with the particular kernel-XXX.  The easiest way is to
perform the build on a system that is running the original gcc that
shipped with kernel-XXX.  Check by comparing "uname -a" with "gcc
--version".  Unfortunately, in general the kernel is neither forward
compatible, nor backward compatible, with any other version of gcc. 
Sometimes the incompatibilies are obvious at build time (such as the
syntax for writing a string constant across multiple lines), but
sometimes the incompatibilities are subtle (such as bad generated code
for "corner cases".)  [Yeah, this sucks; but that's the way it is.]



Comment 10 Paul W. Frields 2004-10-29 17:23:48 UTC
Many of the corrections in #9 above aren't really valid:

The SRPMS folder isn't needed since the instructions only build a
binary kernel (rpmbuild -bb --target=<arch>).

Disabling the SMP build is given as an "if" step. If you don't run
SMP, why would you want to spend many minutes (or even hours, remember
that not everyone is running a multi-GHz P4 or Athlon) compiling a
useless kernel?

Applying patches and reconfiguring: I think the author of comment #9
does not fully grok the way the kernel build happens. The release is
automatically tagged by the build process unless a special file exists
on the building host. No patching is necessary since the user is only
replacing the normal kernel*-<arch>.config with the new one he just
built. The rpmbuild -bp is run only to insure that any previous
patches are applied correctly.

As for making .src.rpm files, this is not necessary *unless* you
distribute them, as the above author noted. This should be in a
"Notes" or "Appendix" section after the main procedure. Most users
building their own kernel are doing it for personal use, and that
should be noted in the Introduction, when it is brought into
compliance with fedora-docs standards later.

Comment 11 John Reiser 2004-10-30 16:33:33 UTC
I want a tutorial to result in more understanding of what is going on,
and why, than is commonly obtained from a "cookbook recipe."

Refering to Comment #9 and the criticisms of Comment #10:

The SRPMS directory is required by "-bb", which will fail if it does
not exist when "rpmbuild -bb" begins.  [Perhaps this is a misfeature
in rpmbuild and/or the .spec file.]  Other parts of Comment #9 depend
on -bb, so making the SRPMS directory is necessary to be consistent.

The default Fedora Core configuration sets CONFIG_OPROFILE the same as
CONFIG_SMP, and more than a few administrators want to run oprofile
even on a uniprocessor box.  There are also technical details
involving the configuration and use of local APIC [Advanced
Programmable Interrupt Controller] hardware: if you have the hardware
on a UP board (often it is cheaper to put it on a UP board because of
design commonality with SMP boards), then it can be better to run a
SMP kernel to take advantage of the local APIC, than to run a UP
kernel which might not utilize the local APIC as well.  Also, part of
the purpose of a tutorial is to explain, and understanding the
difference between "needed for correctness" and "useful to save build
time+space" can be helpful.

As someone who customizes and builds dozens of kernels each year, and
distributes them to customers who pay for this service, explicitly
editing the "%define release" string has proved to be worth it many
times over.  The default tagging is only enough to "shift the blame"
for bad results off of Fedora Core; it is not helpful enough to the
builder herself.  My experience is that if anyone finds out that you
_have_ successfully built a kernel using rpm, then they ask you for a
copy!  You become a distributor, so building the .src.rpm in the first
place actually saves time.

Comment #10 paragraph 4, "No patching is necesssary ... [because only
 the .config is changing]."  This may be true for the most common
changes to .config, but for other sets of changes there is a fair
chance that patching will be required to fixup minor [or major]
incompatibilities.  The set of possible .config is not exhaustively
tested.  I see a fair demand for feature additions which definitely do
require patching.  When patching is required, then a newbie most
probably _will_ be surprised and disappointed by the "loss" of edits
[from the "-bp" BUILD directory that did not get made into patches
before the "-ba" build is started] every time rpmbuild is run.  It is
vital for a tutorial to explain and emphasize this property of
rpmbuild, and the reason behind it.  [If builds are performed using
the known-reproducible procedure only, then it never can happen that
there is a completed, non-reproducible build.]



Comment 12 John Reiser 2004-10-30 16:44:46 UTC
Comment #11, paragraph 3: all instances of "-bb" should say "-ba"
instesad.


Comment 13 Kevin Hobbs 2004-10-30 16:49:26 UTC
Created attachment 105977 [details]
a kernel compilation tutorial

I've moved most of the steps into a procedure. I've added some of the things
people have said. The directory was cleaned up in the tarball, so I put the
html up on our lab web server.

http://crab-lab.zool.ohiou.edu/kevin/kernel-compilation-tutorial-en/

Comment 14 Paul W. Frields 2004-10-31 14:17:07 UTC
John, you make good arguments for your points, so I'll concur. Thanks
for taking the time to explain. In that case, I think the tutorial
probably needs to address those points briefly and give concise
alternatives for the reader.

Comment 15 Kevin Hobbs 2004-11-12 14:55:55 UTC
Created attachment 106566 [details]
A snip from the kernel build output

With the upgrade to Fedora Core 3 even a simple "rpmbuild --rebuild
kernel-2.6.9-1.667.src.rpm" fails complaining about gpg keys.  I'm assuming
this can be fixed in ~/.rpmmacros by pointing rpm at the batch generated gpg
key.  I've missed, and haven't been able to find, usefull discussion on this. 
The best link I've found so far is from a mirror of the kernel archive from a
month ago:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0410.1/index.html#1522

Any other urls would really be apreciated

Comment 16 Paul W. Frields 2004-11-12 17:07:08 UTC
Are you certain this isn't a configuration problem on your system?
This part works fine for me when I tested, e.g.:

++ find
/var/tmp/kernel-2.6.9-1.667.pfrields-root/lib/modules/2.6.9-1.667.pfrields
-name '*.ko' -type f
+ for i in '` find $RPM_BUILD_ROOT/lib/modules/$KernelVer -name "*.ko"
-type f`'+ sh ./scripts/modsign/modsign.sh
/var/tmp/kernel-2.6.9-1.667.pfrields-root/lib/modules/2.6.9-1.667.pfrields/kernel/crypto/blowfish.ko
+ mv -f
/var/tmp/kernel-2.6.9-1.667.pfrields-root/lib/modules/2.6.9-1.667.pfrields/kernel/crypto/blowfish.ko.signed
/var/tmp/kernel-2.6.9-1.667.pfrields-root/lib/modules/2.6.9-1.667.pfrields/kernel/crypto/blowfish.ko
+ for i in '` find $RPM_BUILD_ROOT/lib/modules/$KernelVer -name "*.ko"
-type f`'+ sh ./scripts/modsign/modsign.sh
/var/tmp/kernel-2.6.9-1.667.pfrields-root/lib/modules/2.6.9-1.667.pfrields/kernel/crypto/khazad.ko

...and so on.

I tried this twice; the first time I commented out the %_signature and
%_gpg_name defines in my ~/.rpmmacros to duplicate "clueless user"
status, and the second time with these defaults in place. I'm not
convinced these would have any effect on the build, from reading the
.spec file. (Although you did that as well, right?)  I suppose it's
possible I have something else configured "correctly" here through
dumb luck, but I doubt it.


Comment 17 Kevin Hobbs 2004-11-17 19:33:52 UTC
With kernel-2.6.9-1.678_FC3 my module signing problem went away.

Comment 18 Kam Leo 2005-01-10 00:09:22 UTC
Please make the tutorial printer friendly.  Several sections on page 
2 of Kevin's tutorial exceeds the width of an 8-1/2 inch sheet of 
paper. It would be fairly easy to fold the long lines so that they 
are not truncated when printed.

Comment 19 Kevin Hobbs 2005-01-10 14:43:35 UTC
If you can be more specific about which lines maybe I could do a
little more.

The only line that gets chopped when I print the Steps HTML page from
Firefox is the mkdir line.  It gets chopped by the box it is in not
the page.  I'll split that up.

Comment 20 Karsten Wade 2005-08-18 19:37:55 UTC
Is this updated and available for FC4?

Can we get it into the CVS?

We can add a component in bugzilla, finish editing, and publish this.

'Twould be nice to remove this type of content from the release notes, once and
for all.



Comment 21 Kevin Hobbs 2005-08-18 20:12:40 UTC
Grumble grumble.  I've been lurking since the purchase of compatible
hardware eliminated the need to compile kernels at work. However, I know
it's important to get the Doc in CVS the webserver gets hit constantly
for this.

I'll go read the "Get CVS Access" stuff.

Comment 22 Paul W. Frields 2005-10-11 21:16:14 UTC
Has the author done CVS access for this?  Came up at 11 Oct 2005 FDSCo meeting.

Comment 23 Kevin Hobbs 2005-10-12 03:02:13 UTC
I can no longer spend work time on this, and I do not have the knowledge to add
more detail.

I must throw the kernel compilation tutorial "over the wall" so that I am not
holding up the process any more.

Comment 24 Paul W. Frields 2005-11-27 14:46:17 UTC
This has been addressed in the relnotes as of FC5t1, so in conjunction with the
lack of interest in maintaining this tutorial, closing this bug.