Bug 1104253

By removing a few packages with dependency problems, I was able to update over a hundred packages that wouldn't previously install. I'll check if this bug still appears when the next kernel update happens.

You're using vbox drivers with DKMS.  Does it happen without any of that installed?

Appears that the error is coming from installing the VirtualBox guest modules, based on the following. Sorry for the noise.

[root@localhost ~]# dkms status 
vboxguest, 4.3.12, 3.15.0-0.rc7.git0.1.fc21.x86_64, x86_64: installed
vboxguest, 4.3.12, 3.15.0-0.rc8.git0.1.fc21.x86_64, x86_64: installed
[root@localhost ~]# rpm -q kernel
kernel-3.15.0-0.rc7.git0.1.fc21.x86_64
kernel-3.15.0-0.rc7.git4.2.fc21.x86_64
kernel-3.15.0-0.rc8.git0.1.fc21.x86_64
[root@localhost ~]# dkms install vboxguest/4.3.12 -k 3.15.0-0.rc7.git4.2.fc21.x86_64

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=3.15.0-0.rc7.git4.2.fc21.x86_64 -C /lib/modules/3.15.0-0.rc7.git4.2.fc21.x86_64/build M=/var/lib/dkms/vboxguest/4.3.12/build........................................
cleaning build area....

DKMS: build completed.

vboxguest.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-0.rc7.git4.2.fc21.x86_64/extra/

vboxsf.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-0.rc7.git4.2.fc21.x86_64/extra/

vboxvideo.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-0.rc7.git4.2.fc21.x86_64/extra/
Adding any weak-modules

sed: -e expression #1, char 21: unterminated `s' command
depmod.......

DKMS: install completed.
[root@localhost ~]# dkms status 
vboxguest, 4.3.12, 3.15.0-0.rc7.git0.1.fc21.x86_64, x86_64: installed
vboxguest, 4.3.12, 3.15.0-0.rc7.git4.2.fc21.x86_64, x86_64: installed
vboxguest, 4.3.12, 3.15.0-0.rc8.git0.1.fc21.x86_64, x86_64: installed
[root@localhost ~]#

Filed https://www.virtualbox.org/ticket/13097 for anyone else who stumbles on this.

The VirtualBox ticket was closed, according to the dev it actually appears to be a problem with Rawhide's dkms (the problem does not happen in F20). See https://www.virtualbox.org/ticket/13097#comment:1 for his explanation.

I'm sorry but I'm not able to reproduce this. Also testing as specified in the vbox bug (bash -x /sbin/dkms) doesn't return me the sed error.

The dkms package is identical across all RHEL/Fedora distribution; so it's a bit strange that you have this error on rawhide.

[root@rawhide ~]# dkms build -m xpad/105

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make KERNELRELEASE=3.15.0-0.rc8.git2.1.fc21.x86_64 KVERSION=3.15.0-0.rc8.git2.1.fc21.x86_64....
cleaning build area...

DKMS: build completed.
[root@rawhide ~]# dkms install -m xpad/105

xpad:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-0.rc8.git2.1.fc21.x86_64/extra/
Adding any weak-modules

depmod.....

DKMS: install completed.
[root@rawhide ~]# dkms uninstall -m xpad/105

-------- Uninstall Beginning --------
Module:  xpad
Version: 105
Kernel:  3.15.0-0.rc8.git2.1.fc21.x86_64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.
Removing any linked weak-modules

xpad.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.15.0-0.rc8.git2.1.fc21.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.

One note, I just installed a rawhide VM to check for this.

Which DKMS package are you using?
I don't even get the output that is specified in the vbox report.

(In reply to Simone Caronni from comment #8)
> Which DKMS package are you using?
> I don't even get the output that is specified in the vbox report.

See https://www.virtualbox.org/ticket/13097#comment:2 - I'm using dkms-2.2.0.3-21.fc21 in Rawhide. All packages in my Rawhide guest are Fedora packages from the Rawhide repo, and there are no package problems (all installed packages are still in the Rawhide repo, no duplicate packages, etc.).

Note that I only started seeing this bug within the last few weeks or so, long after the last dkms change, but it could have been around the time of the 3.15 kernel package reorganization, so that could have been the trigger. I've only seen the bug with the VirtualBox kernel modules (those are the only DKMS modules I have).

I noticed that in Rawhide, unlike F20, /etc/sysconfig/kernel is now 0 bytes. Can one of the kernel devs in the CC list say if this is intentional and permanent? From https://www.virtualbox.org/ticket/13097#comment:1 maybe the VirtualBox dkms script is assuming that it's nonempty? (In which case it really is a vbox bug, and I'll reopen that ticket.)

I have no idea what sets /etc/sysconfig/kernel, but it does not come from the kernel package explicitly.  The file isn't owned by anything in the RPM db on my F20 system either.

I think that is the issue. The file, on my fresh rawhide installation should be populated like this:

# UPDATEDEFAULT specifies if new-kernel-pkg should make
# new kernels the default
UPDATEDEFAULT=yes

# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=kernel-core

The file is not owned by any package, that's correct. It's generated at install time.

Can you try filling it with the above information and see if the error persists?

I'm not entirely sure that's correct.  We still ship a kernel metapackage, so DEFAULTKERNEL=kernel sounds more accurate to me.  Similarly kernel-PAE, etc for other situations.  If you just list kernel-core, it likely isn't going to work out well.

Agree, but this is the file that has been installed today as part of the fresh VM installation.

On both my F20 and Rawhide installs, the mtime of /etc/sysconfig/kernel is roughly the same as the install date of the latest kernel. I could populate /etc/sysconfig/kernel in Rawhide, but the question is, if I do, does the next kernel install preserve it? Or does it go back to 0 bytes?

Actually I've never seen in my life that file being 0 in size. Maybe it's the VirtualBox package corrupting it.

Anyway, if you populate it, it should stay as is.

(In reply to Josh Boyer from comment #13)
> I'm not entirely sure that's correct.  We still ship a kernel metapackage,
> so DEFAULTKERNEL=kernel sounds more accurate to me.  Similarly kernel-PAE,
> etc for other situations.  If you just list kernel-core, it likely isn't
> going to work out well.

In F20 it does say DEFAULTKERNEL=kernel. I just asked on #fedora-qa what other people's contents were, and roshi said his Rawhide install was several months old and had DEFAULTKERNEL=kernel (so his file is identical to the F20 version). I'm going to go with "kernel" and see what happens in the next kernel update. Maybe the 3.15 kernel changes (including the introduction of the kernel-core package) changed how it gets created from scratch.

Using the "kernel" version of /etc/sysconfig/kernel (copied from F20) I just did a Rawhide kernel update, and saw the sed error again. I'll try again using the "kernel-core" version and report back after the next kernel update. BTW, the file's mtime was updated during the installation of kernel-core, not kernel, which was installed after, and didn't change the mtime. Josh: please look into what this file's role is (it appears to be read and then rewritten by kernel-core, during its installation), whether it's deprecated (since the vbox dkms script is using it), and whether "kernel" or "kernel-core" is correct. Thanks.

Today's Rawhide kernel update (the first since the 7th), using the "kernel-core" version of /etc/sysconfig/kernel, still shows the sed error. So the fact that the file was empty wasn't relevant, and it's either a Fedora or VirtualBox bug.

My guess is that there is something else manipulating the file on your system and not VirtualBox. For sure not DKMS, as it never writes into that file, only parses it for output.

I've been running DKMS on all supported RHEL/Fedora distributions including rawhide for years and never experienced such problem.

Have you tried using RPMFusion's VirtualBox packages instead of VirtualBox.org provided ones?

As mentioned in the virtualbox.org bug report, this seems to be due to change in behaviour in /bin/bash.  Try running the following on both Fedora 20 and rawhide:

bash -x -c 'arr[0]=; arr[1]=; [[ ${arr[@]} ]] || echo false' 

This affects the line

[[ ${modules_conf_obsoletes[@]} ]] || return 0

in the moduleconfig_update_obsoletes() function of /sbin/dkms: on rawhide it fails to execute the "return 0".

Ouch, that's strange. I've looked on the web about changes in bash 4.3 but could not find anything about this. Even setting compatibility mode in bash to 4.2 does not solve the issue.

Looks like a bug.

For reference, Ubuntu 14.04 with bash 4.3.11(1) behaves like Fedora 20.

(In reply to Michael T from comment #23)
> For reference, Ubuntu 14.04 with bash 4.3.11(1) behaves like Fedora 20.

Many thanks for the information.

Ok, looks pretty certain that it's not dkms, and likely that it's the bash change. So the question is, is the bash change intentional, or a bug? Switching components. (Bash devs: the change in question is the one referred to in comment 21, and in https://www.virtualbox.org/ticket/13097#comment:5 .

Hello,

the bug already blocks #1109736; which is the bug related to the undocumented change. It also references the Virtualbox bug comment.

Michael T has already identified the commit where the issue was introduced.

Regards,
--Simone

Ok, so this needs an update in dkms itself for the more strict syntax. I'm preparing a build of the package with the particular line corrected, but probably this needs also some other adjustments to the code.

I don't have time to audit it all, so I'm posting a koji scratch build with the first patch.

I kindly ask for some testing feedback so we can maybe add all that's missing in additional steps. Unless, of course, someone wants to audit all dkms.

RPMS here:

http://koji.fedoraproject.org/koji/taskinfo?taskID=7051147

Using dkms-2.2.0.3-23.fc21.noarch, when I use dkms remove and dkms install to install the vbox guest modules for one of the kernels, near the end of the dkms install I still see the exact same sed error as with -22.

Adding any weak-modules

sed: -e expression #1, char 21: unterminated `s' command
depmod....

DKMS: install completed.

As an aside, I discovered that running dkms install causes /etc/sysconfig/kernel to be truncated to 0 bytes. This happens with either -22 or -23. (The reason I didn't see the extra output in the previous comment was presumably because the file had already been truncated to 0 bytes before and I didn't notice.)

[root@localhost ~]# ls -l /etc/sysconfig/kernel 
-rw-r--r--. 1 root root 185 Jun  7 09:47 /etc/sysconfig/kernel
[root@localhost ~]# dkms install vboxguest/4.3.12 -k 3.15.0-1.fc21.x86_64

Creating symlink /var/lib/dkms/vboxguest/4.3.12/source ->
                 /usr/src/vboxguest-4.3.12

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=3.15.0-1.fc21.x86_64 -C /lib/modules/3.15.0-1.fc21.x86_64/build M=/var/lib/dkms/vboxguest/4.3.12/build.........................
cleaning build area....

DKMS: build completed.

vboxguest.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-1.fc21.x86_64/extra/

vboxsf.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-1.fc21.x86_64/extra/

vboxvideo.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.15.0-1.fc21.x86_64/extra/
Adding any weak-modules

sed: -e expression #1, char 21: unterminated `s' command
/etc/sysconfig/kernel updated to replace obsoleted module references:
--- /tmp/dkms.u155Am/kernel.new	2014-06-19 03:49:12.205080158 -0400
+++ /etc/sysconfig/kernel	2014-06-07 09:47:50.424778080 -0400
@@ -0,0 +1,6 @@
+# UPDATEDEFAULT specifies if new-kernel-pkg should make
+# new kernels the default
+UPDATEDEFAULT=yes
+
+# DEFAULTKERNEL specifies the default kernel package type
+DEFAULTKERNEL=kernel-core
depmod....

DKMS: install completed.
[root@localhost ~]# ls -l /etc/sysconfig/kernel 
-rw-r--r--. 1 root root 0 Jun 19 03:49 /etc/sysconfig/kernel
[root@localhost ~]#

Ok, I think I got it right. I was able to reproduce the error (including the blanking of /etc/sysconfig/kernel) only with the VirtualBox rpm from virtualbox.org.

All the other DKMS modules I have (nvidia, fglrx, xpad) do not exhibit the issue at all. The fix:

http://pkgs.fedoraproject.org/cgit/dkms.git/tree/dkms-bash-syntax-fix.patch

First step is to make sure that the check on the "modules_conf_obsoletes" array does not bring you in the loop for obsolete modules.

Second part, in case the check is legitimate, makes sure your files are not destroyed by sed.

If you want to make sure the sed command is fixed, first restore line eight here on your system by editing /usr/sbin/dkms directly:

http://pkgs.fedoraproject.org/cgit/dkms.git/tree/dkms-bash-syntax-fix.patch#n8

and relaunch dkms remove/install again.

I've created another scratch build:

http://koji.fedoraproject.org/koji/taskinfo?taskID=7057073

Please re-populate your /etc/sysconfig/kernel before using this rpm and making the test. If it works, I'll build it in rawhide.

> If you want to make sure the sed command is fixed, first restore line eight here on your system by editing /usr/sbin/dkms directly:

I'm not sure whether this is something _I'm_ expected to do, but I'm guessing not, since you provided a binary dkms package. I simply tested that, after repopulating /etc/sysconfig/kernel, and dkms remove and dkms install both work now without any visible error, and without touching /etc/sysconfig/kernel.

(In reply to Andre Robatino from comment #32)
> I'm not sure whether this is something _I'm_ expected to do, but I'm
> guessing not, since you provided a binary dkms package.

Exactly, this was only to test it. Before the fix, line 8 was bringing you into a loop in which you should not have ended, and the rest of the sed lines were breaking your config file.

The package contains fix for both, but if you wanted to check the sed commands you had to restore line 8 (thus restoring the error).

Thanks for reporting; I've built the package in rawhide.

dkms-2.2.0.3-25.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/dkms-2.2.0.3-25.el6

dkms-2.2.0.3-25.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/dkms-2.2.0.3-25.el5

dkms-2.2.0.3-25.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/dkms-2.2.0.3-25.fc20

dkms-2.2.0.3-25.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/dkms-2.2.0.3-25.fc19

dkms-2.2.0.3-25.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

dkms-2.2.0.3-25.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

dkms-2.2.0.3-25.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

dkms-2.2.0.3-25.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.