Bug 290731

Summary: udev error message on startup node_symlink: device /dev/rtc already exists...
Product: [Fedora] Fedora Reporter: Gary Yuen <garyyuen>
Component: kernelAssignee: Bill Nottingham <notting>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: rawhideCC: cra, harald, hdegoede, jfrieben, jonstanley, katzj, k.georgiou, mattdm, michal, mjs, notting, pjones, rvokal, sgrubb, wwoods
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-02-04 11:15:22 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On: 431272, 431274    
Bug Blocks: 235706, 430962    
Attachments:
Description Flags
PATCH: proof of concept fix to mkinitrd
none
clock rules file none

Description Gary Yuen 2007-09-14 08:47:38 EDT
Description of problem:

Error message on startup with Fedora 8 Test 2:

Udev started: udev-devent[919]: node_symlink: device node '/dev/rtc' already exists, link to '/dev/rtc0' 
wil not overwrite it

Version-Release number of selected component (if applicable):

2.fc8

How reproducible:

not sure

Steps to Reproduce:
1. not sure
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Harald Hoyer 2007-09-18 08:45:26 EDT
From linux-hotplug-devel@lists.sourceforge.net:
> > It looks like your kernel configuration is not valid. You have compiled
> > both the old-style "rtc" (or "genrtc"?) driver and the new "rtc_cmos"
> > driver. Please remove one of them, and the problem will go away.
> >
> > (please correct me if I am wrong - I have never played with the new RTC
> > drivers).

> Yes, the old "rtc" module should not be provided if the new ones are
> enabled. The info message of udev is the expected behavior, as no
> symlink should overwrite a device node of a different device.
Comment 2 Gary Yuen 2007-09-18 12:15:41 EDT
I've always been using the included kernel with Fedora 8. Btw, my Fedora is running under VMware.
Comment 3 Harald Hoyer 2007-09-18 12:28:57 EDT
I know... It's a Fedora kernel configuration error.
Comment 4 Dave Jones 2007-09-24 13:07:37 EDT
What exactly is wrong ? We're not building the old driver for some time now.

config-generic:# CONFIG_RTC is not set
config-generic:CONFIG_GEN_RTC=y
config-generic:CONFIG_RTC_DRV_CMOS=m
Comment 5 Will Woods 2007-09-24 15:53:07 EDT
This happens on my machines as well. udev is creating /dev/rtc and /dev/rtc0:
[wwoods@brinstar ~]$ ls -al /dev/rtc*
crw-r--r-- 1 root root  10, 135 2007-09-24 15:45 /dev/rtc
crw-r--r-- 1 root root 254,   0 2007-09-24 15:45 /dev/rtc0

but /etc/udev/rules.d/50-udev-default.rules has:
KERNEL=="rtc|rtc0",             MODE="0644"
KERNEL=="rtc0",                 SYMLINK+="rtc"

so it wants to replace /dev/rtc with a link to /dev/rtc0. As you can see,
though, /dev/rtc and /dev/rtc0 are completely different devices.

So - which is the new one, which is the old one, and why are they both there if
we're only including one? 

I'm guessing here, but maybe udev saved the old /dev/rtc and is restoring it on
startup? 
Comment 6 Michal Jaegermann 2007-09-24 15:54:29 EDT
I do not recall seeing that earlier, although I could have
missed it, and currently I have on my system these kernels:
2.6.23-0.139.rc3.git10.fc8
2.6.23-0.142.rc3.git10.fc8
2.6.23-0.195.rc3.git10.fc8
(I was away and I was catching up with updates).  The error
message shows up always and with all of the above. It is clearly
produced by some udev action and it looks like that both

crw-r--r-- 1 root root  10, 135 2007-09-24 13:43 /dev/rtc
crw-r--r-- 1 root root 254,   0 2007-09-24 13:43 /dev/rtc0

were created by udev before an attempt to create symlink to
/dev/rtc0.  So maybe this is some udev hiccup?
Comment 7 Dave Jones 2007-09-24 17:56:51 EDT
Ah, so there's not two sets of RTC options in the kernel, but three.
I'm about to build a kernel with the GEN_RTC stuff disabled. That should allow
the rtc_cmos driver to claim the hardware, and all this should come out in the wash.
Comment 8 Michal Jaegermann 2007-09-26 18:08:57 EDT
With kernel 2.6.23-0.202.rc8.fc8 the situation sharply deteriorated.
'udev' still creates two nodes /dev/rtc and /dev/rtc0 like the above,
and there is still the same complaint but one also gets immediately
during the startup: "Cannot access the Hardware Clock via any known method"
and a suggestion to use --debug (presumably from hwclock).

'hwclock --debug' ends with this:

hwclock: Open of /dev/rtc failed, errno=19: No such device.
Cannot access the Hardware Clock via any known method.
hwclock from util-linux-ng 2.13
No usable clock interface found.

Rebooting with an older kernel keeps a complaint from node_symlink
but at least the hardware clock can be accessed again.
Comment 9 Michal Jaegermann 2007-09-27 15:32:53 EDT
The same as in comment #8 happens with 2.6.23-0.204.rc8.fc8;
but if I will do:

           cd /dev/ && ln -nsf rtc0 rtc

(which presumably 'node_symlink' attempted in the first place)
then I see:
# hwclock --debug
hwclock from util-linux-ng 2.13
Using /dev interface to clock.
Last drift adjustment done at 1190920537 seconds after 1969
Last calibration done at 1190920537 seconds after 1969
Hardware clock is on UTC time
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
/dev/rtc does not have interrupt functions. Waiting in loop for time from
/dev/rtc to change
...got clock tick
Time read from Hardware Clock: 2007/09/27 19:30:44
Hw clock time : 2007/09/27 19:30:44 = 1190921444 seconds since 1969
Thu 27 Sep 2007 01:30:44 PM MDT  -0.624103 seconds

In other words - it looks like that udev is conspiring with kernel
to screw us up. :-)
Comment 10 Will Woods 2007-09-27 15:40:14 EDT
This is definitely fixed for me in kernel .211, and is probably fixed somewhere
between .204 and here. I'd like for someone to confirm with tomorrow's rawhide
before closing this bug, though.
Comment 11 Chuck Ebbert 2007-09-28 18:18:18 EDT
Fixed here.
Comment 12 David Woodhouse 2007-12-03 18:09:28 EST
Reopening, because I just turned the generic RTC class back on.

Firstly, mkinitrd is creating /dev/rtc with the old device numbers.

Secondly, hwclock isn't managing to fall back to /dev/rtc0 (because it'll only
fall back if it gets -ENOENT, and it's actually getting -ENODEV). The latter is
bug #409551
Comment 13 Michal Jaegermann 2007-12-07 15:42:09 EST
This breaks again with rawhide kernel-2.6.24-0.73.rc4.git1.fc9.
Symptoms and cures are precisely like those described in the
original report and follow up comments.
Comment 14 Hans de Goede 2008-01-24 17:21:54 EST
Hi All,

I've been seeing this too, and I've done some analysing of the situation.

As said by Dave J, there are 3 different rtc drivers for the classic PC RTC:
CONFIG_RTC
CONFIG_GEN_RTC
CONFIG_RTC_DRV_CMOS

With my current rawhide kernel: /2.6.24-0.164.rc8.git4.fc9, according to
/boot/config-2.6.24-0.164.rc8.git4.fc9:
# CONFIG_RTC is not set
# CONFIG_GEN_RTC is not set 
CONFIG_RTC_DRV_CMOS=m   

Notice that is "CONFIG_RTC_DRV_CMOS=m" not "CONFIG_RTC_DRV_CMOS=y", this is part
of the problem.

mkinitrd creates /dev/rtc (currently with the wrong major and minor, for the old
CONFIG_RTC/CONFIG_GEN_RTC driver), because /etc/rc.d/rc.sysinit does a 
"hwclock --hctosys" before doing anything else. So that is before starting udev,
I believe that at this time rtc_cmos.ko has not been loaded yet (and
/var/log/messages confirms this). This causes hwclock to give the error:
"Cannot access the Hardware Clock via any known method."

As at this time, there is no rtc support of any kind in the kernel, this is the
first thing we need to fix one way or the other.

Once we fix this, things still won't work as /dev/rtc is created by initrd with
the major minor of 10, 135. The patch of dwmw2 to make hwclock next try
/dev/rtc0 is correct and is in the current rawhide hwclock, but is if no use
since udev hasn't run yet, so there will be no /dev/rtc0 to fallback to unless
we teach (mk)initrd to create it.

---

That still leaves the problem of udev rightfully complaining:
Udev started: udev-devent[919]: node_symlink: device node '/dev/rtc' already
exists, link to '/dev/rtc0' 
wil not overwrite it

The error is a bit cryptic, but I think udev is trying to say it wants to make
/dev/rtc a link to /dev/rtc0 but that it cannot do that as /dev/rtc already exists.

If we want to get rid of this error too (and we do IMHO) then mkinitrd must be
thaught to recognize which rtc system (old /dev/rtc 10, 135 or new /dev/rtc0
254, 0, /dev/rtc symlink to /dev/rtc0) the kernel is using. I would like to
suggest to detect this by detecting if CONFIG_RTC_CLASS=[y,m] is present in
/boot/config......

Note that I'm not checking for CONFIG_RTC_DRV_CMOS here, as that is too platform
dependent IMHO.

And ifso then create /dev/rtc0 and a /dev/rtc symlink, and otherwise fall back
to the old behaviour.

---

So summarizing we need to:
1) make sure rtc_cmos.ko gets loaded before hwclock gets run by rc.sysinit 
 possible solutions:
 a) Don't make it a module (kernel change)
 b) do udev init before running hwclock (rc.sysinit change)
 c) teach mkinitrd to load rtc_cmos.ko from initrd on relevant archs

2) teach mkinitrd about old versus new rtc subsys, using /boot/config..... 
 to detect this, and generate the correct device (and symlink) depended on
 which rtc subsys is used. I can only hope that udev will not complain if
 the node / symlink it wants to create already exists and is identical to
 what it wants (I haven't tested this yet, will do tomorrow).

Note that I'm willing to write an mkinitrd patch for 2) and for 1c) if that is
the choisen path, but I would first like to hear what the resolution of 1) is
going to be, and if such a patch would need to support things like
CONFIG_RTC_CLASS=m, CONFIG_RTC_INTF_DEV=m
Comment 15 Michal Jaegermann 2008-01-25 15:29:40 EST
This bug is marked as dependent on bug 409551.  But what Hans de Goede
claims amounts to that if hwclock does not fail to another device,
if the one tried first is a non-working one, is a correct behaviour
and tough luck. Assuming that this claim is indeed accepted then
this dependency is wrong and Hans actually closed bug 409551 arguing
like the above and without any changes to the current hwclock behaviour.

That shifts a burden to udev, or whatever else provides the first
device for hwclock to try, to get it right or else ...

It seems possible to work around these issues by changes in
startup files.
Comment 16 Hans de Goede 2008-01-25 15:35:38 EST
(In reply to comment #15)
> This bug is marked as dependent on bug 409551.  But what Hans de Goede
> claims amounts to that if hwclock does not fail to another device,
> if the one tried first is a non-working one, is a correct behaviour
> and tough luck.

No, that is NOT what I'm claiming, hwclock is trying to fall back, but there is
nothing to fall back too, at this (very early) point in the boot sequence the
kernel doesn't know howto talk to the rtc yet as the appropriate driver has not
yet been loaded, so no amount of falling back will help here, what needs to be
done instead is either load the rtc driver earlier, or run hwclock later.

Comment 17 Michal Jaegermann 2008-01-25 16:09:12 EST
> No, that is NOT what I'm claiming, hwclock is trying to fall back

The reasons why this looks that way are as follows:

# ll /dev/rtc*
crw------- 1 root root  10, 135 2008-01-25 13:42 /dev/rtc
crw-r--r-- 1 root root 254,   0 2008-01-25 13:43 /dev/rtc0

This was created by udev.

# /sbin/hwclock --rtc=/dev/rtc --debug
hwclock from util-linux-ng 2.13.1-rc2
hwclock: Open of /dev/rtc failed, errno=19: No such device.
No usable clock interface found.
Cannot access the Hardware Clock via any known method.

OTOH you are right that with the current version of hwclock,
if --rtc=... was not specified and /dev/rtc access returns with
ENODEV, then /dev/rtc0 is tried.  This is indeed a difference
with older hwclock versions.  I should check more carefuly.

That still does not make this bug dependent on bug 409551.
Comment 18 Hans de Goede 2008-01-25 18:00:00 EST
Created attachment 293019 [details]
PATCH: proof of concept fix to mkinitrd

This patch modifies mkinitrd to check if the kernel its generating an initrd
for uses the new rtc class style drivers (check done through /boot/config-xxx).


If the new rtc class driver is detected, then the following is done from the
initrd:
mknod /dev/rtc0 c 254 0
ln -s rtc0 /dev/rtc
modprobe rtc_cmos

instead of:
mknod /dev/rtc c 10 235

(And rtc_cmos.ko and ln are added to the initrd to make this actually work).

With this things work as they should, udev still gives a (new) spurious error
about not being able to set the selinux context of /dev/rtc, but using ls -Z
the context seems fine after boot, so I consider this an udev and/or selinux
bug.

To test this, download the patch somewhere, and then:
cd /sbin
patch -p0 < @path_to@/mkinitrd-bug290731.patch
mkinitrd -f /boot/initrd-@kernel_version@.img @kernel_version@

Where @kernel_version@ is for example: 2.6.24-0.164.rc8.git4.fc9
and @path_to@ depends on where you downloaded the patch.

As said this is a proof of concept, for example I think it would be better to
just have rtc_cmos compiled into the kernel, which would simplify the patch
somewhat, and I wonder if /dev/rtc is needed at all for any internal use by
initrd itself, if not then the creation of the symlink can be omitted too.
Comment 19 Jeremy Katz 2008-01-30 13:30:46 EST
(In reply to comment #18)
> Created an attachment (id=293019) [edit]
> PATCH: proof of concept fix to mkinitrd
> 
> This patch modifies mkinitrd to check if the kernel its generating an initrd
> for uses the new rtc class style drivers (check done through /boot/config-xxx).

Doing something like this is pretty fragile, though, as it requires the
/boot/config-xxx to exist which isn't the case for anyone building a custom
kernel.  Also, depending on module names and config options not changing is a
pretty quick path to having things change.  I think the better approach is going
to involve getting the manual creation of the rtc device node out of the initrd.
 Either by moving hwclock to be after udev in rc.sysinit or by having hwclock be
run as a udev rule.
Comment 20 Hans de Goede 2008-01-30 13:47:13 EST
(In reply to comment #19)
> Doing something like this is pretty fragile.

I agree this ain't pretty.

> I think the better approach is going
> to involve getting the manual creation of the rtc device node out of the initrd.
>  Either by moving hwclock to be after udev in rc.sysinit or by having hwclock be
> run as a udev rule.

Or as a third option if you really want to have hwclock set before starting
udev, you could add something like this at the top of rc.sysinit:

RTC_MAJOR=`grep rtc /proc/devices | cut -d ' ' -f 1`
if [ "$RTC_MAJOR" ] ; then
  # remove oldstyle /dev/rtc created by initrd
  rm /dev/rtc
  # and create newstyle /dev/rtc
  mknod /dev/rtc0 c $RTC_MAJOR 0
fi
Comment 21 Chuck Ebbert 2008-01-30 14:03:44 EST
(In reply to comment #20)
> (In reply to comment #19)
> > Doing something like this is pretty fragile.
> 
> RTC_MAJOR=`grep rtc /proc/devices | cut -d ' ' -f 1`
> if [ "$RTC_MAJOR" ] ; then
>   # remove oldstyle /dev/rtc created by initrd
>   rm /dev/rtc
>   # and create newstyle /dev/rtc
>   mknod /dev/rtc0 c $RTC_MAJOR 0
> fi
> 

You also want to create a symlink from /dev/rtc to /dev/rtc0:

ln -s /dev/rtc0 /dev/rtc


Comment 22 Michal Jaegermann 2008-01-30 14:15:46 EST
RTC_MAJOR=$(grep -w rtc /proc/devices | (read a b ; echo $a)) or
RTC_MAJOR=$(grep -w rtc /proc/devices}; RTC_MAJOR="${RTC_MAJOR%% *}"
constructs will avoid 'cut' if you do not want simply
RTC_MAJOR=$(awk '$2 == "rtc"{print $1}' /proc/devices)
Also /dev/rtc0 may already exist so 'mknod' should be guarded
by a check.

Symlink is not really required with the current hwclock.
Comment 23 Jeremy Katz 2008-01-30 14:19:57 EST
(In reply to comment #20)
> Or as a third option if you really want to have hwclock set before starting
> udev, you could add something like this at the top of rc.sysinit:

Given that we depend on the fact that we're doing udev and a ramfs /dev these
days, this is doable now.  We had to make the devices in the initrd when we were
transitioning from static /dev -> udev and letting people do either.  Bill --
what do you think?
Comment 24 Chuck Ebbert 2008-01-30 14:27:17 EST
(In reply to comment #22)
> 
> Symlink is not really required with the current hwclock.

Other programs might need one though. And /dev/rtc should point to the rtc that
programs should be using -- can hwclock cope with someone loading a second rtc
as /dev/rtc1 and removing /dev/rtc0?
Comment 25 Bill Nottingham 2008-01-30 14:38:46 EST
I'm tempted to just move it out of rc.sysinit and into a udev rule. Something
like a /lib/udev/clock:
#!/bin/bash

UTC=
if [ -f /etc/sysconfig/clock ]; then
   . /etc/sysconfig/clock

   # convert old style clock config to new values
   if [ "${CLOCKMODE}" = "GMT" ]; then
      UTC=true
   fi
fi

CLOCKFLAGS="$CLOCKFLAGS --hctosys"

case "$UTC" in
    yes|true)	CLOCKFLAGS="$CLOCKFLAGS --utc"
    no|false)	CLOCKFLAGS="$CLOCKFLAGS --localtime"
esac

[ -x /sbin/hwclock ] && /sbin/hwclock $CLOCKFLAGS

and a clock rules file that does:

ACTION=="add", SUBSYSTEM=="rtc", RUN+="/lib/udev/clock"
Comment 26 Hans de Goede 2008-01-30 15:49:54 EST
(In reply to comment #24)
> (In reply to comment #22)
> > 
> > Symlink is not really required with the current hwclock.
> 
> Other programs might need one though.

And by the time those other programs run, udev will be up and running and will
have created the symlink, having that symlink created before udev runs actually
causes udev to spew error messages to the console that it already exists.

> And /dev/rtc should point to the rtc that
> programs should be using -- can hwclock cope with someone loading a second rtc
> as /dev/rtc1 and removing /dev/rtc0?
> 

udev should take care of the symlink always pointing at a valid rtc device.
Comment 27 Hans de Goede 2008-01-30 15:52:57 EST
(In reply to comment #25)
> I'm tempted to just move it out of rc.sysinit and into a udev rule. Something
> like a /lib/udev/clock:
>
> and a clock rules file that does:
> 
> ACTION=="add", SUBSYSTEM=="rtc", RUN+="/lib/udev/clock"
> 

How will this work when someone has a system which actually has 2 rtc's ? What
if one of these 2 rtc's come and go through hotplugging? I know this is somewhat
academic, but maybe in the future some usb device with an rtc will show up, or
maybe some gps driver will also export the gps time as an rtc? or ... who knows?

Note that I'm not against an udev based solution just playing devils advocate here.
Comment 28 Bill Nottingham 2008-01-30 22:40:18 EST
Well, hwclock would (almost certainly) just grab the first one. You could pass
--rtc=/dev/%k, if you wanted.
Comment 29 Hans de Goede 2008-01-31 03:07:46 EST
Shouldn't the udev rule to launch the script then be made so that the script
only gets launched for the first rtc detected?
Comment 30 Bill Nottingham 2008-01-31 11:27:48 EST
First is a vague concept in udev.
Comment 31 Bill Nottingham 2008-02-01 16:53:53 EST
Created attachment 293768 [details]
clock rules file

Here's a rules file that should DTRT, modulo the dependent bugs listed below.
Comment 32 Jon Stanley 2008-02-01 23:04:21 EST
Proposing this as a beta blocker.
Comment 33 Joachim Frieben 2008-02-04 08:32:01 EST
After upgrading to initscripts-8.63-1 and mkinitrd-6.0.28-4.fc9
and a subsequent reinstall of kernel 2.6.24-9.fc9, the issue has
ceased to affect my system. No error messages anymore and the
system time is not altered either. Thanks!
Comment 34 Bill Nottingham 2008-02-04 11:15:22 EST
Fixed in initscripts-8.63-1, mkinitrd-6.0.28-4.fc9.