Bug 704730 - fedora-storage-init-late.service is started twice
fedora-storage-init-late.service is started twice
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
15
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: LVM and device-mapper development team
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-05-14 08:31 EDT by cblaauw
Modified: 2013-01-07 14:14 EST (History)
21 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-11-09 04:24:28 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log of boot process (6.01 KB, text/plain)
2011-05-14 08:31 EDT, cblaauw
no flags Details
log of bash -x /lib/systemd/fedora-storage-init (2.69 KB, text/plain)
2011-05-15 11:54 EDT, cblaauw
no flags Details
output of systemctl status fedora-storage-init-late.service (422 bytes, text/plain)
2011-05-15 11:55 EDT, cblaauw
no flags Details
syslog (/var/log/messages) (65.04 KB, text/plain)
2011-05-15 11:56 EDT, cblaauw
no flags Details
output of lvm vgchange -a y --sysinit -vvv (12.92 KB, text/plain)
2011-05-17 12:29 EDT, cblaauw
no flags Details
/var/log/messages after adding 'StandardOutput=syslog' to /lib/systemd/system/fedora-storage-init-late.service (466.88 KB, text/plain)
2011-05-18 16:42 EDT, cblaauw
no flags Details

  None (edit)
Description cblaauw 2011-05-14 08:31:29 EDT
Created attachment 498925 [details]
log of boot process

The systemd service 'fedora-storage-init-late.service' is stared twice during boot and fails the second time. Boot log has been attached.



Additional info:

systemd-26-1.fc15.x86_64
Comment 1 Lennart Poettering 2011-05-15 09:19:52 EDT
Nah, those  two lines are fedora-storage-init.service and fedora-storage-init-late.service. We need this twice, since we need to support LUKS on LVM as well as LVM on LUKS.

I have no clue though, why the second invocation fails.

Could you attach syslog output, please? Also, could you attach the output of "systemctl status fedora-storage-init-late.service" please?

Finally, if you run "bash -x /lib/systemd/fedora-storage-init" as root on a terminal after booting, could you please paste the output that generates, here?

Tentatively reassigning to initscripts, since the storage scripts are from that package.
Comment 2 cblaauw 2011-05-15 11:54:52 EDT
Created attachment 499030 [details]
log of bash -x /lib/systemd/fedora-storage-init
Comment 3 cblaauw 2011-05-15 11:55:49 EDT
Created attachment 499031 [details]
output of systemctl status fedora-storage-init-late.service
Comment 4 cblaauw 2011-05-15 11:56:22 EDT
Created attachment 499032 [details]
syslog (/var/log/messages)
Comment 5 Bill Nottingham 2011-05-16 14:41:48 EDT
	 Process: 793 ExecStart=/lib/systemd/fedora-storage-init (code=exited, status=5)

The script itself never explicitly returns '5'.

Looking at the last thing in the script, that's LVM. And in its source code:

lib/commands/errors.h:#define ECMD_FAILED		5

Looks like lvm is returning an error for some reason.
Comment 6 Milan Broz 2011-05-16 16:09:53 EDT
And what's that last lvm thing in script which fails? (lvm --sysinit ... ?)

Can you run it from script with -vvvv (add -vvv to lvm command or run it manually with that) and paste output here?

I see no lvm error in the provided logs, we need that debug log to check if is lvm bug or bug in fedora-storage-init script.
Comment 7 Bill Nottingham 2011-05-16 16:50:48 EDT
The command, for reference, is:

 /sbin/lvm vgchange -a y --sysinit
Comment 8 cblaauw 2011-05-17 12:29:55 EDT
Created attachment 499408 [details]
output of lvm vgchange -a y --sysinit -vvv
Comment 9 Milan Broz 2011-05-17 13:57:35 EDT
Hm, there is nothing wrong in the log - does it return exit code 5 here?
Comment 10 cblaauw 2011-05-18 03:04:05 EDT
Can't tell right now what the exit code was. When I'm home again tonight I can do more checks. LVM does work without any noticeable deficencies for me. Is there a script that I can modify to capture the output and exit code of 'lvm vgchange -a y --sysinit -vvv' during boot?
Comment 11 Milan Broz 2011-05-18 03:28:13 EDT
You can modify /etc/lvm/lvm.conf to log to syslog or to file, but I think this is still run with root read-only...

Maybe syslog approach will work. For exit code i would simply add something like "echo $?" to that script as last line.
Comment 12 Michal Schmidt 2011-05-18 04:49:38 EDT
(In reply to comment #10)
> Is there a script that I can modify to capture the output and exit code of 'lvm
> vgchange -a y --sysinit -vvv' during boot?

Add this line to the [Service] section of /lib/systemd/system/fedora-storage-init-late.service:
StandardOutput=syslog

Then you'll find the output in /var/log/messages.
Comment 13 cblaauw 2011-05-18 16:42:58 EDT
Created attachment 499690 [details]
/var/log/messages after adding 'StandardOutput=syslog' to /lib/systemd/system/fedora-storage-init-late.service

I think this is the important part:

May 18 22:33:53 m7 kernel: [    5.392188] fedora-storage-init[810]: Logical Volume Management einrichten:   /var/lock/lvm/V_VolGroup00:aux: open failed: Datei oder Verzeichnis nicht gefunden
May 18 22:33:53 m7 kernel: [    5.392198] fedora-storage-init[810]: Can't get lock for VolGroup00
May 18 22:33:53 m7 kernel: [    5.395693] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
May 18 22:33:53 m7 kernel: [    5.405877] fedora-storage-init[810]: [60G[[0;31mFEHLGESCHLAGEN[0;39m]
May 18 22:33:53 m7 kernel: [    5.407215] systemd[1]: fedora-storage-init-late.service: main process exited, code=exited, status=5
May 18 22:33:53 m7 kernel: [    5.415089] systemd[1]: Unit fedora-storage-init-late.service entered failed state.
Comment 14 cblaauw 2011-05-19 01:22:36 EDT
So, now I see a problem, the lvm code tries to open /var/lock/lvm/V_VolGroup00, but /var is a logical volume in my setup, that is the path /var/lock is not there when the lvm script is executed.
Comment 15 Milan Broz 2011-05-19 06:02:03 EDT
Well. separate /var is not something systemd guys will like...

But for "lvm --sysinit ..." it should work (it should ignore locking error), let me try it...
Comment 16 Alasdair Kergon 2011-05-19 06:40:35 EDT
If you want to use /var as a logical volume then you must put your lvm locking dir somewhere else that is accessible by changing your lvm.conf - the new /run probably.  (We're in the process of considering how best to integrate lvm with systemd and default directory placements like this are under review.)
Comment 17 Michal Schmidt 2011-05-19 07:22:09 EDT
(In reply to comment #15)
> Well. separate /var is not something systemd guys will like...

I don't recall anyone objecting to a separate /var. Separate /usr is a different matter.

(In reply to comment #16)
> If you want to use /var as a logical volume then you must put your lvm locking
> dir somewhere else that is accessible by changing your lvm.conf - the new /run
> probably.  (We're in the process of considering how best to integrate lvm with
> systemd and default directory placements like this are under review.)

Yes, please. /run/lock/lvm would be a great default. In fact I don't even see a reason for it to remain configurable. /run is always available.

/var/lock is already a bind mount of /run/lock.
Comment 18 Alasdair Kergon 2011-05-19 07:36:53 EDT
(In reply to comment #17)
> In fact I don't even see a
> reason for it to remain configurable. /run is always available.

There are plenty of reasons - it will remain configurable.  (E.g. Some people can need multiple independent 'instances' of lvm on their system.)
Comment 19 cblaauw 2011-05-19 08:25:42 EDT
So I will reconfigure my lvm.conf to use /run/lock then.

Is the bug resolved then or do keep it open until the default in lvm.conf is changed?

Thanks for all the info and help,

Carsten
Comment 20 Lennart Poettering 2011-05-19 11:29:51 EDT
(In reply to comment #15)
> Well. separate /var is not something systemd guys will like...

Separate /var is absolutely something we support in systemd and I think is a good idea. In fact in the long run I'd really like to see / read-only by default with only /var and /home mounted writable.
Comment 21 cblaauw 2011-05-19 13:20:49 EDT
I reconfigured /etc/lvm/lvm.conf to use /run/lock/lvm as lock dir and the system boots fine now.
Comment 22 Milan Broz 2011-05-19 14:28:18 EDT
great, so I think lvm should switch in next update to using /run in Fedora - it will solve several other problems as well.
Comment 23 Bill Nottingham 2011-05-19 14:52:00 EDT
(In reply to comment #16)
> If you want to use /var as a logical volume then you must put your lvm locking
> dir somewhere else that is accessible by changing your lvm.conf - the new /run
> probably.  (We're in the process of considering how best to integrate lvm with
> systemd and default directory placements like this are under review.)

If I'm understanding this right, how did this work in prior releases? (Or didn't it?)
Comment 24 Peter Rajnoha 2011-11-08 10:04:16 EST
(In reply to comment #23)
> (In reply to comment #16)
> > If you want to use /var as a logical volume then you must put your lvm locking
> > dir somewhere else that is accessible by changing your lvm.conf - the new /run
> > probably.  (We're in the process of considering how best to integrate lvm with
> > systemd and default directory placements like this are under review.)
> 
> If I'm understanding this right, how did this work in prior releases? (Or
> didn't it?)

It did work since the --ignorelockingfailure/--sysinit option was used in rc.sysinit. The LV with /var on it was properly activated. LVM locking is not used at this early stage at all.

As for systemd, this should be the same. I'm not quite sure why we see these errors then (comment #13). I tried to quickly install F15 afresh to test this with /var on an LV, but I was not able to hit this error. Or is the configuration mentioned in this bug report any different from this one?

This the setup I've tested:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                7,7G  2,2G  5,5G  28% /
udev                  491M     0  491M   0% /dev
tmpfs                 500M     0  500M   0% /dev/shm
tmpfs                 500M  588K  500M   1% /run
/dev/mapper/vg-lv_root
                      7,7G  2,2G  5,5G  28% /
tmpfs                 500M     0  500M   0% /sys/fs/cgroup
tmpfs                 500M     0  500M   0% /media
/dev/vda1             485M   32M  429M   7% /boot
/dev/mapper/vg-lv_var
                      1,2G  210M  927M  19% /var


After adding the StandardOutput=syslog in fedora-storage-init.service and these extra lines in /lib/systemd/fedora-storage-init script (the df and lvs):

if [ -x /sbin/lvm ]; then
        /bin/df
        /sbin/lvm lvs
        action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a y --sysinit
fi

...I got this state logged in /var/log/messages:

========== snippet  ==========

/etc/init.d/functions: line 58: /dev/stderr: No such device or address
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 8062400   2217952   5762560  28% /
udev                    502636         0    502636   0% /dev
tmpfs                   511744         0    511744   0% /dev/shm
tmpfs                   511744       516    511228   1% /run
/dev/mapper/vg-lv_root
8062400   2217952   5762560  28% /
tmpfs                   511744         0    511744   0% /sys/fs/cgroup
tmpfs                   511744         0    511744   0% /media
/dev/vda1               495844     31878    438366   7% /boot
LV      VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root vg   -wi-ao   7.81g
lv_swap vg   -wi-ao 512.00m
lv_var  vg   -wi---   1.19g
Setting up Logical Volume Management:   3 logical volume(s) in volume group "vg" now active
^[[60G[^[[0;32m  OK  ^[[0;39m]
/etc/init.d/functions: line 58: /dev/stderr: No such device or address
/dev/mapper/vg-lv_var: clean, 10380/77920 files, 58517/311296 blocks
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 8062400   2217952   5762560  28% /
udev                    502636         0    502636   0% /dev
tmpfs                   511744         0    511744   0% /dev/shm
tmpfs                   511744       520    511224   1% /run
/dev/mapper/vg-lv_root
8062400   2217952   5762560  28% /
tmpfs                   511744         0    511744   0% /sys/fs/cgroup
tmpfs                   511744         0    511744   0% /media
/dev/vda1               495844     31878    438366   7% /boot
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
LV      VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root vg   -wi-ao   7.81g
lv_swap vg   -wi-ao 512.00m
lv_var  vg   -wi-ao   1.19g
Setting up Logical Volume Management:   3 logical volume(s) in volume group "vg" now active
^[[60G[^[[0;32m  OK  ^[[0;39m]

===============================

The "EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)", where dm-2 is lv_var exactly, is the /var that gets mounted.

cblaauw, if you still have this setup, could you please post the output of lsblk to see what's your device setup and the /etc/lvm/lvm.conf, maybe there's some specialty that's causing problems...


As for the transition from /var to /run, there's already a bz for that: bz #742554.
Comment 25 cblaauw 2011-11-09 04:06:52 EST
I moved to F-16 already, but the layout is still the same. It did mount when the lockdir was in "/var/lock/lvm", but there where errors displayed by systemd.

Right now I have "/run/lock/lvm" set as lockdir in lvm.conf and that works perfectly.
Comment 26 Peter Rajnoha 2011-11-09 04:24:28 EST
(In reply to comment #25)
> Right now I have "/run/lock/lvm" set as lockdir in lvm.conf and that works
> perfectly.

This is already scheduled to be set this way by default in next release of Fedora. So let's move on. I'll close this bug with NEXTRELEASE. The other bz to track is the #742554.

Note You need to log in before you can comment on or make changes to this bug.