Bug 704730

Nah, those  two lines are fedora-storage-init.service and fedora-storage-init-late.service. We need this twice, since we need to support LUKS on LVM as well as LVM on LUKS.

I have no clue though, why the second invocation fails.

Could you attach syslog output, please? Also, could you attach the output of "systemctl status fedora-storage-init-late.service" please?

Finally, if you run "bash -x /lib/systemd/fedora-storage-init" as root on a terminal after booting, could you please paste the output that generates, here?

Tentatively reassigning to initscripts, since the storage scripts are from that package.

Created attachment 499030 [details]
log of bash -x /lib/systemd/fedora-storage-init

Created attachment 499031 [details]
output of systemctl status fedora-storage-init-late.service

Created attachment 499032 [details]
syslog (/var/log/messages)

	 Process: 793 ExecStart=/lib/systemd/fedora-storage-init (code=exited, status=5)

The script itself never explicitly returns '5'.

Looking at the last thing in the script, that's LVM. And in its source code:

lib/commands/errors.h:#define ECMD_FAILED		5

Looks like lvm is returning an error for some reason.

And what's that last lvm thing in script which fails? (lvm --sysinit ... ?)

Can you run it from script with -vvvv (add -vvv to lvm command or run it manually with that) and paste output here?

I see no lvm error in the provided logs, we need that debug log to check if is lvm bug or bug in fedora-storage-init script.

The command, for reference, is:

 /sbin/lvm vgchange -a y --sysinit

Created attachment 499408 [details]
output of lvm vgchange -a y --sysinit -vvv

Hm, there is nothing wrong in the log - does it return exit code 5 here?

Can't tell right now what the exit code was. When I'm home again tonight I can do more checks. LVM does work without any noticeable deficencies for me. Is there a script that I can modify to capture the output and exit code of 'lvm vgchange -a y --sysinit -vvv' during boot?

You can modify /etc/lvm/lvm.conf to log to syslog or to file, but I think this is still run with root read-only...

Maybe syslog approach will work. For exit code i would simply add something like "echo $?" to that script as last line.

(In reply to comment #10)
> Is there a script that I can modify to capture the output and exit code of 'lvm
> vgchange -a y --sysinit -vvv' during boot?

Add this line to the [Service] section of /lib/systemd/system/fedora-storage-init-late.service:
StandardOutput=syslog

Then you'll find the output in /var/log/messages.

Created attachment 499690 [details]
/var/log/messages after adding 'StandardOutput=syslog' to /lib/systemd/system/fedora-storage-init-late.service

I think this is the important part:

May 18 22:33:53 m7 kernel: [    5.392188] fedora-storage-init[810]: Logical Volume Management einrichten:   /var/lock/lvm/V_VolGroup00:aux: open failed: Datei oder Verzeichnis nicht gefunden
May 18 22:33:53 m7 kernel: [    5.392198] fedora-storage-init[810]: Can't get lock for VolGroup00
May 18 22:33:53 m7 kernel: [    5.395693] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
May 18 22:33:53 m7 kernel: [    5.405877] fedora-storage-init[810]: [60G[[0;31mFEHLGESCHLAGEN[0;39m]
May 18 22:33:53 m7 kernel: [    5.407215] systemd[1]: fedora-storage-init-late.service: main process exited, code=exited, status=5
May 18 22:33:53 m7 kernel: [    5.415089] systemd[1]: Unit fedora-storage-init-late.service entered failed state.

So, now I see a problem, the lvm code tries to open /var/lock/lvm/V_VolGroup00, but /var is a logical volume in my setup, that is the path /var/lock is not there when the lvm script is executed.

Well. separate /var is not something systemd guys will like...

But for "lvm --sysinit ..." it should work (it should ignore locking error), let me try it...

If you want to use /var as a logical volume then you must put your lvm locking dir somewhere else that is accessible by changing your lvm.conf - the new /run probably.  (We're in the process of considering how best to integrate lvm with systemd and default directory placements like this are under review.)

(In reply to comment #15)
> Well. separate /var is not something systemd guys will like...

I don't recall anyone objecting to a separate /var. Separate /usr is a different matter.

(In reply to comment #16)
> If you want to use /var as a logical volume then you must put your lvm locking
> dir somewhere else that is accessible by changing your lvm.conf - the new /run
> probably.  (We're in the process of considering how best to integrate lvm with
> systemd and default directory placements like this are under review.)

Yes, please. /run/lock/lvm would be a great default. In fact I don't even see a reason for it to remain configurable. /run is always available.

/var/lock is already a bind mount of /run/lock.

(In reply to comment #17)
> In fact I don't even see a
> reason for it to remain configurable. /run is always available.

There are plenty of reasons - it will remain configurable.  (E.g. Some people can need multiple independent 'instances' of lvm on their system.)

So I will reconfigure my lvm.conf to use /run/lock then.

Is the bug resolved then or do keep it open until the default in lvm.conf is changed?

Thanks for all the info and help,

Carsten

(In reply to comment #15)
> Well. separate /var is not something systemd guys will like...

Separate /var is absolutely something we support in systemd and I think is a good idea. In fact in the long run I'd really like to see / read-only by default with only /var and /home mounted writable.

I reconfigured /etc/lvm/lvm.conf to use /run/lock/lvm as lock dir and the system boots fine now.

great, so I think lvm should switch in next update to using /run in Fedora - it will solve several other problems as well.

(In reply to comment #16)
> If you want to use /var as a logical volume then you must put your lvm locking
> dir somewhere else that is accessible by changing your lvm.conf - the new /run
> probably.  (We're in the process of considering how best to integrate lvm with
> systemd and default directory placements like this are under review.)

If I'm understanding this right, how did this work in prior releases? (Or didn't it?)

(In reply to comment #23)
> (In reply to comment #16)
> > If you want to use /var as a logical volume then you must put your lvm locking
> > dir somewhere else that is accessible by changing your lvm.conf - the new /run
> > probably.  (We're in the process of considering how best to integrate lvm with
> > systemd and default directory placements like this are under review.)
> 
> If I'm understanding this right, how did this work in prior releases? (Or
> didn't it?)

It did work since the --ignorelockingfailure/--sysinit option was used in rc.sysinit. The LV with /var on it was properly activated. LVM locking is not used at this early stage at all.

As for systemd, this should be the same. I'm not quite sure why we see these errors then (comment #13). I tried to quickly install F15 afresh to test this with /var on an LV, but I was not able to hit this error. Or is the configuration mentioned in this bug report any different from this one?

This the setup I've tested:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                7,7G  2,2G  5,5G  28% /
udev                  491M     0  491M   0% /dev
tmpfs                 500M     0  500M   0% /dev/shm
tmpfs                 500M  588K  500M   1% /run
/dev/mapper/vg-lv_root
                      7,7G  2,2G  5,5G  28% /
tmpfs                 500M     0  500M   0% /sys/fs/cgroup
tmpfs                 500M     0  500M   0% /media
/dev/vda1             485M   32M  429M   7% /boot
/dev/mapper/vg-lv_var
                      1,2G  210M  927M  19% /var


After adding the StandardOutput=syslog in fedora-storage-init.service and these extra lines in /lib/systemd/fedora-storage-init script (the df and lvs):

if [ -x /sbin/lvm ]; then
        /bin/df
        /sbin/lvm lvs
        action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a y --sysinit
fi

...I got this state logged in /var/log/messages:

========== snippet  ==========

/etc/init.d/functions: line 58: /dev/stderr: No such device or address
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 8062400   2217952   5762560  28% /
udev                    502636         0    502636   0% /dev
tmpfs                   511744         0    511744   0% /dev/shm
tmpfs                   511744       516    511228   1% /run
/dev/mapper/vg-lv_root
8062400   2217952   5762560  28% /
tmpfs                   511744         0    511744   0% /sys/fs/cgroup
tmpfs                   511744         0    511744   0% /media
/dev/vda1               495844     31878    438366   7% /boot
LV      VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root vg   -wi-ao   7.81g
lv_swap vg   -wi-ao 512.00m
lv_var  vg   -wi---   1.19g
Setting up Logical Volume Management:   3 logical volume(s) in volume group "vg" now active
^[[60G[^[[0;32m  OK  ^[[0;39m]
/etc/init.d/functions: line 58: /dev/stderr: No such device or address
/dev/mapper/vg-lv_var: clean, 10380/77920 files, 58517/311296 blocks
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 8062400   2217952   5762560  28% /
udev                    502636         0    502636   0% /dev
tmpfs                   511744         0    511744   0% /dev/shm
tmpfs                   511744       520    511224   1% /run
/dev/mapper/vg-lv_root
8062400   2217952   5762560  28% /
tmpfs                   511744         0    511744   0% /sys/fs/cgroup
tmpfs                   511744         0    511744   0% /media
/dev/vda1               495844     31878    438366   7% /boot
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
LV      VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root vg   -wi-ao   7.81g
lv_swap vg   -wi-ao 512.00m
lv_var  vg   -wi-ao   1.19g
Setting up Logical Volume Management:   3 logical volume(s) in volume group "vg" now active
^[[60G[^[[0;32m  OK  ^[[0;39m]

===============================

The "EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)", where dm-2 is lv_var exactly, is the /var that gets mounted.

cblaauw, if you still have this setup, could you please post the output of lsblk to see what's your device setup and the /etc/lvm/lvm.conf, maybe there's some specialty that's causing problems...


As for the transition from /var to /run, there's already a bz for that: bz #742554.

I moved to F-16 already, but the layout is still the same. It did mount when the lockdir was in "/var/lock/lvm", but there where errors displayed by systemd.

Right now I have "/run/lock/lvm" set as lockdir in lvm.conf and that works perfectly.

(In reply to comment #25)
> Right now I have "/run/lock/lvm" set as lockdir in lvm.conf and that works
> perfectly.

This is already scheduled to be set this way by default in next release of Fedora. So let's move on. I'll close this bug with NEXTRELEASE. The other bz to track is the #742554.