Bug 1206481

Summary: Cannot add dependency job for unit mdmonitor.service, ignoring: Unit mdmonitor.service failed to load: No such file or directory
Product: [Fedora] Fedora Reporter: Harald Reindl <h.reindl>
Component: mdadmAssignee: Nigel Croxon <ncroxon>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 24CC: agk, davide.poletto, dledford, dracut-maint-list, goeran, harald, h.reindl, jonathan, mbroz, ncroxon, xni, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-14 11:50:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
requested debug-log none

Description Harald Reindl 2015-03-27 09:01:04 UTC
Mar 27 09:57:39 rh systemd: Cannot add dependency job for unit mdmonitor.service, ignoring: Unit mdmonitor.service failed to load: No such file or directory.

looks like early boot and dracut because the service is there and running fine

[root@rh:~]$ systemctl status mdmonitor.service
● mdmonitor.service - Software RAID monitoring and management
   Loaded: loaded (/usr/lib/systemd/system/mdmonitor.service; enabled)
   Active: active (running) since Fr 2015-03-27 09:57:44 CET; 3min 10s ago
  Process: 795 ExecStart=/sbin/mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid (code=exited, status=0/SUCCESS)
 Main PID: 796 (mdadm)
   CGroup: /system.slice/mdmonitor.service
           └─796 /sbin/mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid

Mär 27 09:57:44 rh.thelounge.net systemd[1]: Started Software RAID monitoring and management

Comment 1 Harald Hoyer 2015-04-23 11:53:55 UTC
You are right. dracut does not install mdmonitor.service.

What's the output of:
$ grep -r mdmonitor /{etc,usr/lib}/systemd/system/

And:
$ systemctl show mdmonitor.service

Comment 2 Harald Reindl 2015-04-23 12:21:07 UTC
that's all fine and as said that happens in early boot / dracut

[root@rh:~]$ grep -r mdmonitor /{etc,usr/lib}/systemd/system/
/usr/lib/systemd/system/mdmonitor.service:EnvironmentFile=-/etc/sysconfig/mdmonitor

[root@rh:~]$ systemctl status mdmonitor.service
● mdmonitor.service - Software RAID monitoring and management
   Loaded: loaded (/usr/lib/systemd/system/mdmonitor.service; enabled)
   Active: active (running) since Do 2015-04-23 10:14:57 CEST; 4h 5min ago
  Process: 821 ExecStart=/sbin/mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid (code=exited, status=0/SUCCESS)
 Main PID: 822 (mdadm)
   CGroup: /system.slice/mdmonitor.service
           └─822 /sbin/mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid

Apr 23 10:14:57 rh.thelounge.net systemd[1]: Started Software RAID monitoring and management.

Comment 3 Harald Hoyer 2015-04-24 09:13:26 UTC
I have no clue, what other unit, which is in the initramfs, requires mdmonitor.service.

Can you add "systemd.log_level=debug" to the kernel command line and attach the output of:

# journalctl -b -o short-monotonic -t systemd

Comment 4 Harald Reindl 2015-04-25 19:43:14 UTC
Created attachment 1018846 [details]
requested debug-log

requested debug-log

Comment 5 Harald Hoyer 2015-04-27 10:09:32 UTC
Here we go:

$ fgrep -r mdmonitor /usr/lib/udev/rules.d/
/usr/lib/udev/rules.d/63-md-raid-arrays.rules:ENV{MD_LEVEL}=="raid[1-9]*", ENV{SYSTEMD_WANTS}+="mdmonitor.service"

Comment 6 Harald Reindl 2015-04-27 10:11:51 UTC
[root@rh:~]$ fgrep -r mdmonitor /usr/lib/udev/rules.d//usr/lib/udev/rules.d/63-md-raid-arrays.rules:ENV{MD_LEVEL}=="raid[1-9]*", ENV{SYSTEMD_WANTS}+="mdmonitor.service"
grep: /usr/lib/udev/rules.d//usr/lib/udev/rules.d/63-md-raid-arrays.rules:ENV{MD_LEVEL}==raid[1-9]*,: No such file or directory
grep: ENV{SYSTEMD_WANTS}+=mdmonitor.service: No such file or directory

Comment 7 Harald Hoyer 2015-04-27 10:22:52 UTC
(In reply to Harald Reindl from comment #6)
> [root@rh:~]$ fgrep -r mdmonitor
> /usr/lib/udev/rules.d//usr/lib/udev/rules.d/63-md-raid-arrays.rules:
> ENV{MD_LEVEL}=="raid[1-9]*", ENV{SYSTEMD_WANTS}+="mdmonitor.service"
> grep:
> /usr/lib/udev/rules.d//usr/lib/udev/rules.d/63-md-raid-arrays.rules:
> ENV{MD_LEVEL}==raid[1-9]*,: No such file or directory
> grep: ENV{SYSTEMD_WANTS}+=mdmonitor.service: No such file or directory

:) This was not a request. This was displaying my findings and the source of the mdmonitor.service request.

If you want to reproduce, do:

$ grep -r mdmonitor /usr/lib/udev/rules.d/

Comment 8 Harald Reindl 2015-09-15 11:02:10 UTC
"This was displaying my findings and the source of the mdmonitor.service request" - fine but will that ever be fixed? still there in Fedora 22.......

Comment 9 Fedora End Of Life 2015-11-04 10:30:11 UTC
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Harald Reindl 2015-12-05 17:27:34 UTC
well, with F23 the messages got more on *every* machine using RAID1/RAID10

Dec  5 18:24:56 rh systemd: mdmonitor.service: Cannot add dependency job, ignoring: Unit mdmonitor.service failed to load: No such file or directory.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc3' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc1' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc2' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd1' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd2' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb1' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb2' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd3' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb3' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda1' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda2' failed with exit code 1.
Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda3' failed with exit code 1.

Comment 11 Harald Hoyer 2015-12-08 10:45:26 UTC
(In reply to Harald Reindl from comment #10)
> well, with F23 the messages got more on *every* machine using RAID1/RAID10
> 
> Dec  5 18:24:56 rh systemd: mdmonitor.service: Cannot add dependency job,
> ignoring: Unit mdmonitor.service failed to load: No such file or directory.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc3' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc1' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdc2' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd1' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd2' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb1' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb2' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdd3' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sdb3' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda1' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda2' failed
> with exit code 1.
> Dec  5 18:24:59 rh systemd-udevd: Process '/sbin/mdadm -I /dev/sda3' failed
> with exit code 1.

That sounds like a different bug.

Reassigning to get more comments from the mdadm folks. Should mdmonitor be included, and why does mdadm -I fail??

Comment 12 Harald Reindl 2015-12-08 10:59:19 UTC
well from the timestamps the "mdadm -I" looks like happening too in the initrd while my RAID devices are just fine on all involved machines

[root@rh:~]$ raid_status 
---------------------------------------------------------------------------------
/dev/md0:
        Version : 1.0
  Creation Time : Wed Jun  8 13:10:48 2011
     Raid Level : raid1
     Array Size : 511988 (499.99 MiB 524.28 MB)
  Used Dev Size : 511988 (499.99 MiB 524.28 MB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue Dec  8 11:26:47 2015
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

           Name : localhost.localdomain:0  (local to host localhost.localdomain)
           UUID : 1d691642:baed26df:1d197496:4fb00ff8
         Events : 661

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       17        3      active sync   /dev/sdb1
---------------------------------------------------------------------------------
/dev/md1:
        Version : 1.1
  Creation Time : Wed Jun  8 13:10:52 2011
     Raid Level : raid10
     Array Size : 30716928 (29.29 GiB 31.45 GB)
  Used Dev Size : 15358464 (14.65 GiB 15.73 GB)
   Raid Devices : 4
  Total Devices : 4                                                                                                             
    Persistence : Superblock is persistent                                                                                      
                                                                                                                                
  Intent Bitmap : Internal                                                                                                      
                                                                                                                                
    Update Time : Tue Dec  8 11:58:07 2015                                                                                      
          State : clean                                                                                                                                                                                            
 Active Devices : 4                                                                                                                                                                                                
Working Devices : 4                                                                                                                                                                                                
 Failed Devices : 0                                                                                                                                                                                                
  Spare Devices : 0                                                                                                                                                                                                
                                                                                                                                                                                                                   
         Layout : near=2                                                                                                                                                                                           
     Chunk Size : 512K                                                                                                                                                                                             

           Name : localhost.localdomain:1  (local to host localhost.localdomain)
           UUID : b7475879:c95d9a47:c5043c02:0c5ae720
         Events : 2713

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync set-A   /dev/sda2
       1       8       34        1      active sync set-B   /dev/sdc2
       2       8       50        2      active sync set-A   /dev/sdd2
       3       8       18        3      active sync set-B   /dev/sdb2
---------------------------------------------------------------------------------
/dev/md2:
        Version : 1.1
  Creation Time : Wed Jun  8 13:10:56 2011
     Raid Level : raid10
     Array Size : 3875222528 (3695.70 GiB 3968.23 GB)
  Used Dev Size : 1937611264 (1847.85 GiB 1984.11 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Dec  8 11:58:37 2015
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : localhost.localdomain:2  (local to host localhost.localdomain)
           UUID : ea253255:cb915401:f32794ad:ce0fe396
         Events : 583959

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync set-A   /dev/sda3
       1       8       35        1      active sync set-B   /dev/sdc3
       2       8       51        2      active sync set-A   /dev/sdd3
       3       8       19        3      active sync set-B   /dev/sdb3

Comment 13 Jan Kurik 2016-02-24 13:22:37 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase

Comment 14 Jes Sorensen 2016-03-16 14:02:07 UTC
Harald,

mdmonitor is required for BIOS RAID devices eg. IMSM and DDF devices.

Jes

Comment 15 Harald Hoyer 2016-03-17 14:57:38 UTC
(In reply to Jes Sorensen from comment #14)
> Harald,
> 
> mdmonitor is required for BIOS RAID devices eg. IMSM and DDF devices.
> 
> Jes

in case of a generic interface, there is no /etc/mdadm.conf, but mdmonitor.service has:

ConditionPathExists=/etc/mdadm.conf

How would we handle this case?

Comment 16 Harald Hoyer 2016-03-17 14:59:12 UTC
dracut already includes mdmon@.service btw

Comment 17 Harald Hoyer 2016-03-17 15:00:11 UTC
(In reply to Harald Hoyer from comment #16)
> dracut already includes mdmon@.service btw

can you clarify the difference between both?

Comment 18 Jes Sorensen 2016-03-17 15:06:21 UTC
Harald,

I don't understand your question here. /etc/mdadm.conf is optional and shouldn't
be a requirement.

Jes

Comment 19 Harald Hoyer 2016-03-17 15:24:34 UTC
(In reply to Jes Sorensen from comment #18)
> Harald,
> 
> I don't understand your question here. /etc/mdadm.conf is optional and
> shouldn't
> be a requirement.
> 
> Jes

well, if I would include mdmonitor.service in the initramfs and the initramfs has no /etc/mdadm.conf, then the service would not start, because of:
ConditionPathExists=/etc/mdadm.conf
not being met in the service unit file.

Comment 20 Harald Reindl 2016-03-17 15:28:23 UTC
but *why* has it ConditionPathExists=/etc/mdadm.conf at all?

Linux is supposed to auto-detect mdadm raids and so it's bad in general because it won't monitor connected software-raids until the config file exists (been there short ago on a CentOS7 where i installed the OS on a sd-card to have the whole disks for a linux raid10 and for some time missed that the raid is not monitored because the missing config)

Comment 21 Jes Sorensen 2016-03-17 15:34:40 UTC
(In reply to Harald Hoyer from comment #19)
> (In reply to Jes Sorensen from comment #18)
> > Harald,
> > 
> > I don't understand your question here. /etc/mdadm.conf is optional and
> > shouldn't
> > be a requirement.
> > 
> > Jes
> 
> well, if I would include mdmonitor.service in the initramfs and the
> initramfs has no /etc/mdadm.conf, then the service would not start, because
> of:
> ConditionPathExists=/etc/mdadm.conf
> not being met in the service unit file.

Harald,

I honestly do now know why it is there. Looking at my logs, it was Milan Broz
who added it. Hopefully he can explain the rationale for it.

Jes

Comment 22 Milan Broz 2016-03-17 15:54:52 UTC
It was initial switch to systemd, see bug #713573

IIRC we tried to fix provided patch (because it was completely untested) and nobody actually want to do that.

The condition for mdadm.conf existence was probably rewritten from sysvinit script, just remove or change it if needed.

Comment 23 Nigel Croxon 2017-04-27 15:24:52 UTC
This looks fixed in mdadm-3.4.

Comment 24 Nigel Croxon 2017-05-02 14:08:01 UTC
I'm looking to close this BZ. 

-Nigel