Bug 1267031 - dbmail services daemons fail to start (systemd) on CentOS 7 (v7.1)
dbmail services daemons fail to start (systemd) on CentOS 7 (v7.1)
Status: NEW
Product: Fedora EPEL
Classification: Fedora
Component: dbmail (Show other bugs)
epel7
All Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Bernard Johnson
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-28 17:02 EDT by Mauro Mozzarelli
Modified: 2016-01-08 17:28 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mauro Mozzarelli 2015-09-28 17:02:59 EDT
Description of problem:
dbmail-imapd, dbmail-lmtpd, dbmail-pop3d, dbmail-timsieved cannot be started automatically (systemd) at boot time or manually using systemctl start command


Version-Release number of selected component (if applicable):
dbmail.x86_64                           3.2.3-1.el7                    @epel    
dbmail-auth-ldap.x86_64                 3.2.3-1.el7                    @epel


How reproducible:
Configure dbmail with a database (in my case mariadb coming with CentOS 7) and start the daemon manually (from the command line) to test. Once everything is properly configured kill the test daemons and enable the services using

systemctl enable dbmail-imapd
systemctl enable dbmail-lmtpd
systemctl enable dbmail-pop3d
systemctl enable dbmail-timsieved

and reboot. The services will not start and will not leave any trace in the logs not even when adding -vv to the command line (by editing the respective /usr/lib/systemd/system/dbmail-XXXXX.service).

Steps to Reproduce:
1. configure dbmail
2. enable the respective services
3. boot or start the service using systemctl start

Actual results:
The daemons will not start and will not leave any message in /var/log/dbmail.log or dbmail.err

Expected results:
daemons to start as in CentOS 6

Additional info:
The default /usr/lib/systemd/system/dbmail-XXXXX.service files do not include mariadb.service in "After", however this isn't really the issue because after having added mariadb.service the daemons still do not start.

I discovered that if I issue the following commands manually in sequence:

# systemctl daemon-reload
# systemctl start dbmail-XXXXX   #replace XXXXX withimapd|lmtpd|pop3d|timsieved

at this point one service will start. If I want to start the next service I have to issue "systemctl daemon-reload" again, and again before each subsequent service.

As you may understand this is unacceptable and it is most definately a bug, probably lying with "systemd", although dbmail rpm packages and startup files should be amended to cater for mariadb.service too since it is the default DB on CentOS 7.
Comment 1 Mauro Mozzarelli 2015-09-28 21:02:28 EDT
Please find below a working systemd file for dbmail-imapd. The same can be edited and applied to pop3d, lmtpd and timsieved.

/usr/lib/systemd/system/dbmail-imapd.service:

[Unit]
Description=DBMail IMAP Server
After=syslog.target network.target mariadb.service postgresql.service

[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/dbmail-imapd
PIDFile=/var/run/dbmail/dbmail-imapd.pid
ExecStartPre=/usr/bin/install -d -m 0777 -o nobody -g nobody /var/run/dbmail
ExecStart=/usr/sbin/dbmail-imapd -p /var/run/dbmail/dbmail-imapd.pid
ExecStop=/usr/bin/kill -TERM $MAINPID
ExecReload=/usr/bin/kill -HUP $MAINPID
RemainAfterExit=yes
SuccessExitStatus=1

[Install]
WantedBy=multi-user.target
Comment 2 Jim Lieb 2016-01-08 17:28:36 EST
I have tracked this down further because I have the same problem on F23. I have also looked at the solution in comment #1 and it too has the "daemon-reload" problem. I don't think systemd is at fault here because the dbmail code does some strange things on its way to spawning the working daemon proc. Other legacy daemons do work, probably because they let the spawned child do most of the initialization.

My F20 based service had the same problem of failing to start although it would work sometimes rather than the f23 behavior of not at all...

What works is to move from "forked" to "simple" daemon.  Fortunately, dbmail supports this. Here is my override file for imapd:

# cat /etc/systemd/system/dbmail-imapd.service.d/override.conf 
[Unit]
After=nss-lookup.target

[Service]
Type=simple
ExecStart=
ExecStart=/usr/sbin/dbmail-imapd -D

As with comment #1, apply this to all the other daemons.

By going to "simple" we let systemd manage the cgroup itself and eliminate the pid file (always was a) hack. The '-D' flag bypasses the daemonizing bit, saving a fork+exit in startup. However, while fixing the "daemon-reload" problem, the daemons still did not start at boot time even though I could reliably start them later with systemctl.

The second item is the "After". My dbmail instance is isolated in a VM and the database (postgresql) is on a bare-metal host. Without waiting for "nss-lookup.target", each of the daemons fail because they too eagerly and silently bail during initialization because they are racing not only the basic network but the nss services etc. because (in my case) the route table has not settled down. This target waits on DNS connectivity.

BTW, I found this by cranking up the file debugging to "511" but even that was vague. I ended up using strace which showed that the DB connect was failing with an ENOROUTE error when trying to connect to my DNS indicating that network.target is too weak a dependency. Note that stuffing the server address into /etc/host just postpones until the DB socket is connected...

I recommend that the package service files change to use "simple" and "-D". I haven't straced the "forked" daemons but I suspect that dbmail is quietly exiting due to unhappy initialization code in the parent that really should be done by the spawned child.

I also suggest changing the "After" so that partitioned services like mine would work. network.target is too weak and nss-lookup.target would work just fine for local (UNIX socket) database connections.

Note You need to log in before you can comment on or make changes to this bug.