Bug 1296123

Summary: skyring binary is not service
Product: [Red Hat Storage] Red Hat Storage Console Reporter: Martin Kudlej <mkudlej>
Component: coreAssignee: Shubhendu Tripathi <shtripat>
core sub component: configuration QA Contact: Martin Bukatovic <mbukatov>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: unspecified CC: mbukatov, nthomas, sankarshan, shtripat
Version: 2Keywords: TestBlocker
Target Milestone: beta   
Target Release: 2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhscon-ceph-0.0.23-1.el7scon.x86_64, rhscon-core-0.0.24-1.el7scon.x86_64, rhscon-ui-0.0.39-1.el7scon.noarch Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-19 05:30:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1311521, 1326788, 1326854    

Description Martin Kudlej 2016-01-06 11:53:03 UTC
Description of problem:
Skyring binary has no Systemd service so it is not possible to work with it by command systemctl.

Version-Release number of selected component (if applicable):
rhscon-core-0.0.5-0.1.alpha1.el7.x86_64

How reproducible:
100%

Actual results:
Skyring binary is not service.

Expected results:
Skyring binary is service.

Comment 1 Kanagaraj 2016-02-03 08:54:11 UTC
service skyringd start
service skyringd status
service skyringd stop

Comment 2 Martin Bukatovic 2016-02-12 13:35:57 UTC
Using rhscon-core-0.0.8-2.el7.x86_64, I see that skyringd service is available:

~~~
# systemctl list-unit-files | grep skyring
skyringd.service                            static
# systemctl status skyringd
● skyringd.service - skyring server
   Loaded: loaded (/usr/lib/systemd/system/skyringd.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2016-02-12 13:51:03 CET; 6min ago
  Process: 31668 ExecStart=/etc/init.d/skyringd start (code=exited, status=0/SUCCESS)
 Main PID: 31673 (skyring)
   CGroup: /system.slice/skyringd.service
           ├─31673 /usr/bin/skyring
           └─31693 /var/lib/skyring/providers/bigfin {"config":{"host":"0.0.0.0","httpPort":8080,"supportedversions":[1]},"logging":{"logToStderr":false,"filename":"/var/log/skyring/skyring.log"...

Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Starting skyring server...
Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Started skyring server.
Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com skyringd[31668]: Starting skyring:

~~~

That said, I see a big problem with the way it has been implemented:

Systemd service file is just a wrapper. It doesn't start skyring process
directly, but it calls system v style init script which then starts the
skyring process, see:

~~~
# systemctl cat skyringd
# /usr/lib/systemd/system/skyringd.service
[Unit]
Description=skyring server

[Service]
Type=forking
KillMode=process
ExecStart=/etc/init.d/skyringd start
ExecStop=/etc/init.d/skyringd stop
# file /etc/init.d/skyringd
/etc/init.d/skyringd: POSIX shell script, ASCII text executable
# wc -l /etc/init.d/skyringd
105 /etc/init.d/skyringd
~~~

Such approach is not acceptable for a brand new service. Moreover no systemd
features could be configured/used this way.

Eg. instead of hacking handling of pid and lock files in a shell init script, 
you can specify it in a systemd service file.

This direclty leads to several problems, such as (note that the list below
doesn't cover all possible problems):

1) systemd is not avare of any skyring socket:

~~~
# systemctl list-sockets --all | grep skyring
#
~~~

note: compare this with sshd:

~~~
# systemctl list-sockets --all | grep ssh
[::]:22                         sshd.socket
~~~

2) systemd is not aware of skyring service dependencies:

~~~
# systemctl list-dependencies skyringd | grep -i mongo
#
~~~

note: I would expect to see mongob here 

3) When I ask journalctl for logs of skyringd service, I see just list of 
start and stop events. I would expect to see more details in the journal
logs. Eg. I started a create cluster task and still see nothing in the logs.

~~~
# journalctl -u skyringd
-- Logs begin at Tue 2016-02-09 12:56:17 CET, end at Fri 2016-02-12 14:03:43 CET. --
Feb 09 13:35:59 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Starting skyring server...
Feb 09 13:35:59 mbukatov-usm1-server.os1.phx2.redhat.com skyringd[26966]: Starting skyring:
Feb 09 13:35:59 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Started skyring server.
Feb 09 13:36:00 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Started skyring server.
Feb 12 13:50:50 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Stopping skyring server...
Feb 12 13:50:52 mbukatov-usm1-server.os1.phx2.redhat.com skyringd[31602]: Stopping skyring: [  OK  ]
Feb 12 13:50:52 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Stopped skyring server.
Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Starting skyring server...
Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com systemd[1]: Started skyring server.
Feb 12 13:51:03 mbukatov-usm1-server.os1.phx2.redhat.com skyringd[31668]: Starting skyring:
~~~

Comment 4 Martin Bukatovic 2016-02-12 14:15:39 UTC
Moreover: looking at /usr/bin/skyring-setup.sh script, it's obvious that some
of the stuff here should be moved into systemd service file (eg. list of services
required by skyring: salt-master, influxdb and mongod.

Comment 5 Martin Kudlej 2016-02-19 12:13:09 UTC
Also if there is proper systemd service file this will not happen:
2016-02-19T12:28:16.782+01:00 ERROR    ldapauth.go:366 AddUser] User admin already exists
2016-02-19T12:28:16.782+01:00 ERROR    auth.go:37 AddDefaultUser] Unable to create default User:user already exists
2016-02-19T12:28:16.785+01:00 INFO     skyring.go:500 InitializeApplication] Starting clusters syncing
2016-02-19T12:28:16.785+01:00 INFO     main.go:187 start] Starting the providers
2016-02-19T12:28:16.785+01:00 DEBUG    provider-conf.go:53 LoadProviderConfig] File Name: ceph.conf
2016-02-19T12:28:16.786+01:00 DEBUG    provider-conf.go:68 LoadProviderConfig] Collection:%!(EXTRA []conf.ProviderInfo=[{{ceph bigfin} []}])
2016-02-19T12:28:16.786+01:00 INFO     skyring.go:92 StartProviders] Config:%!(EXTRA []conf.ProviderInfo=[{{ceph bigfin} []}])
2016-02-19T12:28:16.786+01:00 DEBUG    skyring.go:95 StartProviders] Config:%!(EXTRA conf.ProviderInfo={{ceph bigfin} []})
2016-02-19T12:28:16.798+01:00 INFO     main.go:194 start] Starting event listener
2016-02-19T12:28:16.801+01:00 CRITICAL listener.go:134 StartListener] listen error:listen unix /var/run/.skyring-event: bind: address already in use

Comment 7 Martin Bukatovic 2016-03-18 15:14:54 UTC
Also note that with the current service file it's not possible to enable the
service. Which means that skyring service would not be running after reboot,
and a manual start is required.

~~~
# systemctl enable skyringd.service
The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's
   .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has
   a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer,
   D-Bus, udev, scripted systemctl call, ...).
# systemctl status skyringd.service
● skyringd.service - skyring server
   Loaded: loaded (/usr/lib/systemd/system/skyringd.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2016-03-18 16:06:09 CET; 4min 52s ago
 Main PID: 3610 (skyring)
   CGroup: /system.slice/skyringd.service
           ├─3610 /usr/bin/skyring
           └─3631 /var/lib/skyring/providers/bigfin {"config":{"host":"0.0.0.0","httpPort":8080,"supportedversions":[1]},"logging":{"logToStderr":false,"fi...
#
~~~

Comment 8 Martin Bukatovic 2016-08-08 18:45:51 UTC
Checking rhscon-core-0.0.41-1.el7scon.x86_64

Skyring now has a proper systemd service unit file:

* start, stop and restart is possible
* enable/disable of the service works
* both mongod and salt-master services are specified as dependencies
  (starting skyring would start them if they are not running)

Moreover, I'm able to check logs via journal using:

~~~
# journalctl -u skyring
~~~