Description of problem: Target service in the gluster container image is not enabled and as a result it is not started by default. [root@dhcp46-207 ~]# oc rsh glusterfs-4wpxm sh-4.2# systemctl status target ● target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor preset: disabled) Active: inactive (dead) Version-Release number of selected component (if applicable): cns-deploy-5.0.0-23.el7rhgs.x86_64 rhgs3/rhgs-server-rhel7:3.3.0-15 How reproducible: always Steps to Reproduce: 1. deploy cns with the above build Actual results: target service is not started. Expected results: target service should be started
h-4.2# cat /usr/lib/systemd/system/target.service [Unit] Description=Restore LIO kernel target configuration Requires=sys-kernel-config.mount After=sys-kernel-config.mount network.target local-fs.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/targetctl restore ExecStop=/usr/bin/targetctl clear SyslogIdentifier=target [Install] WantedBy=multi-user.target sh-4.2# cat /usr/lib/systemd/system/tcmu-runner.service [Unit] Description=LIO Userspace-passthrough daemon Requisite=glusterd.service BindsTo=gluster-block-target.service Before=gluster-block-target.service [Service] Type=dbus BusName=org.kernel.TCMUService1 KillMode=process ExecStart=/usr/bin/tcmu-runner --tcmu-log-dir=/var/log/gluster-block/ sh-4.2# sh-4.2# cat /usr/lib/systemd/system/gluster-blockd.service [Unit] Description=Gluster block storage utility BindsTo=tcmu-runner.service After=tcmu-runner.service [Service] Type=simple Environment="GB_GLFS_LRU_COUNT=5" Environment="GB_LOG_LEVEL=INFO" EnvironmentFile=-/etc/sysconfig/gluster-blockd ExecStart=/usr/sbin/gluster-blockd --glfs-lru-count $GB_GLFS_LRU_COUNT --log-level $GB_LOG_LEVEL $GB_EXTRA_ARGS KillMode=process [Install] WantedBy=multi-user.target sh-4.2# sh-4.2# cat /usr/lib/systemd/system/gluster-block-target.service # This file is part of gluster-block. # # This is a wrapper unit over target.service which ensures that # tcmu-runner.service is running which intern has a check to see if # glusterd.service is active # .include /usr/lib/systemd/system/target.service [Unit] Requisite=tcmu-runner.service Conflicts=target.service It looks to me this service dependency chain is unwantedly complex or broken. Prasanna any changes in latest gluster-block or dependent packages in this area? Isnt it should be simple as "gluster-blockd -> tcmu-runner and gluster-block-target -> target" ? Karthick, as a workaround, once you have the pod running, get into the pod and stop #target,tcmu-runner and gluster-blockd, then start these services.
This is due to the change introduced by gluster-block team in latest builds compared to previous builds. --Previous version -- sh-4.2# cat /usr/lib/systemd/system/tcmu-runner.service [Unit] Description=LIO Userspace-passthrough daemon [Service] Type=dbus BusName=org.kernel.TCMUService1 KillMode=process ExecStart=/usr/bin/tcmu-runner --tcmu-log-dir=/var/log/gluster-block/ sh-4.2# cat /usr/lib/systemd/system/gluster-blockd.service [Unit] Description=Gluster block storage utility Requires=target.service tcmu-runner.service After=target.service tcmu-runner.service [Service] Type=simple Environment="GB_GLFS_LRU_COUNT=5" Environment="GB_LOG_LEVEL=INFO" EnvironmentFile=-/etc/sysconfig/gluster-blockd ExecStart=/usr/sbin/gluster-blockd --glfs-lru-count $GB_GLFS_LRU_COUNT --log-level $GB_LOG_LEVEL $GB_EXTRA_ARGS KillMode=process [Install] WantedBy=multi-user.target --/snip-- Has anyone tested this change at all?
@Humble, The workaround doesn't seem to help. Infact, I had already tried this before filing the bug. sh-4.2# systemctl stop gluster-blockd sh-4.2# systemctl stop target sh-4.2# systemctl stop tcmu-runner sh-4.2# sh-4.2# sh-4.2# systemctl start target sh-4.2# systemctl start tcmu-runner sh-4.2# systemctl start gluster-blockd sh-4.2# systemctl status target ● target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor preset: disabled) Active: inactive (dead) since Sat 2017-08-26 00:53:23 UTC; 13s ago Process: 20544 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS) Process: 20533 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 20533 (code=exited, status=0/SUCCESS) Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Starting Restore... Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com target[20533]: No saved conf... Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Started Restore ... Aug 26 00:53:23 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Stopping Restore... Aug 26 00:53:23 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Stopped Restore ... Hint: Some lines were ellipsized, use -l to show in full.
(In reply to krishnaram Karthick from comment #4) > @Humble, > > The workaround doesn't seem to help. Infact, I had already tried this before > filing the bug. > > sh-4.2# systemctl stop gluster-blockd > sh-4.2# systemctl stop target > sh-4.2# systemctl stop tcmu-runner > sh-4.2# > sh-4.2# > sh-4.2# systemctl start target > sh-4.2# systemctl start tcmu-runner > sh-4.2# systemctl start gluster-blockd > sh-4.2# systemctl status target > ● target.service - Restore LIO kernel target configuration > Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor > preset: disabled) > Active: inactive (dead) since Sat 2017-08-26 00:53:23 UTC; 13s ago > Process: 20544 ExecStop=/usr/bin/targetctl clear (code=exited, > status=0/SUCCESS) > Process: 20533 ExecStart=/usr/bin/targetctl restore (code=exited, > status=0/SUCCESS) > Main PID: 20533 (code=exited, status=0/SUCCESS) > > Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Starting > Restore... > Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com target[20533]: No saved > conf... > Aug 26 00:53:17 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Started Restore > ... > Aug 26 00:53:23 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Stopping > Restore... > Aug 26 00:53:23 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Stopped Restore > ... > Hint: Some lines were ellipsized, use -l to show in full. Well, all the required service are running in your setup, see ● target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor preset: disabled) Active: active (exited) since Sat 2017-08-26 07:09:32 UTC; 1s ago Process: 32025 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 32025 (code=exited, status=0/SUCCESS) Aug 26 07:09:32 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Starting Restore... Aug 26 07:09:32 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Started Restore ... ● tcmu-runner.service - LIO Userspace-passthrough daemon Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; static; vendor preset: disabled) Active: active (running) since Sat 2017-08-26 07:08:47 UTC; 47s ago Main PID: 31950 (tcmu-runner) CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977bd20a_899a_11e7_8738_005056a56b97.slice/docker-7cd2f8ef1428381327748593b012724dce949cc908c77b58a84403da3f29810b.scope/system.slice/tcmu-runner.service └─31950 /usr/bin/tcmu-runner --tcmu-log-dir=/var/log/gluster-block... Aug 26 07:08:47 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Starting LIO Use... Aug 26 07:08:47 dhcp47-23.lab.eng.blr.redhat.com tcmu-runner[31950]: tcmu-runner : load_o... Aug 26 07:08:47 dhcp47-23.lab.eng.blr.redhat.com tcmu-runner[31950]: 2017-08-... Aug 26 07:08:47 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Started LIO User... ● gluster-blockd.service - Gluster block storage utility Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2017-08-26 07:09:21 UTC; 13s ago Main PID: 32006 (gluster-blockd) CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977bd20a_899a_11e7_8738_005056a56b97.slice/docker-7cd2f8ef1428381327748593b012724dce949cc908c77b58a84403da3f29810b.scope/system.slice/gluster-blockd.service └─32006 /usr/sbin/gluster-blockd --glfs-lru-count 5 --log-level IN... Aug 26 07:09:21 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Started Gluster ... Aug 26 07:09:21 dhcp47-23.lab.eng.blr.redhat.com systemd[1]: Starting Gluster... Hint: Some lines were ellipsized, use -l to show in full. ======================= Please dont restart the containers. But you can carry on with further testing.
Sorry, I had tried restarting the container before I saw your comment as I was wondering how all the services came up. I have a couple of questions. 1) How did the service start (any additional steps to be performed?) 2) Why not restart containers? If that's coz of some reason, I can work on testing the +ve cases alone, but please get the services up once again :-) [root@dhcp47-57 ~]# oc rsh glusterfs-cm7t0 sh-4.2# sh-4.2# systemctl status gluster-blockd ● gluster-blockd.service - Gluster block storage utility Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2017-08-26 07:40:34 UTC; 5min ago Main PID: 541 (gluster-blockd) CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb64db776_8a2a_11e7_8738_005056a56b97.slice/docker-e9b730efb1300a4501befc61d85f982cd2478bc56d74a85f165b439b68efd744.scope/system.slice/gluster-blockd.service └─541 /usr/sbin/gluster-blockd --glfs-lru-count 5 --log-level INFO... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Started Gluster ... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Starting Gluster... Hint: Some lines were ellipsized, use -l to show in full. sh-4.2# systemctl status tcmu-runner ● tcmu-runner.service - LIO Userspace-passthrough daemon Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; static; vendor preset: disabled) Active: active (running) since Sat 2017-08-26 07:40:34 UTC; 5min ago Main PID: 534 (tcmu-runner) CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb64db776_8a2a_11e7_8738_005056a56b97.slice/docker-e9b730efb1300a4501befc61d85f982cd2478bc56d74a85f165b439b68efd744.scope/system.slice/tcmu-runner.service └─534 /usr/bin/tcmu-runner --tcmu-log-dir=/var/log/gluster-block/ Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Starting LIO Use... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com tcmu-runner[534]: tcmu-runner : load_our... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com tcmu-runner[534]: 2017-08-26... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Started LIO User... Hint: Some lines were ellipsized, use -l to show in full. sh-4.2# systemctl status target ● target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor preset: disabled) Active: inactive (dead) since Sat 2017-08-26 07:40:34 UTC; 5min ago Process: 535 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS) Process: 479 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 479 (code=exited, status=0/SUCCESS) Aug 26 07:40:20 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Starting Restore... Aug 26 07:40:20 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Started Restore ... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Stopping Restore... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Stopped Restore ... Hint: Some lines were ellipsized, use -l to show in full. sh-4.2# systemctl start target sh-4.2# systemctl status gluster-blockd ● gluster-blockd.service - Gluster block storage utility Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; enabled; vendor preset: disabled) Active: inactive (dead) since Sat 2017-08-26 07:49:45 UTC; 20s ago Process: 541 ExecStart=/usr/sbin/gluster-blockd --glfs-lru-count $GB_GLFS_LRU_COUNT --log-level $GB_LOG_LEVEL $GB_EXTRA_ARGS (code=killed, signal=TERM) Main PID: 541 (code=killed, signal=TERM) Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Started Gluster ... Aug 26 07:40:34 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Starting Gluster... Aug 26 07:49:45 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Stopping Gluster... Aug 26 07:49:45 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Stopped Gluster ... Hint: Some lines were ellipsized, use -l to show in full. sh-4.2# systemctl status target ● target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; disabled; vendor preset: disabled) Active: active (exited) since Sat 2017-08-26 07:49:45 UTC; 30s ago Process: 535 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS) Process: 840 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 840 (code=exited, status=0/SUCCESS) Aug 26 07:49:45 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Starting Restore... Aug 26 07:49:45 dhcp47-69.lab.eng.blr.redhat.com systemd[1]: Started Restore ... Hint: Some lines were ellipsized, use -l to show in full.
https://bugzilla.redhat.com/show_bug.cgi?id=1485412#c9 . Also I dont know why we need 2 bugzillas for the same bug.
Based on https://bugzilla.redhat.com/show_bug.cgi?id=1485101#c8, target service is no more needed for gluster-block to run, instead gluster-block-target is the service we should be looking at. gluster-block-target service is up and running in the latest RHGS container. So we are good here. closing the bug.