Bug 1653357

Summary: ceilometer_agent_notification container is stuck at restarting when panko is disabled
Product: Red Hat OpenStack Reporter: Sasha Smolyak <ssmolyak>
Component: openstack-tripleo-heat-templatesAssignee: Vinay Kapalavai <vkapalav>
Status: CLOSED DUPLICATE QA Contact: Nataf Sharabi <nsharabi>
Severity: medium Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: jruzicka, jschluet, mburns, mrunge, pkilambi
Target Milestone: z1Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-10.2.1-0.20190103200639.1980cfd.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-26 16:14:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sasha Smolyak 2018-11-26 16:01:33 UTC
This bug is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=1633900

Description of problem:

We want to disable panko service since we don't use OpenStack with CloudForms.
We deployed overcloud according with following solutions.

  How to disable Panko service in RHOSP13.
  https://access.redhat.com/solutions/3527641

After that, we noticed ceilometer_agent_notification container didn't start correctly.

  # docker ps | grep ceilometer_agent_notification
  e9914c55235a        192.168.14.1:5000/rhosp13/openstack-ceilometer-notification:13.0-54     "kolla_start"            30 hours ago        Restarting (1) About a minute ago                       ceilometer_agent_notification

  # docker inspect ceilometer_agent_notification | grep Status
            "Status": "restarting",
                "Status": "starting",

  # docker logs  ceilometer_agent_notification
  + sudo -E kolla_set_configs
  INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
  INFO:__main__:Validating config file
  INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
  INFO:__main__:Copying service configuration files
  INFO:__main__:Deleting /etc/ceilometer/ceilometer.conf
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/ceilometer.conf to /etc/ceilometer/ceilometer.conf
  INFO:__main__:Deleting /etc/ceilometer/event_pipeline.yaml
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml
  INFO:__main__:Deleting /etc/ceilometer/polling.yaml.org
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/polling.yaml.org to /etc/ceilometer/polling.yaml.org
  INFO:__main__:Deleting /etc/ceilometer/polling.yaml
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/polling.yaml to /etc/ceilometer/polling.yaml
  ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src-panko/* file is not found

  # docker inspect ceilometer_agent_notification | grep panko
                "/var/lib/config-data/puppet-generated/panko/:/var/lib/kolla/config_files/src-panko:ro",
                "Source": "/var/lib/config-data/puppet-generated/panko",
                "Destination": "/var/lib/kolla/config_files/src-panko",

From the result of above docker commands, it seems that ceilometer_agent_notification container cannot start 
 if /var/lib/config-data/puppet-generated/panko/ is empty.

For testing purpose, we locate dummy file under /var/lib/config-data/puppet-generated/panko/, 
 the container successfully start.

  # touch  /var/lib/config-data/puppet-generated/panko/dummy
  # docker restart ceilometer_agent_notification
  # docker inspect ceilometer_agent_notification | grep Status
            "Status": "running",
                "Status": "healthy",
  # docker logs ceilometer_agent_notification | grep -3 dummy
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/polling.yaml.org to /etc/ceilometer/polling.yaml.org
  INFO:__main__:Deleting /etc/ceilometer/polling.yaml
  INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/polling.yaml to /etc/ceilometer/polling.yaml
  INFO:__main__:Copying /var/lib/kolla/config_files/src-panko/dummy to /dummy
  INFO:__main__:Writing out command to execute
  ++ cat /run_command
  + CMD='/usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log'


Version-Release number of selected component (if applicable):

 openstack-ceilometer-notification 13.0-54
 openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch

How reproducible:

Always

Steps to Reproduce:

Deploy overcloud with ceilometer without panko.
The procedure is described at following solutions.

  How to disable Panko service in RHOSP13.
  https://access.redhat.com/solutions/3527641

Actual results:

ceilometer_agent_notification is stuck at restarting

Expected results:

ceilometer_agent_notification starts successfully

Comment 2 Vinay Kapalavai 2018-12-10 15:44:57 UTC
https://review.openstack.org/#/c/620402/

Comment 5 Mikey Ariel 2019-02-20 12:44:19 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 6 Vinay Kapalavai 2019-02-26 16:14:24 UTC

*** This bug has been marked as a duplicate of bug 1679774 ***