Bug 1601302
| Summary: | [UPGRADE][13] Error "Failed to get D-Bus connection: Operation not permitted" | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Petersingh Anburaj <panburaj> | ||||||||||
| Component: | openstack-tripleo | Assignee: | Jose Luis Franco <jfrancoa> | ||||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | medium | ||||||||||||
| Version: | 13.0 (Queens) | CC: | ccamacho, jfrancoa, lbezdick, mbayer, mburns, michele, mweetman, panburaj, pkilambi | ||||||||||
| Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||||||||
| Target Release: | 13.0 (Queens) | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2019-05-07 12:37:15 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
This seems like a gnocchi configuration error, and the "failures list" command doesn't provide information enough. Petersingh, could you attach all gnocchi-related logs from the overcloud controller, please (/var/log/containers/gnocchi)? Thanks Created attachment 1470587 [details]
Gnocchi logs files
Created attachment 1470588 [details]
Gnocchi conf files
I tested this is my test environment(3 Controller and 2 Compute). I had to re-provision the system for another use-case, so current I don't have that environment. Yesterday, I deployed this is a new environment(Only 1 Controller and 1 Compute). I was again facing the same issue while performing the "upgrade converge" step. PFA the gnocchi logs and conf files. Hello Pradeep, I can see in the gnocchi logs a lot of traceback logs due to missing connectivity: gnocchi-statsd.log http://pastebin.test.redhat.com/624312 app.log: http://pastebin.test.redhat.com/624313 Could you, or anybody from the metmon team have a look at it? Thanks. Created attachment 1508462 [details]
mweetman: output of "openstack stack failures list overcloud --long"
Hey Morgan, Checking at your logs, it seems that the failure you see is due to ovn: Error: Agent 'ocf:ovn:ovndb-servers' is not installed or does not provide valid metadata: Metadata query for ocf:ovn:ovndb-servers failed: Input/output error, use --force to override", And it seems there is an already opened BZ for it https://bugzilla.redhat.com/show_bug.cgi?id=1660652. Missing logs from the failure operation is hard to debug this issue. If no logs can be provided, is it ok for you if we close this BZ and re-open it in case you face it again? Currently, I don't have a system to reproduce it. You can go ahead a close this Bug. I will re-open it if I face this issue again. Thanks Jose.. I've commented in the bz you referenced From the comments in this and linked bug closing as duplicate, feel free to reopen. *** This bug has been marked as a duplicate of bug 1660652 *** |
Created attachment 1459042 [details] Output of "openstack stack failures list --long overcloud" Description of problem: RHOSP Upgrade from 12 to 13 failed with below error. ~~~ overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: fdedb406-0700-49a1-a80d-60f8b5fb1b80 status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | .. .. "Notice: /Stage[main]/Gnocchi::Wsgi::Apache/Openstacklib::Wsgi::Apache[gnocchi_wsgi]/Apache::Vhost[gnocchi_wsgi]/Concat::Fragment[gnocchi_wsgi-file_footer]/Concat_fragment[gnocchi_wsgi-file_footer]: De pendency File[gnocchi_wsgi] has failures: true", "Notice: /Stage[main]/Gnocchi::Wsgi::Apache/Openstacklib::Wsgi::Apache[gnocchi_wsgi]/Apache::Vhost[gnocchi_wsgi]/Apache::Listen[172.17.1.20:8041]/Concat::Fragment[Listen 172.17.1.20:8041]/Concat_fragme nt[Listen 172.17.1.20:8041]: Dependency File[gnocchi_wsgi] has failures: true", "Notice: Applied catalog in 2.35 seconds", "Changes:", " Total: 110", "Events:", " Failure: 1", " Success: 110", " Total: 111", "Resources:", " Failed: 1", " Changed: 110", " Out of sync: 111", " Total: 250", " Skipped: 51", "Time:", " Anchor: 0.00", " Concat fragment: 0.01", " Concat file: 0.01", " Augeas: 0.06", " Gnocchi config: 0.69", " File: 0.77", " Total: 10.73", " Last run: 1531573432", " Config retrieval: 9.18", " Resources: 0.00", "Version:", " Config: 1531573420", " Puppet: 4.8.2", "", "2018-07-14 13:03:53,506 ERROR: 444300 -- + mkdir -p /etc/puppet", "+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet", "+ rm -Rf /etc/puppet/ssl", "+ echo '{\"step\": 6}'", "+ TAGS=", "+ '[' -n file,file_line,concat,augeas,cron,gnocchi_api_paste_ini,gnocchi_config,gnocchi_config,gnocchi_config ']'", "+ TAGS='--tags file,file_line,concat,augeas,cron,gnocchi_api_paste_ini,gnocchi_config,gnocchi_config,gnocchi_config'", "+ origin_of_time=/var/lib/config-data/gnocchi.origin_of_time", "+ touch /var/lib/config-data/gnocchi.origin_of_time", "+ sync", "+ set +e", "+ FACTER_hostname=controller-1", "+ FACTER_uuid=docker", "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,gnocchi_api_paste_ini,gnocchi_config,gnocchi_config,gnocchi_config /etc/config.pp", "Failed to get D-Bus connection: Operation not permitted", "Warning: Undefined variable 'deploy_config_name'; ", " (file & line not available)", "Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/gnocchi/manifests/db.pp\", 42]:[\"/etc/puppet/modules/gnocchi/manifests/init.pp\", 62]", .. .. "2018-07-14 13:06:48,887 INFO: 444299 -- Removing container: docker-puppet-heat_api_cfn", "2018-07-14 13:06:48,970 INFO: 444299 -- Finished processing puppet configs for heat_api_cfn", "2018-07-14 13:06:48,971 ERROR: 444297 -- ERROR configuring gnocchi" ] } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/7839d0d2-1f01-4187-98ce-0524b634362b_playbook.retry ~~~