Bug 1315442
Summary: | rhel-osp-director: Upgrade undercloud 7.3->8.0. First run of "openstack undercloud install" fails, re-running completes fine. | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | ||||
Component: | instack-undercloud | Assignee: | Marios Andreou <mandreou> | ||||
Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 8.0 (Liberty) | CC: | akaris, astupnik, brad, dbecker, dyocum, ebarrera, jcoufal, jslagle, lbezdick, mandreou, mburns, mcornea, morazi, ohochman, rhel-osp-director-maint, tvignaud | ||||
Target Milestone: | ga | ||||||
Target Release: | 8.0 (Liberty) | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | instack-undercloud-2.2.7-4.el7ost | Doc Type: | Bug Fix | ||||
Doc Text: |
Cause: There is a problem with the restart of the systemd-journald service after the undercloud packages have been updated, as part of the undercloud upgrade.
Consequence: This causes the "openstack undercloud upgrade" to exit with an error such as "2016-03-07 09:55:52,925 INFO: ERROR: 2016-03-07 09:55:52,924 -- Hook FAILED.". Re-running "openstack undercloud upgrade" at this point would then complete without error.
Fix: Ensure that the systemd-journald service restart completes successfully, like we do at https://review.openstack.org/#/c/300051/.
Result: This allows the "openstack undercloud upgrade" to complete successfully on first run.
|
Story Points: | --- | ||||
Clone Of: | Environment: | ||||||
Last Closed: | 2016-04-15 14:31:11 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Alexander Chuzhoy
2016-03-07 18:19:11 UTC
Created attachment 1133888 [details]
install-undercloud.log
I am unable to reproduce this. Please retest on latest build. FailedQA. Reproduced against the latest poodle. I reproduced the journal issue on memory constrained system. Enabling swap on undercloud node helps. Just reproduced on a system with enough RAM and swap available, this does not seem to be an issue with memory constraints: [root@undercloud site-packages]# free -m total used free shared buff/cache available Mem: 15887 3518 3228 759 9139 11251 Swap: 32767 0 32767 reproduced with : ---------------- instack-0.0.8-2.el7ost.noarch instack-undercloud-2.2.7-1.el7ost.noarch puppet-3.6.2-2.el7.noarch openstack-puppet-modules-7.0.17-1.el7ost.noarch openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-1.el7ost.noarch python-tripleoclient-0.3.4-1.el7ost.noarch openstack-tripleo-image-elements-0.9.9-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-1.el7ost.noarch openstack-tripleo-0.0.7-1.el7ost.noarch openstack-tripleo-common-0.3.1-1.el7ost.noarch openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch Reproduced with "openstack undercloud upgrade". am trying to understand more about this today. It fails trying to run this file fyi https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/pre-install.d/01-persistent-journal (from the trace above), I'm also reproducing this with a setup today, trying the openstack undercloud upgrade When I first deployed an overcloud and then upgraded my undercloud it reproduced as above. When I just upgraded a fresh new undercloud it completed OK. This is from the journal when the error just happened again: Mar 31 09:30:57 instack.localdomain systemd[1]: Stopping Flush Journal to Persistent Storage... Mar 31 09:30:57 instack.localdomain systemd-journal[393]: Journal stopped Mar 31 09:30:57 instack.localdomain systemd-journal[23175]: Permanent journal is using 8.0M (max allowed 2.9G, trying to leave 4.0G free of 14.5G available → current limit 2.9G). Mar 31 09:30:57 instack.localdomain systemd-journal[23175]: Permanent journal is using 8.0M (max allowed 2.9G, trying to leave 4.0G free of 14.5G available → current limit 2.9G). Mar 31 09:31:07 instack.localdomain systemd-journal[23252]: Permanent journal is using 168.0M (max allowed 2.9G, trying to leave 4.0G free of 14.3G available → current limit 2.9G). Mar 31 09:31:07 instack.localdomain systemd-journald[393]: Received SIGTERM from PID 1 (systemd). Mar 31 09:31:07 instack.localdomain systemd-journald[23175]: Failed to create new runtime journal: No such file or directory Mar 31 09:31:07 instack.localdomain systemd-journald[23175]: Assertion 'f' failed at src/journal/journal-file.c:132, function journal_file_close(). Aborting. Mar 31 09:31:07 instack.localdomain systemd-journal[23252]: Journal started Mar 31 09:31:02 instack.localdomain sudo[23221]: stack : TTY=pts/2 ; PWD=/home/stack ; USER=root ; COMMAND=/bin/journalctl -fn 100 Mar 31 09:31:07 instack.localdomain polkitd[663]: Unregistered Authentication Agent for unix-process:23169:362368 (system bus name :1.37, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus) Mar 31 09:31:07 instack.localdomain systemd[1]: Starting Flush Journal to Persistent Storage... Mar 31 09:31:07 instack.localdomain systemd[1]: Started Flush Journal to Persistent Storage. There is an assertion fail above ^^^ and since we have -e at https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/pre-install.d/01-persistent-journal the undercloud install fails. Now the -e is normally a good thing, we want to make sure nothing fails here. For now could we set +e for that restart, and then have a check to make sure journalctl is indeed running? have a workaround for this will be posting shortly (filed upstream bug at https://bugs.launchpad.net/tripleo/+bug/1564471 ) I just tested the workaround at https://review.openstack.org/300051 and it seems to work for me. I'd appreciate more testing please. For now you can apply manually to your env like: sudo su pushd /usr/share/instack-undercloud/undercloud-install/pre-install.d/ mv 01-persistent-journal 01-persistent-journal.ORIG curl -o 01-persistent-journal "https://review.openstack.org/gitweb?p=openstack/instack-undercloud.git;a=blob_plain;f=elements/undercloud-install/pre-install.d/01-persistent-journal;h=237308a0b002c45fcd7d638d1f1e8b21be4bf8b9;hb=a583594bb967767888f7baf19b062875ecc71ad3" chmod 755 01-persistent-journal popd exit thanks! Verified: Environment: instack-undercloud-2.2.7-4.el7ost.noarch The issue doesn't reproduce with the last puddle. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0637.html *** Bug 1326809 has been marked as a duplicate of this bug. *** *** Bug 1395666 has been marked as a duplicate of this bug. *** |