Bug 1507863
| Summary: | OSP9 -> OSP10: undercloud upgrade fails: mError: /Stage[main]/Ironic::Db::Sync/Exec[ironic-dbsync]/returns: change from notrun to 0 failed: ironic-dbsync --config-file /etc/ironic/ironic.conf returned 1 instead of one of [0] | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||
| Component: | instack-undercloud | Assignee: | Bob Fournier <bfournie> | ||||
| Status: | CLOSED ERRATA | QA Contact: | mlammon | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 10.0 (Newton) | CC: | bfournie, dbecker, jjoyce, jschluet, mburns, mcornea, morazi, rhel-osp-director-maint, sclewis, slinaber, tvignaud, yprokule | ||||
| Target Milestone: | z6 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 10.0 (Newton) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | instack-undercloud-5.3.2-2.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1510063 (view as bug list) | Environment: | |||||
| Last Closed: | 2017-11-15 13:47:19 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1510063 | ||||||
| Attachments: |
|
||||||
|
Description
Marius Cornea
2017-10-31 10:50:21 UTC
Created attachment 1345826 [details]
undercloud upgrade log
Since this issue is related to Ironic I'm assigning this ticket to DFG:HardProv. Thanks Marius. On first glance - as you've seen too - it looks like a permission issue trying to open /var/log/ironic/ironic-dbsync.log by ironic-dbsync. I see that the permissions look OK when you've checked. [stack@undercloud-0 ~]$ sudo ls -l /var/log/ironic/ironic-dbsync.log -rw-r--r--. 1 root root 0 Oct 31 05:45 /var/log/ironic/ironic-dbsync.log But any idea what they were set to in OSP-9? I'm wondering if they were not correct when ironic-dbsync ran. So it looks like this change that makes /var/log/ironic no longer world readable may have landed in this OSP-10z release - https://review.rdoproject.org/r/#/c/9683/ (I'll confirm that). I wonder if this has any affect to the problem of not being able to open /var/log/ironic/ironic-dbsync.log. Checking the permissions during OSP9 deployment: [root@undercloud-0 ~]# ls -l /var/log/ironic/ total 540 -rw-r--r--. 1 ironic ironic 275176 Oct 31 09:41 ironic-api.log -rw-r--r--. 1 ironic ironic 26328 Oct 31 09:39 ironic-conductor.log -rw-r--r--. 1 root root 0 Oct 31 09:18 ironic-dbsync.log Thanks Marius. So yeah, the dir /var/log/ironic was world readable in OSP9 and made not world readable when going to OSP-10. That appears to be causing a problem with permissions when ironic-dbsync attempts to open /var/log/ironic/ironic-dbsync.log. Still looking... Marius - one more thing, can you add the /etc/ironic/ironic.conf you are using to this bug? Thanks. And yes, the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1431468 which made /var/log/ironic not world readable is also in this puddle. Note its actually this BZ that is also fixed in the puddle - https://bugzilla.redhat.com/show_bug.cgi?id=1431479 (and may be potentially be the source of the problem). Marius, can you try one thing - delete /var/log/ironic-dbsync.log prior to the upgrade, then upgrade? Since ironic-dbsync.log is owned by root in OSP-9 I wonder if it can't be accessed by the "ironic" user in OSP-10. (In reply to Bob Fournier from comment #10) > Marius, can you try one thing - delete /var/log/ironic-dbsync.log prior to > the > upgrade, then upgrade? Since ironic-dbsync.log is owned by root in OSP-9 I > wonder if it can't be accessed by the "ironic" user in OSP-10. Yes, deleting /var/log/ironic/ironic-dbsync.log before running the undercloud upgrade allows it to move forward. Thanks Marius. So it looks like an issue with the file owner, not the directory change that was made by https://bugzilla.redhat.com/show_bug.cgi?id=1431479. In OSP-9, the file is owned by root, on the upgrade ironic-dbsync runs as ironic user and fails. -rw-r--r--. 1 root root 0 Oct 31 09:18 ironic-dbsync.log when its created in osp-10 its owned by ironic -rw-r--r--. 1 ironic ironic 0 Nov 1 07:08 ironic-dbsync.log I think this was caused by the following commit: https://github.com/openstack/puppet-ironic/commit/bf00e679016b60b421a9af353dd09dcaef514926#diff-5106a5beab0a246889a8cf98d3d992d0. It does not take upgrades into account. I believe it has to be fixed in puppet-ironic, but I'd double-check with someone from the puppet team. So it looks like this file ownership problem has been fixed in OSP-12 and OSP-11 -https://review.openstack.org/#/c/459871/2/instack_undercloud/undercloud.py but the fix hasn't been backported to OSP-10. I will add a backport to OSP-10. Backport upstream - https://review.openstack.org/#/c/517356/ *** Bug 1509969 has been marked as a duplicate of this bug. *** The patch is not merged yet, moving back to ON_DEV Correcting the project where the fix belongs. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3231 |