Bug 1385852 - rhel-osp-director: Minor update fails right away with "'unicode' object does not support item assignment"
Summary: rhel-osp-director: Minor update fails right away with "'unicode' object doe...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Dougal Matthews
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-17 19:41 UTC by Alexander Chuzhoy
Modified: 2016-12-29 16:53 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-common-5.3.0-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:21:39 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 382738 None None None 2016-10-19 17:08:38 UTC
OpenStack gerrit 385471 None None None 2016-10-19 17:08:05 UTC
Launchpad 1626976 None None None 2016-10-19 17:07:43 UTC

Description Alexander Chuzhoy 2016-10-17 19:41:51 UTC
rhel-osp-director:   Minor update fails right away with "'unicode' object does not support item assignment"


Environment:
instack-undercloud-5.0.0-0.20161007201832.f044a47.el7ost.noarch
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.2.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log

2. Update the undercloud.
3. Attempt to update the overcloud with:
yes ""| openstack overcloud update stack  -i overcloud

Result:
starting package update on stack overcloud
'unicode' object does not support item assignment


The update doesn't even start.

Comment 1 Ryan Hallisey 2016-10-17 21:20:04 UTC
I'm not seeing this upstream, but it looks like it came from this patch: https://github.com/openstack/python-tripleoclient/commit/c7a1b84e713afd59630261d6f3992a4f798504a1

Comment 2 James Slagle 2016-10-18 19:43:46 UTC
(In reply to Ryan Hallisey from comment #1)
> I'm not seeing this upstream, but it looks like it came from this patch:
> https://github.com/openstack/python-tripleoclient/commit/
> c7a1b84e713afd59630261d6f3992a4f798504a1

why is that the case?

Comment 3 Lukas Bezdicka 2016-10-18 20:02:13 UTC
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates AttributeError: 'module' object has no attribute 'OVERCLOUD_J2_EXCLUDES'
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates     self.container, constants.OVERCLOUD_J2_EXCLUDES)[1]
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py", line 109, in _process_custo
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates     self._process_custom_roles()
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py", line 184, in run
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates Traceback (most recent call last):
Oct 18 17:47:37 undercloud-0.redhat.local mistral-server[5275]: 2016-10-18 17:47:37.358 5275 ERROR tripleo_common.actions.templates [-] Error occurred while processing custom roles.
Oct 18 17:47:37 undercloud-0.redhat.local ironic-conductor[3771]: 2016-10-18 17:47:37.358 3771 DEBUG oslo_concurrency.processutils [req-29058448-134b-490b-a8aa-761cf04846f0 - - - - -]

Comment 4 James Slagle 2016-10-18 20:04:23 UTC
dougal, can you have a look at this one?

Comment 5 Alexander Chuzhoy 2016-10-18 22:38:07 UTC
Here's what I see on the console:

[stack@undercloud-0 ~]$ yes ""| openstack overcloud update stack  -i overcloud
u"'module' object has no attribute 'OVERCLOUD_J2_EXCLUDES'"
starting package update on stack overcloud
{   'parameter_defaults': {   'DeployIdentifier': 1476844649,
                              'StackAction': 'UPDATE',
                              'UpdateIdentifier': 1476844649},
    'resource_registry': {   'resources': {   '*': {   '*': {   'UpdateDeployment': {   'hooks': 'pre-update'}}}}}}
u"'module' object has no attribute 'OVERCLOUD_J2_EXCLUDES'"
'unicode' object does not support item assignment

Comment 6 Lukas Bezdicka 2016-10-19 09:04:28 UTC
(In reply to Alexander Chuzhoy from comment #5)
I'm sorry I didn't clean up after debugging, what you see is pretty prints from the code that I forgot to remove.
> Here's what I see on the console:
> 
> [stack@undercloud-0 ~]$ yes ""| openstack overcloud update stack  -i
> overcloud
> u"'module' object has no attribute 'OVERCLOUD_J2_EXCLUDES'"
> starting package update on stack overcloud
> {   'parameter_defaults': {   'DeployIdentifier': 1476844649,
>                               'StackAction': 'UPDATE',
>                               'UpdateIdentifier': 1476844649},
>     'resource_registry': {   'resources': {   '*': {   '*': {  
> 'UpdateDeployment': {   'hooks': 'pre-update'}}}}}}
> u"'module' object has no attribute 'OVERCLOUD_J2_EXCLUDES'"
> 'unicode' object does not support item assignment

Comment 7 Lukas Bezdicka 2016-10-19 17:09:40 UTC
Fix is in upstream we just need to get it downstream.

Comment 8 Dougal Matthews 2016-10-20 07:57:05 UTC
I tried this in a fresh downstream install and got this:

[stack@instack ~]$ yes ""| openstack overcloud update stack  -i overcloud
starting package update on stack overcloud
IN_PROGRESS
WAITING
on_breakpoint: [u'overcloud-controller-0', u'overcloud-novacompute-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear aa019de3-53f9-45ab-ad9c-bde9fcd1a9ac), no=cancel update, C-c=quit interactive mode: WAITING
completed: [u'overcloud-novacompute-0']
on_breakpoint: [u'overcloud-controller-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear 4ef0ffd4-1cff-442a-abdd-d4505a408c76), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED
Stack update failed.

Comment 9 Dougal Matthews 2016-10-20 07:58:08 UTC
It does seem like I am seeing a different problem, I think this bug describes the issue before it was fixed by thrash. 

I am now digging into the heat failure and attempting to understand what the failure I got means:

[stack@instack ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3a83df99-1163-4e18-8f52-21048112049e
  status: UPDATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Matching apachectl 'Server version: Apache/2.4.6 (CentOS)
    Server built:   Jul 18 2016 15:30:14'
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
  deploy_stderr: |
    ...
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
    Warning: Scope(Class[Ceilometer]): Both $metering_secret and $telemetry_secret defined, using $telemetry_secret
    Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 166 in file /etc/puppet/modules/gnocchi/manifests/api.pp
    Warning: Scope(Class[Gnocchi::Api]): gnocchi:api::keystone_identity_uri is deprecated, use gnocchi::keystone::authtoken::auth_url instead
    Warning: Scope(Class[Gnocchi::Api]): gnocchi::api::keystone_auth_uri is deprecated, use gnocchi::keystone::authtoken::auth_uri instead
    Warning: Not collecting exported resources without storeconfigs
    Error: Duplicate declaration: Package[python-memcache] is already declared; cannot redeclare at /etc/puppet/modules/oslo/manifests/cache.pp:159 on node overcloud-controller-0.localdomain
    Error: Duplicate declaration: Package[python-memcache] is already declared; cannot redeclare at /etc/puppet/modules/oslo/manifests/cache.pp:159 on node overcloud-controller-0.localdomain
    (truncated, view all with --long)
overcloud.AllNodesDeploySteps.ComputeDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 4d6999d5-525e-44f8-a36a-49d18431e5a5
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |
    Matching apachectl 'Server version: Apache/2.4.6 (CentOS)
    Server built:   Jul 18 2016 15:30:14'
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for overcloud-novacompute-0.localdomain in environment production in 3.70 seconds
    Notice: /Stage[setup]/Tripleo::Packages::Upgrades/Exec[package-upgrade]/returns: executed successfully
    Notice: Finished catalog run in 2.66 seconds
  deploy_stderr: |
    ...
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova::Compute]): compute_manager is marked as deprecated in Nova but still needed when Ironic is used. It will be removed once Nova removes it.
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::host'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::port'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Ceilometer]): Both $metering_secret and $telemetry_secret defined, using $telemetry_secret
    Warning: Scope(Class[Ceilometer::Agent::Compute]): This class is deprecated. Please use ceilometer::agent::polling with compute namespace instead.
    (truncated, view all with --long)

Comment 10 Dougal Matthews 2016-10-20 07:59:10 UTC
re-adding rhallise@redhat.com as I didn't intend to clear that flag.

Comment 11 Ryan Hallisey 2016-10-20 14:46:33 UTC
Lukas covered the why in comment 3.  Sorry I should've included that after my comment.

Comment 13 Alexander Chuzhoy 2016-11-08 23:11:11 UTC
Verified:
Environment:
openstack-tripleo-common-5.3.0-3.el7ost.noarch

The reported issue doesn't reproduce.

Comment 15 errata-xmlrpc 2016-12-14 16:21:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.