Bug 1378029

Summary: OSP9/10 Init stage fails: Incorrect arguments to "list_join" should be: "list_join"
Product: Red Hat OpenStack Reporter: Sofer Athlan-Guyot <sathlang>
Component: openstack-tripleo-heat-templatesAssignee: mathieu bultel <mbultel>
Status: CLOSED NOTABUG QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: augol, ggillies, ipetrova, jcoufal, mandreou, mbultel, mburns, mcornea, rhel-osp-director-maint, sathlang, slee
Target Milestone: ---Keywords: Triaged
Target Release: 10.0 (Newton)Flags: ipetrova: needinfo? (mbultel)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-13 10:58:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Sofer Athlan-Guyot 2016-09-21 11:02:37 UTC
Description of problem:  Running the init stage of the upgrade fails with UPDATE_FAILED .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]

How reproducible: always

Comment 2 Sofer Athlan-Guyot 2016-09-21 11:08:15 UTC
Adding upstream review.

Comment 4 Jaromir Coufal 2016-10-10 03:40:03 UTC
Seemsnot to be reproducable regarding the upstrema comments, Matthie can you update the bug accordingly and close if not reproducable?

Comment 5 Marios Andreou 2016-10-13 10:58:27 UTC
I am going to close this as not a bug based on the upstream bug comments (TBD if what Graeme reports is due to old packages?)

Please reopen if you disagree

Comment 6 Marios Andreou 2016-10-13 11:00:17 UTC
adding @matbu since he worked on the fix when we thought we needed one - are we OK to close this now. I believe I also hit this at the time but have since not encountered it; could have been addressed by one of the client/commmon patches?

Comment 7 Sofer Athlan-Guyot 2016-10-14 05:33:24 UTC
I confirm that we are not using any upstream patch related to this bug.  Good to close on my side.

Comment 8 Graeme Gillies 2016-10-20 23:06:32 UTC
Hi,

Can we get this bug re-opened? I'm still experiencing it 100% of the time trying to upgrade from mitaka to newton (9-10) using the latest packages.

I can provide access to the environment if needed to see for yourself.

Regards,

Graeme

Comment 10 Sean Lee 2017-01-12 22:23:52 UTC
This appears to be caused by Join() vs JoinMultiple() in heat/engine/hot/functions.py.

If I understand the code correctly, per heat/engine/hot/template.py, template versions older than 20151015 would set list_join to use Join(). 20151015 and newer would use JoinMultiple().

Join() expects one list:

    self._delim, self._strings = self.args

JoinMultiple() supports more than one list:

    self._delim = args[0]
    self._joinlists = args[1:]

"enabled_services" (overcloud.j2.yaml) is trying to join multiple lists. So use of Join() would fail.

The 'Incorrect arguments to "list_join"' message is misleading. If you replace the line in the code with:

    raise ValueError(traceback.format_exc())

you would see the actual python error:

    ValueError: too many values to unpack

Comment 11 Irina Petrova 2017-04-10 10:04:25 UTC
Hi, we have a new customer hitting the bug:

~~~ heat-engine.log 
2017-04-07 11:56:29.792 5409 INFO heat.engine.stack [req-83fcf90b-6675-400e-b2d8-4dd6aea2e213 - - - - -] Stack UPDATE FAILED (overcloud): resources.allNodesConfig.properties.enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]    
~~~

$ grep heat sosreport-20170406-153529/gxospdir41.isamgmt.local/installed-rpms 
heat-cfntools-1.3.0-2.el7ost.noarch                         Mon Apr  3 16:39:59 2017
openstack-heat-api-7.0.2-1.el7ost.noarch                    Mon Apr  3 16:40:36 2017
openstack-heat-api-cfn-7.0.2-1.el7ost.noarch                Mon Apr  3 16:40:36 2017
openstack-heat-api-cloudwatch-7.0.2-1.el7ost.noarch         Mon Apr  3 16:40:36 2017
openstack-heat-common-7.0.2-1.el7ost.noarch                 Mon Apr  3 16:40:16 2017
openstack-heat-engine-7.0.2-1.el7ost.noarch                 Mon Apr  3 16:40:36 2017
openstack-heat-templates-0-0.11.1e6015dgit.el7ost.noarch    Mon Apr  3 16:39:53 2017
openstack-tripleo-heat-templates-5.2.0-3.el7ost.noarch      Mon Apr  3 16:39:26 2017
openstack-tripleo-heat-templates-compat-2.0.0-41.el7ost.noarch Mon Apr  3 16:41:50 2017
puppet-heat-9.5.0-1.el7ost.noarch                           Mon Apr  3 16:39:27 2017
python-heat-agent-0-0.11.1e6015dgit.el7ost.noarch           Mon Apr  3 16:40:03 2017
python-heat-tests-7.0.2-1.el7ost.noarch                     Mon Apr  3 16:40:36 2017
python-heatclient-1.5.0-1.el7ost.noarch                     Mon Apr  3 16:39:55 2017

$ openstack stack template show overcloud | grep heat_template_version
heat_template_version: '2015-04-30'


Please advise/comment.

Comment 12 mathieu bultel 2017-04-10 13:59:30 UTC
Hi, thank you for additional information.
I checked the latest 10 package (from the undercloud) and I got this:
rpm -qa | grep heat
python-heat-agent-0-0.11.1e6015dgit.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-templates-0-0.11.1e6015dgit.el7ost.noarch
openstack-tripleo-heat-templates-5.2.0-9.el7ost.noarch
openstack-heat-api-cfn-7.0.2-4.el7ost.noarch
python-heat-tests-7.0.2-4.el7ost.noarch
openstack-heat-api-7.0.2-4.el7ost.noarch
python-heatclient-1.5.0-2.el7ost.noarch
openstack-heat-common-7.0.2-4.el7ost.noarch
openstack-heat-engine-7.0.2-4.el7ost.noarch
puppet-heat-9.5.0-1.el7ost.noarch


So few packages is newer than what the customer has on this undercloud.
I suggest first, (and this is what we recommend too), to do a minor to the very latest 10 packages (on the undercloud).
Then it would be nice to see if the issue is still happening. I think it's the first very simple step to do before jumping in larger debugging.