Description of problem: Redeployment of already deployed stack fails with OSP + ODL deployment Version-Release number of selected component (if applicable): OSP13 + ODL How reproducible: Always Steps to Reproduce: 1. Deploy overcloud with ODL 2. Redeploy it without deleting the previous deployment Actual results: Deployment fails Expected results: Deployment should not fail Additional info:
Karaf log snippet: 2018-10-12T12:10:54,728 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper | 52 - config-persister-feature-adapter - 0.8.3.redhat-3 | Could not parse XML file /opt/opendaylight/etc/opendaylight/karaf/00-netty.xml java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/karaf/00-netty.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:?] at java.io.FileInputStream.open(FileInputStream.java:195) [?:?] at java.io.FileInputStream.<init>(FileInputStream.java:138) [?:?] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.isConfigSnapshot(AbstractFeatureWrapper.java:116) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.getFeatureConfigSnapshotHolders(AbstractFeatureWrapper.java:75) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:89) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] ... 2018-10-12T12:10:54,762 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper | 52 - config-persister-feature-adapter - 0.8.3.redhat-3 | Could not parse XML file /opt/opendaylight/etc/opendaylight/karaf/01-md-sal.xml java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/karaf/01-md-sal.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:?] at java.io.FileInputStream.open(FileInputStream.java:195) [?:?] at java.io.FileInputStream.<init>(FileInputStream.java:138) [?:?] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.isConfigSnapshot(AbstractFeatureWrapper.java:116) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.getFeatureConfigSnapshotHolders(AbstractFeatureWrapper.java:75) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:89) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfig(FeatureConfigPusher.java:101) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfigs(FeatureConfigPusher.java:69) [52:config-persister-feature-adapter:0.8.3.redhat-3] ... 2018-10-12T12:10:54,763 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper | 52 - config-persister-feature-adapter - 0.8.3.redhat-3 | Could not parse XML file /opt/opendaylight/etc/opendaylight/karaf/05-clustering.xml java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/karaf/05-clustering.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:?] at java.io.FileInputStream.open(FileInputStream.java:195) [?:?] at java.io.FileInputStream.<init>(FileInputStream.java:138) [?:?] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.isConfigSnapshot(AbstractFeatureWrapper.java:116) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.getFeatureConfigSnapshotHolders(AbstractFeatureWrapper.java:75) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:89) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfig(FeatureConfigPusher.java:101) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfigs(FeatureConfigPusher.java:69) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.processFeatureEvent(ConfigPushingRunnable.java:70) [52:config-persister-feature-adapter:0.8.3.redhat-3] at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.run(ConfigPushingRunnable.java:52) [52:config-persister-feature-adapter:0.8.3.redhat-3] at java.lang.Thread.run(Thread.java:748) [?:?] 2018-10-12T12:10:54,764 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper | 52 - config-persister-feature-adapter - 0.8.3.redhat-3 | Could not parse XML file /opt/opendaylight/etc/opendaylight/karaf/06-clustered-entity-ownership.xml java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/karaf/06-clustered-entity-ownership.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:?] at java.io.FileInputStream.open(FileInputStream.java:195) [?:?] at java.io.FileInputStream.<init>(FileInputStream.java:138) [?:?] @ All the /opt/opendaylight/etc/opendaylight/karaf/* files are missing 8:xx timestamps are for the first original deployment. 13:xx for the redeployment. Container is restarted just once [heat-admin@controller-0 ~]$ sudo journalctl CONTAINER_NAME=opendaylight_api | grep "sudo -E kolla_set_configs" Oct 12 08:15:31 controller-0 dockerd-current[19485]: + sudo -E kolla_set_configs Oct 12 13:10:30 controller-0 dockerd-current[19485]: + sudo -E kolla_set_configs [heat-admin@controller-0 ~]$ grep -r "etc/opendaylight/karaf" odl_journal.log Oct 12 08:15:31 controller-0 dockerd-current[19485]: INFO:__main__:Creating directory /opt/opendaylight/etc/opendaylight/karaf Oct 12 08:15:31 controller-0 dockerd-current[19485]: INFO:__main__:Setting permission for /opt/opendaylight/etc/opendaylight/karaf Oct 12 13:10:31 controller-0 dockerd-current[19485]: INFO:__main__:Creating directory /opt/opendaylight/etc/opendaylight/karaf
Root cause: /opt/opendaylight/etc/opendaylight/karaf from container is mounted to /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf on host. Contents of /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf are reflected at /opt/opendaylight/etc/opendaylight/karaf. During the first install, /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf on host is empty because files inside this folder are not generated by puppet but by kolla inside the container. During redploy, the empty /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf is mounted to container. So /opt/opendaylight/etc/opendaylight/karaf inside container is empty and hence when ODL restarts, it complais of /opt/opendaylight/etc/opendaylight/karaf/* files missing and ODL can't restart leading to failed deployment. Solution: Delete empty var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf on host before restarting ODL.
This also affects while running "update converge"
Per Janki's observation this is needed for updates to work, re-targeting to OSP 13 z4
https://review.openstack.org/#/c/615122/ is a minor patch of fixing an indentation.
To verify: 1. Deploy OSP + ODL 2. Redeploy the same stack again by running the overcloud deploy command or ./overcloud_deploy.sh script (Donot delete 'openstack stack delete overlcoud --wait --yes' the exisitng stack)
I've tested this bug with the solution Janki provided [1]. I can confirm that it solves the root cause and the overcloud update succeeds. [1] https://review.openstack.org/620053 Putting it to ON_DEV as it needs further work and that patch to land for this bug to be resolved.
With the current implementation, redeployment is sometimes still fails with the same error because /opt/opendaylight/data folder is mounted to container on restart. This folder has information which makes ODL believe that it has features installs so ODL doesnot generate /opt/opendaylight/etc/opendaylight/karaf/*.xml files which leads ODL not able to boot. Needs https://review.openstack.org/#/c/620053/ which doesnot mount ../data folder on container to ensure 100% success rate.
(In reply to Janki from comment #7) > Root cause: > /opt/opendaylight/etc/opendaylight/karaf from container is mounted to > /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/ > opendaylight/karaf on host. Contents of > /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/ > opendaylight/karaf are reflected at > /opt/opendaylight/etc/opendaylight/karaf. During the first install, > /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/ > opendaylight/karaf on host is empty because files inside this folder are not > generated by puppet but by kolla inside the container. During redploy, the > empty > /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/ > opendaylight/karaf is mounted to container. So > /opt/opendaylight/etc/opendaylight/karaf inside container is empty and hence > when ODL restarts, it complais of /opt/opendaylight/etc/opendaylight/karaf/* > files missing and ODL can't restart leading to failed deployment. > > Solution: > Delete empty > var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/ > opendaylight/karaf on host before restarting ODL. Well actually /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/opendaylight/karaf is mounted to /var/lib/kolla/config_files/src from where it is copied to /opt/opendaylight/etc/opendaylight/karaf.
I tested it with patch https://review.openstack.org/#/c/620053/ and it works fine
I verified that the cherry-pick to stable/queens (https://review.openstack.org/#/c/623324) fixes this for OSP13. Also, to be safe, I verified that without this patch the failure occurs.
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.0.7-21.el7ost. This build is available now.
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality