Description of problem: Minor update support for ODL with OSP13 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1472060 [details] minor_update.sh
Created attachment 1472061 [details] prepare_container.sh
Noam, please follow the steps as in minor_update.sh. Check L2 update.
Run the scripts (with L2 option) on OSP 13 2018-07-13.1. Script completed, but it still shows the same version: [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 13 -p 2018-07-13.1
Created attachment 1472780 [details] odl minor update - console output
Your script seems to missing the update converge part: openstack overcloud update converge \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/config_heat.yaml \ -e /home/stack/virt/nodes_data.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/update-odl.yaml \ |& tee update_converge.log
Created attachment 1472820 [details] odl minor update converge - console output Also after running the update converge, the puddle on undercloud is still: OSP13 2018-07-13.1
Created attachment 1473379 [details] HTTP error 500 after update (overcloud) [stack@undercloud-0 ~]$ openstack server list The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5ea368cf-7000-4b5b-bfc2-3dfa3da6f974)
Noam, did you run "openstack undercloud upgrade" on undercloud. The whole update process takes about 3 4 hours to complete. You need to start with beta puddle and update to latest puddle.
Yes, the process took 3-4 hours, please see attachments (console output and logs). Here's the undercloud sosreports too: http://rhos-release.virt.bos.redhat.com/log/bz1597666
Created attachment 1475287 [details] odl critical error in neutron server.log The root cause for HTTP 500 - opendaylight_v2 driver was not found after update: 018-08-02 21:19:48.262 1 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['opendaylight_v2'] 2018-08-02 21:19:48.262 1 CRITICAL neutron.plugins.ml2.managers [-] The following mechanism drivers were not found: set(['opendaylight_v2']) 2018-08-02 21:19:48.263 1 DEBUG oslo_concurrency.lockutils [-] Lock "manager" released by "neutron.manager._create_instance" :: held 0.714s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
The neutron_api container hangs on restarting: [heat-admin@controller-0 ~]$ sudo -s [heat-admin@controller-0 ~]$ docker exec neutron_api grep -A3 ml2_odl /etc/neutron/* Error response from daemon: Container bd06fbe73909eb570f24dee8f731c66506589b614709bb2462dacc6eba9e6f40 is restarting, wait until the container is running [heat-admin@controller-0 ~]$ sudo -s [root@controller-0 heat-admin]# [root@controller-0 heat-admin]# docker info Containers: 113 Running: 60 Paused: 0 Stopped: 53 Images: 41 Server Version: 1.13.1 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: journald Cgroup Driver: systemd Plugins: Volume: local Network: bridge host macvlan null overlay Authorization: rhel-push-plugin Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Init Binary: /usr/libexec/docker/docker-init-current containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1) runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f) init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574) Security Options: seccomp WARNING: You're not using the default seccomp profile Profile: /etc/docker/seccomp.json Kernel Version: 3.10.0-862.9.1.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo) OSType: linux Architecture: x86_64 Number of Docker Hooks: 3 CPUs: 8 Total Memory: 31.26 GiB Name: controller-0 ID: UBEC:5J4T:QRTA:JERX:GHV4:PBEJ:N2C2:23C2:3SUF:X5OR:JFSQ:PXF5 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 695 Goroutines: 285 System Time: 2018-08-12T09:47:36.579834771+01:00 EventsListeners: 0 Registry: https://registry.access.redhat.com/v1/ Experimental: false Insecure Registries: 192.168.24.1:8787 127.0.0.0/8 Live Restore Enabled: true Registries: registry.access.redhat.com (secure), docker.io (secure) [root@controller-0 heat-admin]# [root@controller-0 heat-admin]# docker version Client: Version: 1.13.1 API version: 1.26 Package version: docker-1.13.1-68.gitdded712.el7.x86_64 Go version: go1.9.2 Git commit: dded712/1.13.1 Built: Tue Jun 12 18:30:09 2018 OS/Arch: linux/amd64 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Package version: docker-1.13.1-68.gitdded712.el7.x86_64 Go version: go1.9.2 Git commit: dded712/1.13.1 Built: Tue Jun 12 18:30:09 2018 OS/Arch: linux/amd64 Experimental: false [root@controller-0 heat-admin]# [root@controller-0 heat-admin]# uname -a Linux controller-0 3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux [root@controller-0 heat-admin]# [root@controller-0 heat-admin]# docker ps -f name=neutron_api CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bd06fbe73909 registry.access.redhat.com/rhosp13/openstack-neutron-server:latest "kolla_start" 9 days ago Restarting (1) 38 hours ago neutron_api [root@controller-0 heat-admin]# docker kill bd06fbe73909 ^Z ## This docker proccess kill has hanged for too long, restarting docker service [1]+ Stopped docker kill bd06fbe73909 [root@controller-0 heat-admin]# bg [1]+ docker kill bd06fbe73909 & [root@controller-0 heat-admin]# systemctl stop docker error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/bd06fbe73909/kill?signal=KILL: EOF [1]+ Exit 1 docker kill bd06fbe73909 [root@controller-0 heat-admin]# systemctl start docker [root@controller-0 heat-admin]# docker ps -f name=neutron_api CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bd06fbe73909 registry.access.redhat.com/rhosp13/openstack-neutron-server:latest "kolla_start" 9 days ago Up 1 second (health: starting) neutron_api [root@controller-0 heat-admin]# docker exec neutron_api grep -A3 ml2_odl /etc/neutron/* Error response from daemon: Container bd06fbe73909eb570f24dee8f731c66506589b614709bb2462dacc6eba9e6f40 is restarting, wait until the container is running
After checking the container files it seems that the networking-odl package isnt installed on the container, thus neutron can't load it and fails. The thing to check now is if it's present in the new image or not?
(In reply to Mike Kolesnik from comment #27) > After checking the container files it seems that the networking-odl package > isnt installed on the container, thus neutron can't load it and fails. > > The thing to check now is if it's present in the new image or not? Janki, this is probably due to the minor update process (logs and sosreport in previous comments). please note that we could access the neutron-api image, but not the running container, since it was constantly stuck on restarting, this way: [root@controller-0 heat-admin]# docker ps -f name=neutron_api CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bd06fbe73909 registry.access.redhat.com/rhosp13/openstack-neutron-server:latest "kolla_start" 9 days ago Restarting (1) 38 hours ago neutron_api docker run --rm -it --entrypoint=/bin/bash registry.access.redhat.com/rhosp13/openstack-neutron-server:latest ()[neutron@dd6d07035bc9 /]$ rpm -qa | grep networking-odl ()[neutron@dd6d07035bc9 /]$ rpm -qa | grep odl perl-podlators-2.5.1-3.el7.noarch ()[neutron@dd6d07035bc9 /]$
Hi The image itself is wrong. We need neutron-server image tagged for opendaylight "openstack-neutron-server-opendaylight" But the system has "openstack-neutron-server:latest". I see minor_update.sh script run as is. It expects /home/stack/docker_registry.yaml which is absent. prepare_contaier.sh doesnot seem to be run properly. And hence the wrong image on overcloud because noone said which image to fetch so it just fetched the default one. @Noam, lets test it together tomorrow. Reprovision this and deploy openstack with an older puddle and we will update to newer one.
looks like testing was done wrong you should be getting openstack-neutron-server-opendaylight Please look at the artifact generated by your openstack overcloud container image prepare. Next check to make sure /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml hasn't shifted and regressed in pointing to containerized deployment. # docker run -it --entrypoint=/bin/bash -u root docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-server-opendaylight:latest ()[root@e9f8738bebf4 /]# rpm -qa | grep networking-odl python-networking-odl-12.0.1-0.20180228140716.8733cf6.el7ost.noarch
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Minor update of OSP13 with ODL, from Puddle 2018-08-03.3 to Puddle 2018-08-14.4 Passed successfully! (undercloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/rhos-release-13.repo | grep puddle_baseurl # puddle_baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/2018-08-14.4/RH7-RHOS-13.0/$basearch/os (undercloud) [stack@undercloud-0 ~]$ openstack --version openstack 3.14.2 [heat-admin@controller-1 ~]$ rpm -qa | grep opend puppet-opendaylight-8.2.2-2.9126c8dgit.el7ost.noarch [heat-admin@controller-1 ~]$ curl -v -H "Content-Type:application/json" -u odladmin:redhat http://172.17.1.18:8081/controller/nb/v2/neutron/networks * About to connect() to 172.17.1.18 port 8081 (#0) * Trying 172.17.1.18... * Connected to 172.17.1.18 (172.17.1.18) port 8081 (#0) * Server auth using Basic with user 'odladmin' > GET /controller/nb/v2/neutron/networks HTTP/1.1 > Authorization: Basic b2RsYWRtaW46cmVkaGF0 > User-Agent: curl/7.29.0 > Host: 172.17.1.18:8081 > Accept: */* > Content-Type:application/json > < HTTP/1.1 200 OK < Set-Cookie: JSESSIONID=1mht4nrffa1psrizl6q26rdtv;Path=/controller/nb/v2/neutron < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Set-Cookie: rememberMe=deleteMe; Path=/controller/nb/v2/neutron; Max-Age=0; Expires=Wed, 15-Aug-2018 15:52:39 GMT < Content-Type: application/json < Transfer-Encoding: chunked < { "networks" : [ { "id" : "bd8db3a8-2b30-4083-a8b3-b3fd46401142", "tenant_id" : "bd8db3a82b304083a8b3b3fd46401142", "project_id" : "bd8db3a8-2b30-4083-a8b3-b3fd46401142", "name" : "Sync Canary Network", "admin_state_up" : false, "status" : "ACTIVE", "shared" : false, "router:external" : false, "provider:network_type" : "flat", "segments" : [ ], "vlan_transparent" : false } ] * Connection #0 to host 172.17.1.18 left intact }
Also need to verify 594731 : * ODL Post update tasks during openstack overcloud update run --nodes Controller: https://bugzilla.redhat.com/show_bug.cgi?id=1594731 Check if any of the these tasks show up in the update logs on controller node: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/opendaylight-api.yaml#L223 it would be these tasks. The 2nd block is conditional by update level, but at least the 1st block should be executed unconditionally. https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/opendaylight-api.yaml#L284-L287 So at least this should be visible in the ansible log from update: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/opendaylight-api.yaml#L254-L258
How did you generate this docker_registry.yaml file? Use prepare_container.sh script (attached before) or use this command PREPARE_ARGS=${PREPARE_ARGS:-"-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml"} namespace=rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp13 tag=2018-08-22.2 prefix=openstack openstack overcloud container image prepare \ --images-file /home/stack/overcloud_containers.yaml \ ${PREPARE_ARGS} \ --namespace $namespace \ --tag $tag \ --prefix $prefix \ --push-destination 192.168.24.1:8787 --env-file /home/stack/docker_registry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-opendaylight.yaml
Created attachment 1479494 [details] openstack overcloud update run --nodes Controller failed (controller-0) openstack overcloud update run --nodes Controller failed on controller-0 : Warning: ModuleLoader: module \'neutron\' has unresolved dependencies - it will only see those that are resolved. Use \'puppet module list --tree\' to see information about modules", ', u' "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\\"/etc/puppet/modules/snmp/manifests/params.pp\\", 310]:[\\"/var/lib/tripleo-config/puppet_step_config.pp\\", 42]", ', u' " with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ssh/manifests/server.pp\\", 12]:[\\"/var/lib/tripleo-config/puppet_step_config.pp\\", 44]", ', u' "Error: curl -k -o /dev/null --fail --silent --head -u odladmin:redhat http://172.17.1.18:8081/restconf/operational/network-topology:network-topology/topology/netvirt:1 returned 22 instead of one of [0]", ', u' "Error: /Stage[main]/Neutron::Plugins::Ovs::Opendaylight/Exec[Wait for NetVirt OVSDB to come up]/returns: change from notrun to 0 failed: curl -k -o /dev/null --fail --silent --head -u odladmin:redhat http://172.17.1.18:8081/restconf/operational/network-topology:network-topology/topology/netvirt:1 returned 22 instead of one of [0]", ', u' "Warning: /Stage[main]/Neutron::Plugins::Ovs::Opendaylight/Exec[Set OVS Manager to OpenDaylight]: Skipping because of failed dependencies"', u' ]', u'}', u'', u'PLAY RECAP *********************************************************************', u'controller-0 : ok=242 changed=92 unreachable=0 failed=1 ', u'controller-1 : ok=259 changed=93 unreachable=0 failed=0 ', u'controller-2 : ok=259 changed=93 unreachable=0 failed=0 ',
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.0.7-21.el7ost. This build is available now.
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality