Bug 1742991
| Summary: | Failing to deploy OC with Glance over NFS | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Tzach Shefi <tshefi> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Pranali Deore <pdeore> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Mike Abrams <mabrams> | ||||
| Severity: | urgent | Docs Contact: | Chuck Copello <ccopello> | ||||
| Priority: | urgent | ||||||
| Version: | 15.0 (Stein) | CC: | abishop, cjeanner, cschwede, cyril, eharney, gcharot, gfidente, igallagh, jpichon, jschluet, jvisser, lmarsh, mburns, nacurry, pgrist, shdunne | ||||
| Target Milestone: | z1 | Keywords: | Regression, Triaged, ZStream | ||||
| Target Release: | 15.0 (Stein) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-10.6.2-0.20190916170446.7db107a.el8ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
The OpenStack Image Service (glance) can now be configured to store images on an NFS server.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1752950 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-10-03 08:41:44 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1752950 | ||||||
| Attachments: |
|
||||||
|
Description
Tzach Shefi
2019-08-18 08:21:27 UTC
Failed to verify/deploy overcloud failed
(undercloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed
15-trunk -p RHOS_TRUNK-15.0-RHEL-8-20190924.n.2
(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-10.6.2-0.20190923210442.7db107a.el8ost.noarch
cat /var/lib/mistral/overcloud/ansible.log
2019-09-25 08:42:17,302 p=4638 u=mistral | TASK [cinder enable the LVM losetup service] ***********************************
2019-09-25 08:42:17,302 p=4638 u=mistral | Wednesday 25 September 2019 08:42:17 -0400 (0:00:00.205) 0:02:19.412 ***
2019-09-25 08:42:17,367 p=4638 u=mistral | skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"}
2019-09-25 08:42:17,432 p=4638 u=mistral | skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"}
2019-09-25 08:42:17,453 p=4638 u=mistral | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"}
2019-09-25 08:42:17,513 p=4638 u=mistral | TASK [create persistent logs directory] ****************************************
2019-09-25 08:42:17,513 p=4638 u=mistral | Wednesday 25 September 2019 08:42:17 -0400 (0:00:00.210) 0:02:19.622 ***
2019-09-25 08:42:17,789 p=4638 u=mistral | changed: [controller-0] => (item={'path': '/var/log/containers/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/containers/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/containers/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 0}
2019-09-25 08:42:17,866 p=4638 u=mistral | changed: [controller-1] => (item={'path': '/var/log/containers/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/containers/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/containers/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 0}
2019-09-25 08:42:17,933 p=4638 u=mistral | changed: [controller-2] => (item={'path': '/var/log/containers/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/containers/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/containers/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 0}
2019-09-25 08:42:18,041 p=4638 u=mistral | changed: [controller-0] => (item={'path': '/var/log/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 24, "state": "directory", "uid": 0}
2019-09-25 08:42:18,131 p=4638 u=mistral | changed: [controller-1] => (item={'path': '/var/log/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 24, "state": "directory", "uid": 0}
2019-09-25 08:42:18,165 p=4638 u=mistral | changed: [controller-2] => (item={'path': '/var/log/glance', 'setype': 'svirt_sandbox_file_t'}) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": {"path": "/var/log/glance", "setype": "svirt_sandbox_file_t"}, "mode": "0755", "owner": "root", "path": "/var/log/glance", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 24, "state": "directory", "uid": 0}
2019-09-25 08:42:18,234 p=4638 u=mistral | TASK [glance logs readme] ******************************************************
2019-09-25 08:42:18,234 p=4638 u=mistral | Wednesday 25 September 2019 08:42:18 -0400 (0:00:00.721) 0:02:20.343 ***
2019-09-25 08:42:18,842 p=4638 u=mistral | ok: [controller-0] => {"changed": false, "checksum": "e368ae3272baeb19e1113009ea5dae00e797c919", "dest": "/var/log/glance/readme.txt", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/var/log/glance/readme.txt", "secontext": "system_u:object_r:glance_log_t:s0", "size": 80, "state": "file", "uid": 0}
2019-09-25 08:42:18,909 p=4638 u=mistral | ok: [controller-1] => {"changed": false, "checksum": "e368ae3272baeb19e1113009ea5dae00e797c919", "dest": "/var/log/glance/readme.txt", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/var/log/glance/readme.txt", "secontext": "system_u:object_r:glance_log_t:s0", "size": 80, "state": "file", "uid": 0}
2019-09-25 08:42:18,986 p=4638 u=mistral | ok: [controller-2] => {"changed": false, "checksum": "e368ae3272baeb19e1113009ea5dae00e797c919", "dest": "/var/log/glance/readme.txt", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/var/log/glance/readme.txt", "secontext": "system_u:object_r:glance_log_t:s0", "size": 80, "state": "file", "uid": 0}
2019-09-25 08:42:19,042 p=4638 u=mistral | TASK [Mount NFS on host] *******************************************************
2019-09-25 08:42:19,043 p=4638 u=mistral | Wednesday 25 September 2019 08:42:19 -0400 (0:00:00.808) 0:02:21.152 ***
2019-09-25 08:42:19,506 p=4638 u=mistral | fatal: [controller-1]: FAILED! => {"changed": false, "msg": "Error mounting /var/lib/glance/images: mount: nfs: mount point does not exist.\n"}
2019-09-25 08:42:19,511 p=4638 u=mistral | fatal: [controller-2]: FAILED! => {"changed": false, "msg": "Error mounting /var/lib/glance/images: mount: nfs: mount point does not exist.\n"}
2019-09-25 08:42:19,512 p=4638 u=mistral | fatal: [controller-0]: FAILED! => {"changed": false, "msg": "Error mounting /var/lib/glance/images: mount: nfs: mount point does not exist.\n"}
2019-09-25 08:42:19,513 p=4638 u=mistral | NO MORE HOSTS LEFT *************************************************************
2019-09-25 08:42:19,514 p=4638 u=mistral | PLAY RECAP *********************************************************************
Ignore above I know why it failed, retesting. Verified on:
openstack-tripleo-heat-templates-10.6.2-0.20190923210442.7db107a.el8ost.noarch
(undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh
#!/bin/bash
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/storage/glance-nfs.yaml \ -------------------> added for Glance over NFS
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock1.rdu2.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/network/dvr-override.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/extra_templates.yaml \ ----> here is where I pushed glance staff see below
--log-file overcloud_deployment_70.log
(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/virt/extra_templates.yaml
parameter_defaults:
BarbicanSimpleCryptoGlobalDefault: true
CinderEnableIscsiBackend: false
GlanceBackend: 'file' This and the line below are defaults on NFS yaml, suspect I've tested without them.
GlanceNfsEnabled: true
GlanceNfsShare: 'x.y.160.111:/export/ins_glance'
Overcloud deployment completed without errors.
Glance uses a file (mapped to nfs) as backend as expected
Successfully upload an image
+----+------+
(overcloud) [stack@undercloud-0 ~]$ glance image-create --disk-format qcow2 --container-format bare --file cirros-0.4.0-x86_64-disk.img --name cirros2
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2019-09-25T17:55:31Z |
| direct_url | file:///var/lib/glance/images/7a0aa78c-0e7c-449e-a197-396e6ad56eab | -This path is correct as nfs mount seen later below mounts on /var/lib/glance/images/
| disk_format | qcow2 |
| id | 7a0aa78c-0e7c-449e-a197-396e6ad56eab |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros2 |
| os_hash_algo | sha512 |
| os_hash_value | 6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e |
| | 2161b5b5186106570c17a9e58b64dd39390617cd5a350f78 |
| os_hidden | False |
| owner | fe36ceb372704608821b957ee1c128cf |
| protected | False |
| size | 12716032 |
| status | active | Image is active/alive :)
| tags | [] |
| updated_at | 2019-09-25T17:55:32Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
upload another image:
glance image-create --disk-format qcow2 --container-format bare --file cirros-0.4.0-x86_64-disk.img --name cirros3
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2019-09-25T17:57:59Z |
| direct_url | file:///var/lib/glance/images/ab3c446b-b21a-4955-abcc-0e877b453886 |
| disk_format | qcow2 |
| id | ab3c446b-b21a-4955-abcc-0e877b453886 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros3 |
| os_hash_algo | sha512 |
| os_hash_value | 6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e |
| | 2161b5b5186106570c17a9e58b64dd39390617cd5a350f78 |
| os_hidden | False |
| owner | fe36ceb372704608821b957ee1c128cf |
| protected | False |
| size | 12716032 |
| status | active |
| tags | [] |
| updated_at | 2019-09-25T17:58:01Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+-----------------------------------------------------
Now lets check the NFS share I see both images created.
ignore the ספט local date forma, plus first image started with 7 so the output looks mixed 7 before the time 20:55 so it's ok.
ssh into nfs server at x.y.160.111
ll on /export/ins_glance
-rw-r-----. 1 42415 42415 12716032 ספט 25 20:55 7a0aa78c-0e7c-449e-a197-396e6ad56eab
-rw-r-----. 1 42415 42415 12716032 ספט 25 20:58 ab3c446b-b21a-4955-abcc-0e877b453886
As expected controller has the mount else this wouldn't have worked.
[root@controller-0 ~]# mount | grep 10.35
x.y.160.111:/export/ins_glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:container_file_t:s0,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.144,local_lock=none,addr=x.y.160.111,_netdev)
[root@controller-0 ~]# ll /var/lib/glance/images
total 24840
-rw-r-----. 1 42415 42415 12716032 Sep 25 17:55 7a0aa78c-0e7c-449e-a197-396e6ad56eab
-rw-r-----. 1 42415 42415 12716032 Sep 25 17:58 ab3c446b-b21a-4955-abcc-0e877b453886
Just to be extra sure also consumed image via booting an instance from cirros2 image
+--------------------------------------+-------+--------+-----------------------------------+---------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------+--------+-----------------------------------+---------+--------+
| 59cd1629-b541-45f7-8da7-710e74fd1591 | inst1 | ACTIVE | internal=192.168.0.22, 10.0.0.246 | cirros2 | tiny |
+--------------------------------------+-------+--------+-----------------------------------+---------+--------+
Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2958 |