Hide Forgot
OC with NFS backend: glance_api container is restarting: ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found Environment: python-glanceclient-2.8.0-1.el7ost.noarch python-glance-15.0.0-3.el7ost.noarch python-glance-store-0.22.0-1.el7ost.noarch puppet-glance-11.3.1-0.20171102185757.0c03057.el7ost.noarch openstack-glance-15.0.0-3.el7ost.noarch openstack-tripleo-heat-templates-7.0.3-6.el7ost.noarch puppet-tripleo-7.4.3-5.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch instack-undercloud-7.4.3-2.el7ost.noarch Steps to reproduce: 1) Deploy overcloud with: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/nfs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml (undercloud) [stack@undercloud-0 ~]$ cat nfs.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'nosharecache' CinderNfsServers: 192.168.24.1:/export/cinder GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: '192.168.24.1:/export/glance' GlanceNfsMountOptions: 'nosharecache' 2) Try to run glance commands against overcloud Result: (overcloud) [stack@undercloud-0 ~]$ openstack image list Service Unavailable (HTTP 503) (overcloud) [stack@undercloud-0 ~]$ glance image-list 503 Service Unavailable: No server is available to handle this request. (HTTP 503) Checking glance on controller: [root@overcloud-controller-0 ~]# docker ps|grep glance 402b9076af8b 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:20171115.1 "kolla_start" 10 hours ago Restarting (1) 2 hours ago glance_api [root@overcloud-controller-0 ~]# docker logs glance_api INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found
The glance_api config volume does not exist. It fails to run puppet during the config_generation step: Error: /Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]: Could not evaluate: No such file or directory - /etc/fstab
I've successfully tested a fix. Patch incoming.
Taking this bug to the containers DFG.
Indeed, puppet-tripleo wants to write to /etc/fstab and this fails again to generate the configuration file. Before generating the glance config, /etc/fstab on the host looks like: 192.168.24.1:/export/glance /var/lib/glance nfs4 intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 After generating the glance config, bind mounting /etc/fstab:rw this time, it looks like: 192.168.24.1:/export/glance /var/lib/glance nfs4 intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 Which basically overwrites the config for 192.168.24.1:/export/glance. So I'm wondering if the best approach here is to mount the file rw (in which case puppet-tripleo override the existing config) or tell puppet-tripleo to not write to /etc/fstab by setting tripleo::glance::nfs_mount::edit_fstab to false. It would be nice to get input from the storage DFG for the correct approach.
Cyril, can someone from the storage DFG help us determine which is the correct setting for /etc/fstab? Should it be 192.168.24.1:/export/glance /var/lib/glance nfs4 intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 or 192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 This will help us decide if we bind mount the /etc/fstab file rw or set puppet-tripleo not to write to /etc/fstab.
@martin, keeping _netdev is recommended because NFS end point mentioned in fstab will be mounted after starting the networking service.
*** Bug 1515801 has been marked as a duplicate of this bug. ***
Pranali, please also note the mount point is different, /var/lib/glance vs /var/lib/glance/images. I assume the config coming from https://github.com/openstack/puppet-tripleo/blob/master/manifests/glance/nfs_mount.pp is the correct one, but that raises the question of what initially sets the /etc/fstab on the host. I'm pretty sure the mount command [1] is ignored in the container because we restrict the puppet catalog to only generate configuration files. On second thoughts, mounting /etc/fstab rw in the container doesn't seem appropriate. IIUC, we need to: - find what sets /etc/fstab on the host - ensure the mount point and options are correct - disable writing to /etc/fstab from the container by setting tripleo::glance::nfs_mount::edit_fstab to false in the t-h-t I'm moving this bug back to the storage DFG, let me know if you need any help on the containers part to tell puppet not to write to /etc/fstab from within the container. [1] https://github.com/openstack/puppet-tripleo/blob/9b438a0249d59399c63c016921a9b6add8159dd2/manifests/glance/nfs_mount.pp#L66-L70
Martin, thanks for the details.. :) So, the below is the right value for /etc/fstab 192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0 So, it would be better keep the mounting on host logic by adding these addintional options (_netdev and bg) and renaming the directory to /var/lib/glance/images. And the edit_fstab needs to be disabled in this case as you said.
I just posted an updated patch to https://review.openstack.org/521103
Tested the patch in link https://review.openstack.org/521103 The issue didn't reproduce with the patch applied.
Martin revamped the upstream patch, which I tested and it works for me. I refreshed the downstream patch (although gerrit assigned it a new review ID). Sasha, could you try the latest downstream patch [1]? [1] https://code.engineering.redhat.com/gerrit/124028
(In reply to Alan Bishop from comment #15) > Martin revamped the upstream patch, which I tested and it works for me. I > refreshed the downstream patch (although gerrit assigned it a new review ID). > > Sasha, could you try the latest downstream patch [1]? > > [1] https://code.engineering.redhat.com/gerrit/124028 Successfully tested the downstream patch in link.
Verified: Environment: openstack-tripleo-heat-templates-7.0.3-13.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462