Bug 2218012

Moving this to PIDONE and/or Storage, as I'm not sure whether the error is due to Glance itself, or HAProxy.

@storage @pidone - the job from comment #15 failed because Nova failed to connect to Glance. We have all the logs in that link, but I'm failing to see a smoking gun, either from HAProxy, or from Glance. Need your help.

Just a wild guess but in our lab we had to add an additional service for glance_api_internal, following https://review.opendev.org/c/openstack/tripleo-heat-templates/+/863142 :

    - OS::TripleO::Services::GlanceApiInternal

resulting in haproxy being configured as:

listen glance_api
  bind 10.0.0.134:13292 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.17.1.122:9292 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  http-response replace-header Location http://(.*) https://\1
  option httpchk GET /healthcheck
  option httplog
  option forwardfor
  redirect scheme https code 301 if { hdr(host) -i 10.0.0.134 } !{ ssl_fc }
  server controller-0.internalapi.redhat.local 172.17.1.111:9292 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-0.internalapi.redhat.local
  server controller-1.internalapi.redhat.local 172.17.1.142:9292 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-1.internalapi.redhat.local
  server controller-2.internalapi.redhat.local 172.17.1.110:9292 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-2.internalapi.redhat.local

listen glance_api_internal
  bind 172.17.1.122:9293 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk GET /healthcheck
  option httplog
  option forwardfor
  server controller-0.internalapi.redhat.local 172.17.1.111:9293 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-0.internalapi.redhat.local
  server controller-1.internalapi.redhat.local 172.17.1.142:9293 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-1.internalapi.redhat.local
  server controller-2.internalapi.redhat.local 172.17.1.110:9293 ca-file /etc/ipa/ca.crt check crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem fall 5 inter 2000 rise 2 ssl verify required verifyhost controller-2.internalapi.redhat.local


I can't find this service defined in http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-hardware_provisioning-rqci-17.1-3cont_6comp_6ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-geneve-localregistry-spineleaf-multiple_subnets/29/undercloud-0/home/stack/virt/roles/roles_data.yaml.gz 

your endpoint list contains the image service listening (supposedly) on 9293:

(overcloud) [stack@undercloud ~]$ openstack endpoint list |grep 9293
| 369066f093ad468c9defa066d18800c0 | regionOne | glance       | image          | True    | admin     | http://[fd00:fd00:fd00:1000::385]:9293                                      |
| 5e076b93aa3d4bb7b3f78a6d7b652132 | regionOne | glance       | image          | True    | internal  | http://[fd00:fd00:fd00:1000::385]:9293                                      |
(overcloud) [stack@undercloud ~]$ 

but there is not service behind it, so afaicu haproxy returns 404 when someone tries to look for the cirros image. 

can you add the glance_api_internal service to your roles file and see if it helps?

running test with edited roles_data.yaml file

Tested on two jobs, the job finished to completion with no errors.  so adding the GlanceApiInternal as in https://bugzilla.redhat.com/show_bug.cgi?id=2218012#c22 fixed it (as well as also fixing BZ 2218321).  Have edited our roles_data.yaml files accordingly.