Created attachment 1086309 [details]
tempest.conf

After having used RDOManager (Director) to deploy a testing Overcloud (1 controller node and 1 compute node on KVM), and following advice from Red Hat documentation, when running the Tempest suite of tests to validate the Overcloud by running the following command:

$ source ~/overcloudrc
$ keystone role-create --name heat_stack_owner
$ openstack overcloud validate --overcloud-auth-url $OS_AUTH_URL --overcloud-admin-password $OS_PASSWORD

However, some of the Tempest tests fail. After tracking down some of them to the S3 tests:

$ cd /home/stack/tempest
./tools/run-tests.sh '.*s3.*'

As far as It  can  be read from the source code, the S3 tests rely on the "boto" Python library. The "boto" logging is limited  and after quick hacking the code to figure out why these S3 tests are failing. Turns out the S3 tests are failing because they are trying to hit the S3 API on "localhost:8080" which clearly won't work: the Tempest tests are ignoring Keystone endpoints completely:

$ keystone endpoint-get --service object-store
+------------------------+----------------------------------------------------------------------+
|        Property        |                                Value                                 |
+------------------------+----------------------------------------------------------------------+
| object-store.publicURL | http://192.168.246.100:8080/v1/AUTH_4b2cfc10af4546bb88e078435912d6c2 |
+------------------------+----------------------------------------------------------------------+

From there  Swift S3 API listens on 192.168.246.100:8080 (not localhost).

Potential rca is etc/tempest.conf is not properly configured. This is an extract from etc/tempest.conf.sample in particular for the "boto" configuration:

[boto]

#
# From tempest.config
#

# EC2 URL (string value)
#ec2_url = http://localhost:8773/services/Cloud

# S3 URL (string value)
#s3_url = http://localhost:8080

# AWS Secret Key (string value)
#aws_secret = <None>

It doesn't look a coincidence that S3 tests are using http://localhost:8080 (the assumed default value for s3_url) instead of http://192.168.246.100:8080.

The "overcloud validate" plugin runs the following shell commands to configure and run Tempest:

$ mkdir /home/stack/tempest
$ cd /home/stack/tempest
$ /usr/share/openstack-tempest-kilo/tools/configure-tempest-directory
$ ./tools/config_tempest.py --out etc/tempest.conf --deployer-input /dev/null --debug --create identity.uri http://192.168.246.100:5000/v2.0/ compute.allow_tenant_isolation true object-storage.operator_role SwiftOperator identity.admin_password xxxxxx compute.build_timeout 500 compute.image_ssh_user cirros compute.ssh_user cirros network.build_timeout 500 volume.build_timeout 500 scenario.ssh_user cirros

attached to the BZ the resulting etc/tempest.conf file.

Expected the Swift S3 tests to pass, and have them talk to the correct Swift service (not localhost).

When does the behavior occur? Frequently?  Repeatedly?   At certain times?

Always.