Hide Forgot
Description of problem: ======================================= Ran into an instance where imagefactory built an image but got a bad user error trying to push the image into warehouse. After some troubleshooting it looks like the culprit is running aeolus-configure multiple times for separate providers. like "aeolus-configure -p ec2; aeolus-configure -p rhevm" beaker box: ibm-hs22-03.lab.bos.redhat.com
I was able to solve this problem by restarting Conductor. There's some talk about how aeolus-configure might better configure things. I'm not sure it should generate a new set of keys every time it's run.
Should we always restart Conductor when Configure is run?
I'm not sure exactly what we do today, but I think what we should do is either: a.) Restart _all_ OAuth services when running aeolus-configure, or restart _no_ OAuth services when running aeolus-configure. The problem is that we ended up with some running with new keys and some running with old keys. -or- b.) Don't generate new keys on each run. This is the approach I took with the patch to aeolus-devel yesterday -- if an oauth.json file exists, we won't overwrite it.
if we do a.), we can put something in the puppet manifest to cause updates to the keys to cause restarts (look @ notify in puppet)
Richard and I have solved this. In Conductor, the commit is: commit 31c14ddeb979e64142ca6c5ece5c739b3e58da4b Author: Matt Wagner <matt.wagner> Date: Wed Oct 26 17:31:18 2011 -0400 OAuth configuration moved to oauth.json Rake task to write config file if it doesn't already exist. RPM spec updated to run in %post It pairs with Richard's commit on aeolus-configure: commit f01b0519a636cc735ff312f690bde84b3e7d6eeb Author: Richard Su <rwsu> Date: Fri Oct 28 11:12:23 2011 -0700 BZ 749254 - running configure for multiple providers causes oauth issues. The problem was aeolus-configure generated new uuids each time it is invoked. Matt has created a patch for conductor to have the rpm generate and install a oauth.json file which will be the canonical source for the keys. This patch augments configure to read oauth.json and to produce custom facts replacing the auto generated uuids in aeolus-configure.
root@qeblade31 ~]# aeolus-image list --images ID Name OS OS Version Arch Description ------------------------------------ ---- -- ---------- ---- ----------- b0a556b2-e7e3-42ab-a23a-6ac1de5e2f24 [root@qeblade31 ~]# aeolus-configure -p vsphere Launching aeolus configuration recipe... notice: /Stage[main]/Apache/Exec[permit-http-networking]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: conductor already exists notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Migrate::Db[migrate_aeolus_database]/Exec[migrate_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: % Total % Received % Xferd Average Speed Time Time Time Current notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: Dload Upload Total Spent Left Speed notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/retur 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: executed successfully notice: /Stage[main]/Aeolus::Deltacloud::Core/Exec[deltacloud-core-startup-wait]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Selinux::Mode[permissive]/Exec[set_selinux_permissive]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Login[admin]/Web_request[admin-conductor-login]/post: post changed '' to 'https://localhost/conductor/user_session' notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: post changed '' to 'https://localhost/conductor/hardware_profiles' err: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Provider[vsphere]/Web_request[provider-vsphere]/post: change from to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Expecting //html/body//li[text() = 'Provider added.'] in the result notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Provider::Account[vsphere]/Web_request[provider-account-vsphere]: Dependency Web_request[provider-vsphere] has failures: true warning: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Provider::Account[vsphere]/Web_request[provider-account-vsphere]: Skipping because of failed dependencies notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]: Dependency Web_request[provider-vsphere] has failures: true warning: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]: Skipping because of failed dependencies notice: Finished catalog run in 9.34 seconds [root@qeblade31 ~]# aeolus-image list --images ID Name OS OS Version Arch Description ------------------------------------ ---- -- ---------- ---- ----------- b0a556b2-e7e3-42ab-a23a-6ac1de5e2f24 [root@qeblade31 ~]# rpm -qa | grep aeolus aeolus-configure-2.2.0-1.20111028150641git1189dd7.el6.noarch rubygem-rack-mount-0.7.1-3.aeolus.el6.noarch rubygem-ZenTest-4.3.3-2.aeolus.el6.noarch aeolus-conductor-0.5.0-0.20111028214047git58c6e71.el6.noarch rubygem-arel-2.0.10-0.aeolus.el6.noarch aeolus-conductor-doc-0.5.0-0.20111028214047git58c6e71.el6.noarch aeolus-all-0.5.0-0.20111028214047git58c6e71.el6.noarch rubygem-aeolus-cli-0.1.0-3.20111028152758git7063136.el6.noarch rubygem-aeolus-image-0.1.0-4.20111024205454git6b2b696.el6.noarch aeolus-conductor-daemons-0.5.0-0.20111028214047git58c6e71.el6.noarch Just to note.. I was able to reproduce this bug previously by simply running configure for a provider and then running list images
[root@qeblade31 ~]# aeolus-cleanup notice: /Stage[main]/Aeolus::Deltacloud::Disabled/Service[deltacloud-core]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Iwhd::Disabled/Service[iwhd]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Iwhd::Disabled/Service[mongod]/ensure: ensure changed 'running' to 'stopped' notice: /File[/var/lib/iwhd]/ensure: removed notice: /Stage[main]/Aeolus::Image-factory::Disabled/Exec[remove_aeolus_templates]/returns: executed successfully notice: /File[/var/lib/aeolus-conductor]/ensure: removed notice: /Stage[main]/Aeolus::Conductor::Disabled/Service[aeolus-conductor]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Conductor::Disabled/Service[conductor-dbomatic]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Conductor::Disabled/Service[httpd]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Conductor::Disabled/Rails::Drop::Db[drop_aeolus_database]/Exec[drop_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor::Disabled/Rails::Drop::Db[drop_aeolus_database]/Exec[drop_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor::Disabled/Postgres::User[aeolus]/Exec[drop_aeolus_postgres_user]/returns: executed successfully notice: /Stage[main]/Aeolus::Image-factory::Disabled/Service[imagefactory]/ensure: ensure changed 'running' to 'stopped' notice: /Stage[main]/Aeolus::Image-factory::Disabled/Service[qpidd]/ensure: ensure changed 'running' to 'stopped' notice: Finished catalog run in 5.27 seconds [root@qeblade31 ~]# aeolus-image list --images Unable to perform task: - Internal Error: Connection refused - connect(2) [root@qeblade31 ~]# aeolus-configure -p ec2 Launching aeolus configuration recipe... notice: /Stage[main]/Aeolus::Conductor/Selinux::Mode[permissive]/Exec[set_selinux_permissive]/returns: executed successfully notice: /Stage[main]/Aeolus::Iwhd/Service[mongod]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Conductor/Postgres::User[aeolus]/Exec[create_aeolus_postgres_user]/returns: executed successfully notice: /File[/var/lib/iwhd]/ensure: created notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Migrate::Db[migrate_aeolus_database]/Exec[migrate_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Seed::Db[seed_aeolus_database]/Exec[seed_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor/Rails::Seed::Db[seed_aeolus_database]/Exec[seed_rails_database]/returns: executed successfully notice: /File[/var/lib/aeolus-conductor]/ensure: created notice: /File[/var/lib/aeolus-conductor/production.seed]/ensure: created notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: User admin registered notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: Granting administrator privileges for admin... notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: executed successfully notice: /Stage[main]/Aeolus::Deltacloud::Core/Service[deltacloud-core]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Deltacloud::Core/Exec[deltacloud-core-startup-wait]/returns: executed successfully notice: /Stage[main]/Aeolus::Image-factory/Service[qpidd]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Apache/Exec[permit-http-networking]/returns: executed successfully notice: /Stage[main]/Aeolus::Image-factory/Service[imagefactory]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Apache/Service[httpd]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Conductor/Service[conductor-dbomatic]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Conductor/Service[aeolus-conductor]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Login[admin]/Web_request[admin-conductor-login]/post: post changed '' to 'https://localhost/conductor/user_session' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: post changed '' to 'https://localhost/conductor/hardware_profiles' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Provider[ec2-us-west-1]/Web_request[provider-ec2-us-west-1]/post: post changed '' to 'https://localhost/conductor/providers' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Provider[ec2-us-east-1]/Web_request[provider-ec2-us-east-1]/post: post changed '' to 'https://localhost/conductor/providers' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]/post: post changed '' to 'https://localhost/conductor/logout' notice: /Stage[main]/Aeolus::Iwhd/Service[iwhd]/ensure: ensure changed 'stopped' to 'running' notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: % Total % Received % Xferd Average Speed Time Time Time Current notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0create-bucket-aeolus]/returns: notice: /Stage[main]/Aeolus::Profiles::Ec2/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: executed successfully notice: Finished catalog run in 45.38 seconds [root@qeblade31 ~]# aeolus-image list --images ID Name OS OS Version Arch Description ------------------------------------ ---- -- ---------- ---- ----------- b0a556b2-e7e3-42ab-a23a-6ac1de5e2f24 [root@qeblade31 ~]# aeolus-configure -p vsphere Launching aeolus configuration recipe... notice: /Stage[main]/Aeolus::Conductor/Selinux::Mode[permissive]/Exec[set_selinux_permissive]/returns: executed successfully notice: /Stage[main]/Apache/Exec[permit-http-networking]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: % Total % Received % Xferd Average Speed Time Time Time Current notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: Dload Upload Total Spent Left Speed notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/retur 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: conductor already exists notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Migrate::Db[migrate_aeolus_database]/Exec[migrate_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Login[admin]/Web_request[admin-conductor-login]/post: post changed '' to 'https://localhost/conductor/user_session' notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: post changed '' to 'https://localhost/conductor/hardware_profiles' notice: /Stage[main]/Aeolus::Deltacloud::Core/Exec[deltacloud-core-startup-wait]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Provider[vsphere]/Web_request[provider-vsphere]/post: post changed '' to 'https://localhost/conductor/providers' notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Provider::Account[vsphere]/Web_request[provider-account-vsphere]/post: post changed '' to 'https://localhost/conductor/provider_accounts' notice: /Stage[main]/Aeolus::Profiles::Vsphere/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]/post: post changed '' to 'https://localhost/conductor/logout' notice: Finished catalog run in 9.92 seconds [root@qeblade31 ~]# aeolus-image list --images ID Name OS OS Version Arch Description ------------------------------------ ---- -- ---------- ---- ----------- b0a556b2-e7e3-42ab-a23a-6ac1de5e2f24 [root@qeblade31 ~]# aeolus-configure -p rhevm Launching aeolus configuration recipe... notice: /Stage[main]/Apache/Exec[permit-http-networking]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Selinux::Mode[permissive]/Exec[set_selinux_permissive]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: conductor already exists notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: Using gem require instead of bundler notice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Conductor/Rails::Migrate::Db[migrate_aeolus_database]/Exec[migrate_rails_database]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Conductor::Login[admin]/Web_request[admin-conductor-login]/post: post changed '' to 'https://localhost/conductor/user_session' notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: post changed '' to 'https://localhost/conductor/hardware_profiles' notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: % Total % Received % Xferd Average Speed Time Time Time Current notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: Dload Upload Total Spent Left Speed notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Rhevm/Exec[/sbin/service iwhd restart]/returns: executed successfully notice: /Stage[main]/Aeolus::Deltacloud::Core/Exec[deltacloud-core-startup-wait]/returns: executed successfully notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Conductor::Provider[rhevm]/Web_request[provider-rhevm]/post: post changed '' to 'https://localhost/conductor/providers' notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Conductor::Provider::Account[rhevm]/Web_request[provider-account-rhevm]/post: post changed '' to 'https://localhost/conductor/provider_accounts' notice: /Stage[main]/Aeolus::Profiles::Rhevm/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]/post: post changed '' to 'https://localhost/conductor/logout' notice: Finished catalog run in 9.76 seconds