Bug 728019

Marc,
This is has been reported several times in various incantations.  In my own personal experience I have not seen this issue w/ the production 0.3.0 code, however I'm sure it can happen based on config/hardware diff. 

Can you please execute, aeolus-cleanup -v ; aeolus-configure -v; several times to see how consistent the issue is.

Thanks

I am still running into this. This has happened off and on for us in our lab, but the last 3 installs for me have done this. I am using RHEL 6.1 x86_64, and our redeploy does a complete kickstart, then runs aeolus-configure with parameters enabled for RHEV 2.2 an vSphere. First I get the same errors, then an aeolus-cleanup and aeolus-configure both run clean, and all providers are present.

Hardware:
IBM BladeCenter HS22

aeolus-all 0.3.0-2.el6
aeolus-conductor 0.3.0-3.el6
aeolus-conductor-daemons 0.3.0-3.el6 
aeolus-configure 2.0.1-2.el6

err: /Stage[main]/Aeolus::Conductor::Seed_data/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: change from notrun to 0 failed: /usr/bin/curl --proxy '' -X PUT http://localhost:9090/templates returned 7 instead of one of [0] at /usr/share/aeolus-configure/modules/aeolus/manifests/iwhd.pp:63
err: /Stage[main]/Aeolus::Rhevm/Aeolus::Conductor::Provider[rhevm]/Web_request[provider-rhevm]/post: change from  to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404, 
err: /Stage[main]/Aeolus::Conductor::Seed_data/Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Web_request[provider-ec2-us-west-1]/post: change from  to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404, 
err: /Stage[main]/Aeolus::Conductor::Seed_data/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: change from  to https://localhost/conductor/hardware_profiles failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404, 
err: /Stage[main]/Aeolus::Vmware/Aeolus::Conductor::Provider[vsphere]/Web_request[provider-vsphere]/post: change from  to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404, 
err: /Stage[main]/Aeolus::Conductor::Seed_data/Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Web_request[provider-mock]/post: change from  to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404, 
err: /Stage[main]/Aeolus::Conductor::Seed_data/Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Web_request[provider-ec2-us-east-1]/post: change from  to https://localhost/conductor/providers failed: An exception was raised when invoking web request: Invalid HTTP Return Code: 404,

so just to document some prior issues w/ providers getting setup via configure..

https://bugzilla.redhat.com/show_bug.cgi?id=693369
https://bugzilla.redhat.com/show_bug.cgi?id=716948
https://bugzilla.redhat.com/show_bug.cgi?id=715506

Using a kickstart to install cloud engine should work, but was never tested / verified by QE.   Again I think that *should* work, but I'm not surprised by the issues. 

To give some sanity to whom ever is install Cloud Engine, can we verify that you are having aeolus-configure provider errors if the install is done manually (outside of a kickstart)

1. install RHEL6.1 x86_64
2. subscribe to the appropriate rhn channels
3. yum -y install aeolus-all
4. configure /etc/aeolus-configure/nodes/default_configure default_cleanup for your rhevm env. 
5. run aeolus-configure -v

Anything outside of the above steps have not been cleared by QE.

one other note from QE.. due to the inconsistent nature of adding providers/hwp via REST calls to localhost we have found that rerunning aeolus-cleanup/configure the best and easiest work around.

Some additional bugs that I believe are caused by the same root issue.

https://bugzilla.redhat.com/show_bug.cgi?id=717843
https://bugzilla.redhat.com/show_bug.cgi?id=725328

Our kickstart is just a standard Satellite install, followed by a post script that calls this script:

http://refarch.cloud.lab.eng.bos.redhat.com/pub/scripts/post_install_configuration_scripts/cf-newinstall-beta-rhn.sh

We have since stopped calling it from post, and instead run this by hand after an install. So far I have only seen this happen once, and a cleanup and configure worked fine.

Problems w/ underlying aeolus components often manifest themselves through configure. In this case there seems to be a problem w/ conductor (or perhaps a conductor dependency not yet initialized) resulting in this failed request.

Could you paste any errors from the conductor log (/var/log/aeolus-conductor/{thin|rails}.log as well.

making sure all the bugs are at the right version for future queries

ping? Is this still an issue? If so can we get the aforementioned logs? If not can this be closed out?

Thanks.

I do not have access to this environment any longer. I asked Steve Reichard and Scott Collier to follow-up on this, as it was their equpiment I original saw this on.

With updated versions of aeolus, this problem has not been reproduced.

spr

OK in that case, moving this to ON_QA

201112082145:59.255 - FINE: ssh root.lab.eng.bos.redhat.com aeolus-configure -d -v -p mock, ec2, vsphere, rhevm (com.redhat.qe.tools.SSHCommandRunner.run)
201112082147:08.255 - FINE: Stdout: 
Launching aeolus configuration recipe...
[0;32minfo: Loading facts in oauth[0m
[0;32minfo: Applying configuration version '1323398760'[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/File[/var/tmp/imagefactory-mock]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/Service[libvirtd]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/etc/rsyslog.d]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/etc/rsyslog.d/aeolus.conf]/ensure: defined content as '{md5}2d45434a072b4f9d1518ce026b92c547'[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/etc/rsyslog.d/aeolus.conf]: Scheduling refresh of Service[rsyslog][0m
[0;36mnotice: /Stage[main]/Postgres::Server/Exec[pginitdb]/returns: executed successfully[0m
[0;32minfo: /Stage[main]/Postgres::Server/Exec[pginitdb]: Scheduling refresh of Service[postgresql][0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/File[/root/.aeolus-cli]/ensure: defined content as '{md5}d25602b6f13216cbe64451d1a911bae6'[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/Augeas[imagefactory.conf]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/usr/share/aeolus-conductor/config/initializers/secret_token.rb]/ensure: defined content as '{md5}702d2e1e47e03070749491552934825b'[0m
[0;32minfo: FileBucket adding {md5}5f984898775145f23d80d2322d360373[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/usr/share/aeolus-conductor/config/settings.yml]: Filebucketed /usr/share/aeolus-conductor/config/settings.yml to puppet with sum 5f984898775145f23d80d2322d360373[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/usr/share/aeolus-conductor/config/settings.yml]/content: content changed '{md5}5f984898775145f23d80d2322d360373' to '{md5}b4e5215249fe04e825d8c9bc6bd2e7d8'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/rsyslog]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/rsyslog/work]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Service[rsyslog]: Triggered 'refresh' from 1 events[0m
[0;36mnotice: /Stage[main]/Aeolus::Deltacloud::Core/Service[deltacloud-core]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Deltacloud::Core/Exec[deltacloud-core-startup-wait]/returns: executed successfully[0m
[0;32minfo: FileBucket adding {md5}75d10aa75d5abb0893682de8622f3061[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]: Filebucketed /var/lib/pgsql/data/pg_hba.conf to puppet with sum 75d10aa75d5abb0893682de8622f3061[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]/content: content changed '{md5}75d10aa75d5abb0893682de8622f3061' to '{md5}2d3f1f2e30c9b188961b1acd3ade3b8b'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]/owner: owner changed 'postgres' to 'root'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]/group: group changed 'postgres' to 'root'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]/mode: mode changed '600' to '644'[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]: Scheduling refresh of Service[postgresql][0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]: Scheduling refresh of Service[postgresql][0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]: Scheduling refresh of Service[postgresql][0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/File[/var/lib/pgsql/data/pg_hba.conf]: Scheduling refresh of Service[postgresql][0m
[0;36mnotice: /Stage[main]/Postgres::Server/Service[postgresql]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Postgres::Server/Service[postgresql]: Triggered 'refresh' from 5 events[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Postgres::User[aeolus]/Exec[create_aeolus_postgres_user]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: Using gem require instead of bundler[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Create::Db[create_aeolus_database]/Exec[create_rails_database]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Migrate::Db[migrate_aeolus_database]/Exec[migrate_rails_database]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Seed::Db[seed_aeolus_database]/Exec[seed_rails_database]/returns: Using gem require instead of bundler[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Seed::Db[seed_aeolus_database]/Exec[seed_rails_database]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Rails::Seed::Db[seed_aeolus_database]/File[/var/lib/aeolus-conductor/production.seed]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: Using gem require instead of bundler[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: User admin registered[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[create_site_admin_user]/returns: executed successfully[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: Using gem require instead of bundler[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: Granting administrator privileges for admin...[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: executed successfully[0m
[0;32minfo: FileBucket adding {md5}1d1cc4ee732cdfd775c2bac6096e18fc[0m
[0;32minfo: /Stage[main]/Aeolus::Iwhd/File[/etc/iwhd/users.js]: Filebucketed /etc/iwhd/users.js to puppet with sum 1d1cc4ee732cdfd775c2bac6096e18fc[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/File[/etc/iwhd/users.js]/content: content changed '{md5}1d1cc4ee732cdfd775c2bac6096e18fc' to '{md5}bea0715b77d1e746bafdf10691013817'[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/File[/etc/iwhd/users.js]/mode: mode changed '600' to '700'[0m
[0;32minfo: FileBucket adding {md5}132980451cabf2b363c33e2e0c345ae9[0m
[0;32minfo: /Stage[main]/Aeolus::Iwhd/File[/etc/init.d/iwhd]: Filebucketed /etc/init.d/iwhd to puppet with sum 132980451cabf2b363c33e2e0c345ae9[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/File[/etc/init.d/iwhd]/content: content changed '{md5}132980451cabf2b363c33e2e0c345ae9' to '{md5}6e6aa5a83a15929d7d3ef426475a5f1e'[0m
[0;32minfo: FileBucket adding {md5}2ce42df01efc8b1745097faa7b83d9e5[0m
[0;32minfo: /Stage[main]/Ntp::Client/File[/etc/ntp.conf]: Filebucketed /etc/ntp.conf to puppet with sum 2ce42df01efc8b1745097faa7b83d9e5[0m
[0;36mnotice: /Stage[main]/Ntp::Client/File[/etc/ntp.conf]/content: content changed '{md5}2ce42df01efc8b1745097faa7b83d9e5' to '{md5}2dfb9420ebf32c7d97f86526fe6c21e8'[0m
[0;32minfo: /Stage[main]/Ntp::Client/File[/etc/ntp.conf]: Scheduling refresh of Service[ntpd][0m
[0;32minfo: FileBucket adding {md5}32fdc5cdf3f181e7b6987410cf607d15[0m
[0;32minfo: /Stage[main]/Ntp::Client/File[/etc/ntp/step-tickers]: Filebucketed /etc/ntp/step-tickers to puppet with sum 32fdc5cdf3f181e7b6987410cf607d15[0m
[0;36mnotice: /Stage[main]/Ntp::Client/File[/etc/ntp/step-tickers]/content: content changed '{md5}32fdc5cdf3f181e7b6987410cf607d15' to '{md5}2dfb9420ebf32c7d97f86526fe6c21e8'[0m
[0;32minfo: /Stage[main]/Ntp::Client/File[/etc/ntp/step-tickers]: Scheduling refresh of Service[ntpd][0m
[0;36mnotice: /Stage[main]/Ntp::Client/Service[ntpd]/enable: enable changed 'false' to 'true'[0m
[0;36mnotice: /Stage[main]/Ntp::Client/Service[ntpd]: Triggered 'refresh' from 2 events[0m
[0;36mnotice: /Stage[main]/Apache/Service[httpd]/ensure: ensure changed 'stopped' to 'running'[0m
[0;32minfo: FileBucket adding {md5}8129c1ef2030ef66c0437b5d49115c3d[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/Apache::Site[aeolus-conductor]/File[/etc/httpd/conf.d/aeolus-conductor.conf]: Filebucketed /etc/httpd/conf.d/aeolus-conductor.conf to puppet with sum 8129c1ef2030ef66c0437b5d49115c3d[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Apache::Site[aeolus-conductor]/File[/etc/httpd/conf.d/aeolus-conductor.conf]/content: content changed '{md5}8129c1ef2030ef66c0437b5d49115c3d' to '{md5}78aad09a0e062bd08d34b9905d76d5ce'[0m
[0;32minfo: /Stage[main]/Aeolus::Conductor/Apache::Site[aeolus-conductor]/File[/etc/httpd/conf.d/aeolus-conductor.conf]: Scheduling refresh of Exec[reload-apache][0m
[0;36mnotice: /Stage[main]/Apache/Exec[reload-apache]: Triggered 'refresh' from 1 events[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Service[conductor-dbomatic]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Conductor/Service[aeolus-conductor]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Login[admin]/Web_request[admin-conductor-login]/post: post changed '' to 'https://localhost/conductor/user_session'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Provider[mock]/Web_request[provider-mock]/post: post changed '' to 'https://localhost/conductor/providers'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Provider::Account[mockuser]/Web_request[provider-account-mockuser]/post: post changed '' to 'https://localhost/conductor/providers/0/provider_accounts'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Hwp[hwp1]/Web_request[hwp-hwp1]/post: post changed '' to 'https://localhost/conductor/hardware_profiles'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Conductor::Logout[admin]/Web_request[admin-conductor-logout]/post: post changed '' to 'https://localhost/conductor/logout'[0m
[0;32minfo: FileBucket adding {md5}9536dca919c71073ef0b836e84a66663[0m
[0;32minfo: /Stage[main]/Aeolus::Image-factory/File[/etc/qpidd.conf]: Filebucketed /etc/qpidd.conf to puppet with sum 9536dca919c71073ef0b836e84a66663[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/File[/etc/qpidd.conf]/content: content changed '{md5}9536dca919c71073ef0b836e84a66663' to '{md5}3b762fe02ab4c355304d933bad24aee0'[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/Service[qpidd]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Image-factory/Service[imagefactory]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/File[/data]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/File[/data/db]/ensure: created[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/Service[mongod]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Iwhd/Service[iwhd]/ensure: ensure changed 'stopped' to 'running'[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns:                                  Dload  Upload   Total   Spent    Left  Speed[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[0m
[0;36mnotice: /Stage[main]/Aeolus::Profiles::Mock/Aeolus::Create_bucket[aeolus]/Exec[create-bucket-aeolus]/returns: executed successfully[0m
[0;32minfo: Creating state file /var/lib/puppet/state/state.yaml[0m
[0;36mnotice: Finished catalog run in 66.43 seconds[0m

[root@qeblade31 yum.repos.d]# rpm -qa | grep aeolus
aeolus-conductor-daemons-0.7.0-4.el6.noarch
aeolus-conductor-doc-0.7.0-4.el6.noarch
rubygem-aeolus-cli-0.2.0-3.el6.noarch
rubygem-aeolus-image-0.2.0-1.el6.noarch
aeolus-all-0.7.0-4.el6.noarch
aeolus-configure-2.4.0-3.el6.noarch
aeolus-conductor-0.7.0-4.el6.noarch
[root@qeblade31 yum.repos.d]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0586.html