Bug 1392706

For customer deployment detail
######################################
1. Added AWS as a provider
2. Set up AD integration  (the lab system you saw we just kept database for now but will setup IDM soon)
3. Added a cloudformation Orchestration Template  (you can fine singleLAMP on github or online my AWS guy says)
4. Created a service dialog from that template  (select template, CONFIGURATION button, create dialog from orchestration template)
5. Created a self-service service catalog from the new template/dialog item (Did NOT change last 3 fields)
6. tested by doing a successful AWS provision from that new catalog entity

Bill - Please work with Tina if needed.  My guess is that the properties they are looking for are not in $evm.root during the approval process and need to be accessed from a different object.

Created attachment 1218613 [details]
validate_request script

Please follow the example to implement validate_request method. Note this script validate all requests, not only the orchestration service.

Once it exits with MIQ_STOP, you need to go to the manual approval page to approve/deny it.

Is my understanding below correct:

evm root is "miq_request" type

therefore template will be:
   root = $evm.root['miq_request']
   template = root.template

and the format to get data from any dialog should be:

variable = root['xxxx'][:dialog]['dialog_<ELEMENT NAME here>']

where xxxx is the name of the box item inside a tab in the dialog that was used to start the process, ":dialog" references the last used service dialog, and "dialog_<ELEMENT NAME here>" is the standard convention used to point to the element name in the <> area from the last used dialog

Is this correct?


Would love to  help with DOCs (author, review, maintainer) for CFME/RH and can do DOCs for you on these things if my understanding is right as my "give back" to community effort.    :)


Next quick question:  How would I take the value from the form and set the actual expire date?  Would I just do that by setting a tag on the template (assuming my template set above is right)?

$evm.root['miq_request'] gives you the request object, so you should always use ['miq_request'].

miq_request['options'] gives you the options field of the request (miq_requests table). [:dialog] gives you all the user filled data through the service dialog. You should open the actual service dialog to find out the key for each element that you want to reference, which is in the format of 'dialog_<ELEMENT NAME>' from the options. Note that you don't see prefix dialog_ when you examine the dialog from the UI.

Below if a revised sample code. I also added a check for source type so it works only for orchestration provisioning.


$evm.log("info", "Validating request")
miq_request = $evm.root["miq_request"]
dialog_options = miq_request['options'][:dialog]

if miq_request.source.type == "ServiceTemplateOrchestration" && dialog_options['dialog_InstanceType'] != 't2.micro'
  $evm.log("info", "Should go to manual approval")
  exit MIQ_STOP
end
$evm.log("info", "auto approval validated")

To set expiration date I would suggest to do it in a post-provision step. I will get back to you after some test.

Sample code to set retirement date in method Cloud/Orchestration/Provisioning/StateMachines/Methods/postprovision

################################
$evm.log("info", "Starting Orchestration Post-Provisioning")

task = $evm.root["service_template_provision_task"]
service = task.destination
stack = service.orchestration_stack

service.post_provision_configure

# set the retirement dates for provisioned VMs after post_provision_configure
days = task['options'][:dialog]['dialog_Retirementdays']

service.vms.each do |v|
  v.retires_on = Time.now + days
end

# set the retirement date for the service itself. 

service.retires_on = Time.now + days
#################################

Note: for AWS you don't need to set retirement dates for both the service and the VMs. When a service is retired, all the VMs with it will be retired.

Passing questions from customer to engineering.
##############################################

what is the "options" field of a request?  How does CFME create that?  what
does it get and from where to get a "options" field?  Are there other
plausable things that can fit there besides the options keyword?  Do you
happen to have and examples of those things, what they are, and where they
come from?

why would I want the  ServiceTemplateOrchestration  check?  I thought
broader I could use in more dialogue was better?

Curious
question also..... Why only for service dialogue id it so easy to create a
dialog?  Is there a reason they are separate and the "dialogue builder" can't
build ANY dialog which you have a choice to see in text like prov dialogues
or gui like service?

For the question about "Dialog Builder" for the provisioning dialogs, these are older dialogs from when the product was first created and there are two reasons why we do not have a UI tool to modify them:

1) We are moving away from these dialogs and eventually want to replace them with the Service Dialogs.  Therefore we are not putting resources into building new features for them.  (We maintain and perform minor updates when needed.)

2) The Provisioning dialogs have a static set of fields that are available and cannot be dynamically expanded, unlike the service dialogs.  So each new field is manually added and configured in the dialog as well as with the code.  A UI here would be limited in value at this time.

It is not require to check for ServiceTemplateOrchestration. The sample code illustrates how to limit the scope by the type if it is desired.

"options" field is meant to be more internally used. The content varies per service type. For more information about the request, or anything else about automate, there is this book for reference: Mastering CloudForms Automation: An Essential Guide for Cloud Administrators.

We seem to have an issue with post prov....

we wanted to view the stack outputs.  The template has this section:


  "Outputs" : {
    "PrivateIP" : {
      "Value" : { "Fn::GetAtt" : [ "StandaloneServer" , "PrivateIp" ] },
      "Description" : "Private IP"
    },
    "PublicIP" : {
      "Value" : { "Fn::GetAtt" : [ "StandaloneServer" , "PublicIp" ] },
      "Description" : "Public IP"
    },
    "URL" : {
      "Value" : { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "StandaloneServer" , "PublicIp" ] }] ] },
      "Description" : "HTTP URL"
    }
  }



When we clone:

/cloud/orchestration/provisioning/statemachines/methods/postprovision


and use this (just uncommented and requested stack output):

$evm.log("info","========================================KSU")   <=============== easy way to find spot in log

def dump_stack_outputs(stack)
  $evm.log("info", "Outputs from stack #{stack.name}")
  stack.outputs.each do |output|
    $evm.log("info", "Key #{output.key}, value #{output.value}")
  end
end

$evm.log("info", "Starting Orchestration Post-Provisioning")


task = $evm.root["service_template_provision_task"]
service = task.destination
stack = service.orchestration_stack
dump_stack_outputs(stack) <======================= to dump stack options
service.post_provision_configure



The log just has this:

[----] I, [2016-11-09T15:02:31.461400 #3521:57cf3bc]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod postprovision> ========================================KSU
[----] I, [2016-11-09T15:02:31.463010 #3521:57cf3bc]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod postprovision> Starting Orchestration Post-Provisioning
[----] I, [2016-11-09T15:02:31.468291 #3521:57cf3bc]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod postprovision> Outputs from stack RHELLAMP258PM
[----] I, [2016-11-09T15:02:31.559768 #3521:54f998]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod [/KSU/Cloud/Orchestration/Provisioning/StateMachines/Methods/postprovision]> Ending
[----] I, [2016-11-09T15:02:31.559888 #3521:54f998]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) Method exited with rc=MIQ_OK
[----] I, [2016-11-09T15:02:31.560322 #3521:54f998]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) Followed  Relationship [miqaedb:/Cloud/Orchestration/Provisioning/StateMachines/Methods/PostProvision#create]


Where is the stack option output?   We need the PublicIP and PrivateIP so that we can then write code to make a htttp put to the client's DNS to set the name/IP entries in their DNS.

Steven - This is a separate issue from what was initially reported in this ticket.

Please open a support ticket and we can address the new issue there.

Thanks

I am using this also as a test of my understanding of how to get into in and out/  This is some of what I came up with based on my understanding:


$evm.log("info","******************************************* KSU2")
miq_request = $evm.root['miq_request']
testIP1 = miq_request['Outputs']['PublicIP']
vm = $evm.root['miq_request']['vm']
testip2 = vm['ip']
$evm.log("info","Is this a public IP? #{testIP1}")
$evm.log("info","Is this a VM? #{vm}")
$evm.log("info","Is this possibly the VM IP? #{testIP2}")

Steven - your code will not work.

In the first attempt, please try adjust the code sequence:

service = task.destination
service.post_provision_configure
stack = service.orchestration_stack
dump_stack_outputs(stack)

If it still does not work, try force a reload:

dump_stack_outputs($evm.vmdb("orchestration_stacks").find(stack.id))


As Greg suggested above, if this quick fix does not work, please open another support ticket.

Update from customer
############################
Thanks.  I have gotten this to work:

$evm.log("info","========================================KSU")

def dump_stack_outputs(stack)
  $evm.log("info", "Outputs from stack #{stack.name}")
  stack.outputs.each do |output|
    $evm.log("info", "Key #{output.key}, value #{output.value}")
  end
end

$evm.log("info", "Starting Orchestration Post-Provisioning")

task = $evm.root["service_template_provision_task"]
service = task.destination
service.post_provision_configure
stack = service.orchestration_stack
$evm.log("info","***DUMP START***")
dump_stack_outputs(stack)
$evm.log("info","***DUMP STOP***")


This broke though but fortunately I didn't need it:

dump_stack_outputs($evm.vmdb("orchestration_stacks").find(stack.id))

With this error in the lag:

[----] E, [2016-11-09T17:05:34.528367 #3521:60d3440] ERROR -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> The following error occurred during method evaluation:
[----] E, [2016-11-09T17:05:34.528971 #3521:60d3440] ERROR -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision>   NameError: uninitialized constant
MiqAeMethodService::MiqAeServiceOrchestrationStacks
[----] E, [2016-11-09T17:05:34.530171 #3521:60d3440] ERROR -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision>   (druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/inflector/methods.rb:270:in
`const_get'
(druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/inflector/methods.rb:270:in
`block in constantize'
(druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/inflector/methods.rb:266:in
`each'
(druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/inflector/methods.rb:266:in
`inject'
(druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/inflector/methods.rb:266:in
`constantize'
(druby://127.0.0.1:40917)
/opt/rh/cfme-gemset/gems/activesupport-5.0.0.1/lib/active_support/core_ext/string/inflections.rb:66:in
`constantize'
(druby://127.0.0.1:40917)
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_service/miq_ae_service_model_legacy.rb:96:in
`service_model_lookup'
(druby://127.0.0.1:40917)
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_service/miq_ae_service_vmdb.rb:11:in
`rescue in service_model'
(druby://127.0.0.1:40917)
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_service/miq_ae_service_vmdb.rb:9:in
`service_model'
(druby://127.0.0.1:40917)
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_service/miq_ae_service_vmdb.rb:4:in
`vmdb'
(druby://127.0.0.1:40917)
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1624:in
`perform_without_block'
(druby://127.0.0.1:40917)
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1584:in `perform'
(druby://127.0.0.1:40917)
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1657:in `block (2 levels)
in main_loop'


Fortunately I did get this in the log which is what I needed:

[----] I, [2016-11-09T17:05:34.417638 #3521:596e31c]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> ========================================KSU
[----] I, [2016-11-09T17:05:34.418945 #3521:596e31c]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> Starting Orchestration Post-Provisioning
[----] I, [2016-11-09T17:05:34.484235 #3521:596e31c]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> ***DUMP START***
[----] I, [2016-11-09T17:05:34.485746 #3521:596e31c]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> Outputs from stack RUBY-PART2
[----] I, [2016-11-09T17:05:34.505816 #25541:54f998]  INFO -- :
Instantiating
[/System/Process/Event?EventStream%3A%3Aevent_stream=1000000000768&MiqEvent%3A%3Amiq_event=1000000000768&MiqServer%3A%3Amiq_server=1000000000001&User%3A%3Auser=1000000000001&VmOrTemplate%3A%3Avm=1000000000015&ae_state=CheckPreRetirement&ae_state_previous=---%0A%22%2FManageIQ%2FCloud%2FVM%2FRetirement%2FStateMachines%2FVMRetirement%2FDefault%22%3A%0A%20%20ae_state%3A%20CheckPreRetirement%0A%20%20ae_state_retries%3A%2013%0A%20%20ae_state_started%3A%202016-11-09%2021%3A51%3A01%20UTC%0A&ae_state_retries=13&ae_state_started=2016-11-09%2021%3A51%3A01%20UTC&event_stream_id=1000000000768&event_type=request_vm_retire&miq_event_id=1000000000768&object_name=Event&retirement_initiator=user&type=ManageIQ%3A%3AProviders%3A%3AAmazon%3A%3ACloudManager%3A%3AVm&userid=admin&vm_id=1000000000015&vmdb_object_type=vm]
[----] I, [2016-11-09T17:05:34.517834 #3521:60d3440]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> Key PrivateIP, value 10.134.0.39
[----] I, [2016-11-09T17:05:34.520514 #3521:60d3440]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> Key PublicIP, value 54.157.49.77
[----] I, [2016-11-09T17:05:34.522779 #3521:60d3440]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> Key URL, value http://54.157.49.77
[----] I, [2016-11-09T17:05:34.523541 #3521:596e31c]  INFO -- :
Q-task_id([service_template_provision_task_1000000000020]) <AEMethod
postprovision> ***DUMP STOP***



Now I am starting on the following:
adding to the schema these new attributes:
PrivateIP - string
PublicIP - string
and will attempt to try to write the code to assign local variables to
those IP values and then using $ovm.object write those values to the schema
attributes.

This is what I have so far:

task = $evm.root["service_template_provision_task"]
service = task.destination
service.post_provision_configure
stack = service.orchestration_stack
PubIP = stack.outputs['PublicIP']
PrvIP = stack.outputs['PrivateIP']
$evm.object['PublicIP'] = PubIP
$evm.object['PrivateIP'] = PrvIP
$evm.log("info","Private IP is: #{PrvIP}")
$evm.log("info","Public IP is: #{PubIP}")

If my concepts are not right:
1. stack object run with outputs method returns key/value hash
2. addind the key name like this ---> ['KeyNameHere']   returns the value
in the hash for that key
3. to "push" a value into the instance you use
$evm.ojject['NameOfAttributeHere']  on left of "=" and LOCAL METHOD
mariable on right side of "="

Please let me know.   Thx.

BTW: If you could,  Do you have any recommended (what do you use to look
these up?) easy lookup /reference links to available methods with readable
brief descriptions of what the method does and maybe example or 2 listed
top to bottom by "level" where a level is something like $evm, $evm.root,
$evm.object, etc. going from the top "$evm" and stepping one level by level
to the right with each method, object, etc in the "$evm.*" line being a
"level"???   (any recommended cheat sheet links you have found helpful
would be GREAT too!)

Thx again!

Good to know that dump_stack_outputs is working.

You cannot use stack.outputs['PublicIP'] to retrieve outputs. stack.outputs is an array of OrchestrationStackOutput objects. You have to iterate the array and use output.key and output.value to access the data. Here is the same code that may be of your interest.

stack.outputs.each |output|
  pubIP = output.value if output.key == 'PublicIP'
  prvIP = output.value if output.key == 'PrivateIP'
end

I don't know why you want to write code like 
$evm.object['PublicIP'] = PubIP

The value you push to $evm.object does not persist through sessions.

I don't have a cheat sheet. Please reference the book for details how to traverse $evm.

Update from customer
###########################
Bill Wei,

Regarding:
The value you push to $evm.object does not persist through sessions.

Are you referring to a session as the life of the request to the end state of provisioning at which point $evm environment is gone?  That I believe I know and if needed I would use tags so that at some later point/request (like add/remove/change VM request) we can use the persistent values tagged to a VM/instance.  Am I understanding that part OK?

I have been reading through the book but is my understanding wrong that you can set an $evm.object in a method and then the child instance/method can then see and use it?  Am I just setting it wrong?  The reason I was doing the below is because somehow the instances have the name attributes populated in the gui:compute-->cloud-->instances-->click instance the IP Address field is empty for some reason.  Although when Sat6 creates an instance  via AWS as a compute resource the IP address value is in there.  (sat6 is setup as a provider btw)

Please see attached screenshot.  What we are doing is this in post provision:

task = $evm.root["service_template_provision_task"]
service = task.destination
service.post_provision_configure
stack = service.orchestration_stack
$evm.log("info","***DUMP START***")
dump_stack_outputs(stack)  <=== function
$evm.log("info","***DUMP STOP***")
ip = stack.outputs['PrivateIP']
dns = stack.outputs['PrivateDNS']
$evm.log("info","Postprovision method #{ip}->#{dns}")
$evm.object['PrivateIP'] = ip
$evm.object['PrivateDNS'] = dns


then we added an instance/method AddHostRecord as a next state machine step:

#KSU AddHostRecord
# based on CFME Automate Method: Infoblox_ReclaimPAddress
#
###################################
begin
  # Method for logging
  def log(level, msg, update_message=false)
    $evm.log(level, "#{msg}")
    $evm.root['miq_provision'].message = "#{msg}" if $evm.root['miq_provision'] && update_message
  end

  # dump_root
  def dump_root()
    log(:info, "Root:<$evm.root> Begin $evm.root.attributes")
    $evm.root.attributes.sort.each { |k, v| log(:info, "Root:<$evm.root> Attribute - #{k}: #{v}")} if $evm.root
    log(:info, "Root:<$evm.root> End $evm.root.attributes")
    log(:info, "")
  end

  def call_infoblox(action, ref='record:host', content_type=:xml, body=nil )
    require 'rest_client'
    require 'xmlsimple'
    require 'json'

    servername = nil
    servername ||= $evm.object['servername']
    username = nil
    username ||= $evm.object['username']
    password = nil
    password ||= $evm.object.decrypt('password')
    api_version = nil
    api_version ||= $evm.object['api_version']
    
    # if ref is a url then use that one instead
    url = ref if ref.include?('http')
    url ||= "https://#{servername}/wapi/#{api_version}/"+"#{ref}"

    params = {
        :method=>action,
        :url=>url,
        :user=>username,
        :password=>password,
        :headers=>{ :content_type=>content_type, :accept=>:xml }
    }
    content_type == :json ? (params[:payload] = JSON.generate(body) if body) : (params[:payload] = body if body)
    log(:info, "Calling -> Infoblox: #{url} action: #{action} payload: #{params[:payload]}")
    response = RestClient::Request.new(params).execute
    log(:info, "Inspecting response: #{response.inspect}")
    raise "Failure <- Infoblox Response: #{response.code}" unless response.code == 200 || response.code == 201 || response.code == 202 || response.code == 203

    # use XmlSimple to convert xml to ruby hash
    response_hash = XmlSimple.xml_in(response)
    log(:info, "Inspecting response_hash: #{response_hash.inspect}")
    return response_hash
  end

In a normal workflow it may work if use $evm.object to pass attributes, but it gets lost if a retry is happening. The better way is to use $evm.set_state_var and $evm.get_state_var

$evm.set_state_var('PrivateIP', privateIP)
and later you can read it back
$evm.get_state_var('PrivateIP')

Again ip = stack.outputs['PrivateIP'] will not work. Please follow the sample code in comment #17.

Please reference http://talk.manageiq.org/t/control-auto-or-manual-approval-of-a-service-request-through-validate-request/1866 for document purpose.

Moving to 'NEW' while assigned to the default assignee.

Thank you for raising this bug.

We have evaluated this request, and while we recognize that it is a valid request for the documentation, we do not expect this to be implemented in the product in the foreseeable future. We are therefore closing this out as WONTFIX. 

If you have any concerns about this, please feel free to contact Andrew Dahms.