Description of problem: If you have a custom attribute with a space or colon in it then it will not show up in a report. Version-Release number of selected component (if applicable): 5.8.2.3.20171016155816_aaec796 How reproducible: Always Steps to Reproduce: 1. set a custom attribute on a VM with a space in the name 2. set a custom attribute on a VM with a colon in the name 3. Create a report with the VM name and the two custom attributes 4. run the report Actual results: The columns for the custom attribute will be blank. Expected results: The columns for the custom attribute should have their values. Additional info: If you open the report back up to edit it then the selected field names have been truncated to the first space or colon. The suspicion is when saving the report its not saving the full custom attribute name. Screen shots will be attached. This is blocking a client from being able to run business related reports.
Created attachment 1406256 [details] creating the report - selecting the custom attribute fields
Created attachment 1406257 [details] creating the report - custom attribute fields selected
Created attachment 1406258 [details] edit the report after save - custom attribute field names missing
Created attachment 1406260 [details] proof that custom attributes exist on a VM
Created attachment 1406261 [details] report where you can see no values for ian-test0002 which clearly has the custom attributes
*** Bug 1544493 has been marked as a duplicate of this bug. ***
I'm going to need some help from the UI team. I have a database export I can share if needed. I have a custom_attribute named "Last Backup". I followed the instructions, created a simple report based on vm and added the custom attribute and saved it. The miq_reports table entry for this report has the "Last Backup" truncated to just "Last" in the cols and col_order column. cols: --- - name - virtual_custom_attribute_Last col_order: --- - name - virtual_custom_attribute_Last From the backend miq report side, I was able to modify the miq_report object, cols and col_order values to "quote" the values with spaces and I could get it to generate a report with html values containing the correct values. In other words, my cols and col_order columns now look like this: cols: --- - name - "virtual_custom_attribute_Last Backup" col_order: --- - name - "virtual_custom_attribute_Last Backup" I could then find the report and generate the data in rails console: > MiqReport.find(123)._async_generate_table(nil) Lots of queries that looked like: CustomAttribute Load (0.2ms) SELECT "custom_attributes".* FROM "custom_attributes" WHERE "custom_attributes"."resource_id" = $1 AND "custom_attributes"."resource_type" = $2 AND "custom_attributes"."name" = $3 LIMIT $4 [["resource_id", 1000000000355], ["resource_type", "VmOrTemplate"], ["name", "Last Backup"], ["LIMIT", 1]] Note, when I did a "preview" of the report in the UI, which does something similar, the "Last Backup" was truncated to "Last" in those queries. You can tell it worked because it inserted the proper values for the vms that had values for this custom attribute: SQL (0.3ms) INSERT INTO "miq_report_result_details" ("miq_report_result_id", "data_type", "data") VALUES ($1, $2, $3) RETURNING "id" [["miq_report_result_id", 123], ["data_type", "html"], ["data", "<tr class='row0-nocursor'><td>REDACTED1</td><td>7/19/2017 8:13:50 PM</td></tr>"]] SQL (0.3ms) INSERT INTO "miq_report_result_details" ("miq_report_result_id", "data_type", "data") VALUES ($1, $2, $3) RETURNING "id" [["miq_report_result_id", 123], ["data_type", "html"], ["data", "<tr class='row1-nocursor'><td>REDACTED2</td><td>8/17/2017 10:02:25 PM</td></tr>"]] From what I can tell, it appears that the UI is truncating the values with spaces and possibly colons in several places: 1) when saving the report, cols and col_order values are truncated as virtual_custom_attribute_Last instead of "virtual_custom_attribute_Last Backup" 2) upon loading a report that I have "fixed" in the backend to properly quote these cols and col_order columns, clicking preview also truncates these values in the miq_report object that is enqueued. [----] I, [2018-03-15T17:54:35.138930 #28560:3ff197544c64] INFO -- : MIQ(MiqQueue.put) Message id: [1000096080026], id: [], Zone: [], Role: [reporting], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqReport._async_generate_table], Timeout: [3600], ... #<MiqReport id: nil, name: "last backup report", title: "last backup report", rpt_group: "Custom", rpt_type: "Custom", priority: nil, db: "ManageIQ::Providers::InfraManager::Vm", cols: ["name", "virtual_custom_attribute_Last"], include: {:custom_attributes=>{}}, col_order: ["name", "virtual_custom_attribute_Last"], headers: ["Name", "Custom Field: Last Backup"], ... {:userid=>"admin", :session_id=>"6d66601174dfdd5276949a2c4c83408c", :limit=>50, :mode=>"adhoc"}]
I really don't see how we can make this happen. The MIQ reporting system was designed for active record attributes (DB columns), which can only have non-blank characters and underscores. The best bet is to have the customer create their custom attributes to comply. To support anything else will be a major RFE, because we can't just change reporting, we would need to change the expression editor, builder, and back end evaluator. There would be other areas that would break as well. My suggestion is to document that we only support custom attributes that comply with the rails active record column naming standard.
Good point Dan. Even though the reporting backend appears to support columns with spaces in them (provided they're quoted), there are many other areas of the code that accept columns and would likely not support spaces/colons and any other non-column character. I'd be concerned that even if we get this to work for this BZ, the changes required for such things as the expression editor frontend and backend, which can be used nearly everywhere, would be very difficult.
@Dan and @Joe, If this isn't going to be supported then we need a way to set the human readable description for custom attributes. You can see from my attached screen shot (https://bugzilla.redhat.com/attachment.cgi?id=1406260) that if/when we change these attributes to not have spaces or colons then they will become a pain to read let alone a horrible eye sore compared to everything else on the VM details page which has spaces. Blue skies, Ian
@Dan and @Joe, Also, do you have a suggestion for how to go through the entire DB and update all the existing custom attributes? The client already has thousands of VMs with custom attributes with spaces and colons in them. Also, beyond just documenting it, can you update the function for setting custom attributes to error if the name is invalid? Otherwise no one is ever going to see whatever document update is done and is going to have to go through the pain of doing what we are going to have to do and update all of their attributes after they stumble upon the Bugzilla and then the documentation. If certain charters are not going to be allowed then the function should error. Blue skies, Ian
Hi Ian, I can't speak for the UI side but I believe the custom attributes are coming from vsphere or another external management system. If so, they'd need to be renamed there or we cannot report/build expressions on them. Custom attributes, for better or worse, are treated as columns and columns have always have had this restriction unfortunately. I'm not sure there's a way to fix the database side to fix the existing custom attributes if they are coming from vsphere or another management system since these invalid keys will be re-synchronized the next time we refresh this data from the ems. I'm hoping Dan knows the function for setting custom attributes. I don't know that we have this exposed anywhere in the UI. I thought 100% of custom attributes are coming from external systems and we have no way to control that data. Ian, if there are places in the UI or API you can set custom attributes with invalid names (with spaces, colons, etc.), please describe them and include screenshots if applicable so we can add validations. Thanks, Joe
@Joe, The custom attributes I am speaking of are being set by us via Automate. So we can update the automate code to use different naming scheme for future attributes but need a way to go through and update all the existing custom attributes. This done via `vm#set_custom`. Reference can be seen here, https://github.com/rhtconsulting/miq-Utilities/blob/31ce8dae60ad001d5ab1b614755bbf7cc7f3e775/Automate/RedHatConsulting_Utilities/Infrastructure/VM/Provisioning/StateMachines/Methods.class/__methods__/process_telemetry_data.rb#L44 Hence if there is a hard requirement on the naming of the custom attribute then the #set_custom function should be updated to error if an invalid name is passed. Blue skies, Ian
Hi Ian, Sorry, I am not very familiar with automate. I wasn't aware this was exposed in automate via custom_set. A script to fix this would need to query CustomAttribute.where("name like '% %'") or CustomAttribute.where("name like '%:%'") and replace the values with a column character such as underscore: _ I'll open a BZ to add validation of miq_custom_set which is called from automate's custom_set to not allow invalid characters such as space or colons.
Opened bug 1558618 so we can validate custom attributes so we don't add them with values such as space and colon. Note, this could be tricky because we can't break existing refresh / vm scanning code that creates custom attributes with spaces as they were provided by the management system.
Comment #17 mentions we'll be fixing the validations on custom attributes to not allow these types of characters in bug 1558618. We're not making that change in this bug. I think we can close this bug since we won't be fixing the reporting front end and expression backend to allow these types of characters.