Created attachment 1198338 [details] engine log from approx the time of attempted upgrade Description of problem: When trying to upgrade cluster level from 3.6 to 4.0 of a medium-sized long-lived cluster (~60 running VMs ~120 defined, ~10 hosts), the upgrade fails with the following error message: Error while executing action Edit Cluster properties: Internal Engine Error Version-Release number of selected component (if applicable): 4.0.3-0.1 Additional info: Engine log with 3 upgrade attempts attached Looking at the log the following message seems to be relevant: 2016-09-06 18:12:25,604 WARN [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-91) [2fecb3a] Validation of action 'UpdateVm' failed for user .... Reasons: VAR__ACTION__UPDATE, VAR__TYPE__VM,ACTION_TYPE_FAILED_INVALID_CUSTOM_PROPERTIES_INVALID_KEYS,$MissingKeys macspoof The "UserDefinedVMProperties" engine config value looks ATM like the following: # engine-config -g UserDefinedVMProperties UserDefinedVMProperties: macspoof=(true|false) version: 3.6 UserDefinedVMProperties: version: 4.0
Arik, is this something you've seen?
(In reply to Yaniv Kaul from comment #1) Yes, this is one of problems we faced while upgrading rhev.tlv. Previously, the procedure of cluster upgrade included the following step: engine-config -s "UserDefinedVMProperties=macspoof=(true|false);another_property=regexp" --cver=3.5 As for upgrade to 4.0, the recommended way is to set non-filtering on the network instead, but similar step would work as well (danken is supposed to file a ticket to use the new configuration on rhev.tlv). I think the problem here is just the uninformative message.
(In reply to Arik from comment #2) > Previously, the procedure of cluster upgrade included the following step: where is it documented? Was it part of upgrade docs?
(In reply to Arik from comment #2) > As for upgrade to 4.0, the recommended way is to set non-filtering on the > network instead, but similar step would work as well The system is already configured with non-filtering network: # engine-config -g EnableMACAntiSpoofingFilterRules EnableMACAntiSpoofingFilterRules: false version: general This doesn't help with the cluster upgrade because we still have form VMs with the custom properties configured... Doing the following and restarting the engine resolves the issue: # engine-config -s 'UserDefinedVMProperties=macspoof=(true|false)' --cver=4.0 But yeah, the error message should suggest something that would tell me this is the place to look without digging the engine loges. Or perhaps engine-setup should warn about this, or ever better, configure this automatically. I guess this would also happen with other custom properties, not just network related ones?
So changing the scope of this bug to enhance the error reporting. Targeting to 4.0.5 since it can safe lots of headache to users.
Was able to re produce the bug on rhevm-4.0.6-0.1.el7ev.noarch , please check if this patch made this build, if so, then there some further issues, but my guess is it's just not in the build. changing status back to Modified until this is sorted out.
Verified with rhevm-4.0.6.1-0.1.el7ev.noarch according to the description. This time got the informative error message with details of specific reason.