Bug 1348267
| Summary: | 500 Internal Server Error when virt-who attempts to post Hypervisors to Satellite 6.1.9 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Wilson Harris <wharris> | ||||
| Component: | Candlepin | Assignee: | Chris Snyder <csnyder> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Perry Gagne <pgagne> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | Unspecified | CC: | abalakht, bbuckingham, bcourt, bkearney, csnyder, ehelms, hsun, jcallaha, kbidarka, mlele, mmccune, mstead, ovasik, sghai, wharris, xdmoon | ||||
| Target Milestone: | Unspecified | Keywords: | Reopened, Triaged | ||||
| Target Release: | Unused | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | candlepin-0.9.54.7-1,virt-who-0.17-10 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1349491 1405620 (view as bug list) | Environment: | |||||
| Last Closed: | 2018-02-21 16:54:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1361491 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Wilson Harris
2016-06-20 15:42:07 UTC
It seems to be problem in the Satellite itself, as it replies with error code 500: java.lang.NullPointerException 4821 org.candlepin.common.exceptions.mappers.RollbackExceptionMapper.toResponse(RollbackExceptionMapper.java:55) 4822 org.candlepin.common.exceptions.mappers.RollbackExceptionMapper.toResponse(RollbackExceptionMapper.java:36) Any chance of getting a copy of the candlepin logs after reproducing with DEBUG logging turned on? Debug logging in candlepin that is. You can do this by updating the candlepin config file and restarting sat6 services. log4j.logger.org.candlepin=DEBUG Would also help to get the log of the host/guest mapping that is sent from virt-who to candlepin. This is happening because candlepin Looking at the logs in the customer case, This is happening because a guest id of "" is being reported to candlepin, which has a not-null constrain on that field.
Looking at the logs in the customer's sos report, you can see that when virt-who checks in with candlepin, it is passing GuestId with an empty guestId attribute.
{
"guestId": "",
"state": 5,
"attributes": {
"active": 0,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
}
virt-who should not be reporting these guests. Candlepin should also reject any reported guest with a null/empty guestId attibute.
This has been fixed in virt-who [1] and will be available in the next released version. [1] https://github.com/virt-who/virt-who/commit/208b7d5afa8ef12b6b0fabe5e9d5e28b0f638f1d The only workaround right now that wouldn't require updating either candlepin or virt-who, would be to ensure that esx does not have any guests without a guest id. This would ensure that virt-who does not report them to candlepin. This was already fixed in candlepin-0.9.49.13+ Updating the candlepin version on the satellite should resolve this problem. Fixed in candlepin-0.9.49.13+ Check virt-who-0.17-7.el7.noarch package, the commit[1] is not built in virt-who package [1]. https://github.com/virt-who/virt-who/commit/208b7d5afa8ef12b6b0fabe5e9d5e28b0f638f1d I have two question: 1). How to get a empty guestId value? and why is the guestId value missing? 2). Radek, could you need to merge the above commit to fix this issue? Eko, I've merged that commit to virt-who-0.17-8.el7. Radek, can you answer Eko's first question? I am currently unable to get virt-who to report vm's with a blank guestId.
...
{
"guestId": "422e760d-94ff-5e5a-e75d-41507db21a73",
"state": 5,
"attributes": {
"active": 0,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
},
{
"guestId": "422e9fc7-3929-cac0-b67b-2cca66ac8a5d",
"state": 5,
"attributes": {
"active": 0,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
},
{
"guestId": "422e04e8-164a-bb7e-7b60-633c3ea63ab7",
"state": 1,
"attributes": {
"active": 1,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
},
...
Digging a bit more into this and i'm not seeing the changes in the file from the commit mentioned above. I've put the contents of the upstream file (https://github.com/virt-who/virt-who/blob/208b7d5afa8ef12b6b0fabe5e9d5e28b0f638f1d/virtwho/virt/esx/esx.py) into a local file and these are the results of a diff. [root@rhsm-qe-1 /]# diff upstream.py /usr/lib/python2.7/site-packages/virtwho/virt/esx/esx.py 254c254,257 < uuid = "%(config.network.dnsConfig.hostName)s.%(config.network.dnsConfig.domainName)s" % host --- > uuid = host['config.network.dnsConfig.hostName'] > domain_name = host['config.network.dnsConfig.domainName'] > if domain_name: > uuid = '{0}.{1}'.format(uuid, domain_name) 271,273d273 < if not vm['config.uuid'].strip(): < self.logger.debug("Guest '%s' has empty 'config.uuid' property", vm_id.value) < continue 286c286,289 < name = '%(config.network.dnsConfig.hostName)s.%(config.network.dnsConfig.domainName)s' % host --- > name = host['config.network.dnsConfig.hostName'] > domain_name = host['config.network.dnsConfig.domainName'] > if domain_name: > name = '{0}.{1}'.format(name, domain_name) I've got the latest version available in the Sat 6.2.1 Snap 1.2 tools package. [root@rhsm-qe-1 /]# rpm -qa | grep virt-who virt-who-0.17-9.el7sat.noarch Failed QA in Satellite 6.2.1 Snap 1.2 These changes are not in the current build. see above. I'm sorry but I don't know how to reproduce empty guest UUIDs in the ESX. I didn't manage to reproduce it anyhow so I just created a blind workaround for it. There are some version mismatch happening right now, the patch for this issue is applied in virt-who-0.17-8.el7, but not in the virt-who-0.17-9.el7sat. I will try to fix this mess and let you know. *** Bug 1359825 has been marked as a duplicate of this bug. *** Hello,
The Customer has updated the virt-who to virt-who-0.17-11.el7_3 and his candlepin is at candlepin-0.9.54.20-1.el7.noarch. Still he is facing the same error:
# virt-who --one-shot
2017-06-07 14:29:28,614 INFO: Using configuration "xyz-vcneter" ("esx" mode)
2017-06-07 14:29:28,615 INFO: Using reporter_id='uxvckv005a0412.cbp.dhs.gov-d110c144efcc447ebeed404404040404dfa'
2017-06-07 14:29:32,141 INFO: Sending update in hosts-to-guests mapping for config "xyz-vcneter": 11 hypervisors and 111 guests found
2017-06-07 14:29:43,834 ERROR: Response: 500
2017-06-07 14:29:43,835 ERROR: JSON parsing error: No JSON object could be decoded
2017-06-07 14:29:43,835 ERROR: Unable to send data: Communication with subscription manager failed with code 500: Server error attempting a POST to /rhsm/hypervisors?owner=test&env=Library returned status 500
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
>
> For information on the advisory, and where to find the updated files, follow the link below.
>
> If the solution does not work for you, open a new bug report.
>
> https://access.redhat.com/errata/RHSA-2018:0336
|