Investigating... It seems the long hostname gets truncated somewhere inside CloudForms, as preceding logs contain the full name: [----] I, [2019-09-11T07:02:57.696204 #27430:d7f118] INFO -- : MIQ(MiqEvent#process_evm_event) target = [#<ContainerImage id: 18, tag: "v3.11.104", name: "mcpaasatdxctechnology-test-cv_openshift-rhocpinstr...", image_ref: "docker-pullable://pln-n1-rhcap1-p.env01.mcloud.ent...", container_image_registry_id: 1, ems_id: 1, last_sync_on: nil, last_scan_attempt_on: nil, digest: "sha256:1ef034a2a641aae54cd88a4111ad21f9de926 8ac79c...", registered_on: nil, architecture: nil, author: nil, command: [], entrypoint: [], docker_version: nil, exposed_ports: {}, environment_variables: {}, size: nil, created_on: "2019-08-22 22:44:14", old_ems_id: nil, deleted_on: nil, type: "ContainerImage">] [----] I, [2019-09-11T07:02:57.703581 #27430:d7f118] INFO -- : MIQ(MiqEvent#process_evm_event) Event Raised [request_containerimage_scan] [----] I, [2019-09-11T07:02:57.879582 #27430:d7f118] INFO -- : MIQ(MiqEvent#process_evm_event) Alert for Event [request_containerimage_scan] [----] I, [2019-09-11T07:02:57.880771 #27430:d7f118] INFO -- : MIQ(MiqAlert.evaluate_alerts) [request_containerimage_scan] Target: ContainerImage Name: [mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy], Id: [18] [----] I, [2019-09-11T07:02:57.924593 #27430:d7f118] INFO -- : MIQ(MiqQueue#delivered) Message id: [167651], State: [ok], Delivered in [1.424995987] seconds [----] I, [2019-09-11T07:02:58.158646 #27430:d7f118] INFO -- : MIQ(MiqQueue#m_callback) Message id: [167651], Invoking Callback with args: [:raw_scan_job_create, "ContainerImage", 18, "system", "mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy", "ok", "Message delivered successfully", "#<MiqAeEngine::MiqAeWorkspaceRuntime:0x000000000aad09d0 @readonly=false, @nodes=[#<MiqAeEngine::MiqAeObject:0x000000000acbccf8 @workspace=#<MiqAeEngine::MiqAeWorkspaceRuntime:0x000000000aad09d0 ...>, @namespace=\"ManageIQ/System\", @klass=\"Process\", @instance=\"Event\", @attributes={\"event_stream_id\"=>\"21930\", \"event_type\"=>\"request_containerimage_scan\", \"miq_event_id\"=>\"21930\", \"object_name\"=>\"Event\", \"vmdb_object_type\"=>\"container_image\", \"container_image_id\"=>\"18\", \"container_image\"=>#<MiqAeServi..."] [----] I, [2019-09-11T07:02:58.455688 #27430:d7f118] INFO -- : Job created: guid: [d5e2e767-5ec8-4a65-a009-7053bcf2cd30], userid: [system], name: [Container Image Analysis: 'mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy'], target class: [ContainerImage], target id: [18], process type: [ManageIQ::Providers::Kubernetes::ContainerManager::Scanning::Job], server id: [], zone: [default] ... [----] I, [2019-09-11T07:03:08.247831 #27446:d7f118] INFO -- : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::Scanning::Job#start) Creating pod [management-infra/manageiq-img-scan-d5e2e] to analyze docker image [pln-n1-rhcap1-p.env01.mcloud.entsvcs.net:5000/mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy@sha256:1ef034a2a641aae54cd88a4111ad21f9de9268ac79c0d6ca9cf98a72b39da284] [{"apiVersion":"v1","kind":"Pod","metadata":{"name":"manageiq-img-scan-d5e2e","namespace":"management-infra","labels":{"name":"manageiq-img-scan-d5e2e","manageiq.org":"true"},"annotations":{"manageiq.org/hostname":"prodz8cla01","manageiq.org/guid":"f69f1341-e3d0-4927-b4fa-7b6a593dd462","manageiq.org/image":"pln-n1-rhcap1-p.env01.mcloud.entsvcs.net:5000/mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy@sha256:1ef034a2a641aae54cd88a4111ad21f9de9268ac79c0d6ca9cf98a72b39da284","manageiq.org/jobid":"d5e2e767-5ec8-4a65-a009-7053bcf2cd30"}},"spec":{"hostname":"mcpaasatdxctechnologytestcvopenshiftrhocpinstregopenshift3oa...","restartPolicy":"Never","containers":[{"name":"image-inspector","image":"pln-n1-rhcap1-p.env01.mcloud.entsvcs.net:5000/mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_image-inspector:2.1","imagePullPolicy":"Always","command":["/usr/bin/image-inspector","--chroot","--image=pln-n1-rhcap1-p.env01.mcloud.entsvcs.net:5000/mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy@sha256:1ef034a2a641aae54cd88a4111ad21f9de9268ac79c0d6ca9cf98a72b39da284","--scan-type=openscap","--serve=0.0.0.0:8080","--dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-r7j75/.dockercfg"],"ports":[{"containerPort":8080}],"securityContext":{"privileged":true},"volumeMounts":[{"mountPath":"/var/run/docker.sock","name":"docker-socket"},{"name":"inspector-admin-secret-inspector-admin-dockercfg-r7j75","mountPath":"/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-r7j75","readOnly":true}],"env":[{"name":"NO_PROXY","value":"pln-n1-rhcap1-p.env01.mcloud.entsvcs.net"},{"name":"HTTP_PROXY","value":"16.85.88.10:8080"}],"readinessProbe":{"initialDelaySeconds":15,"periodSeconds":5,"httpGet":{"path":"/healthz","port":8080}}}],"volumes":[{"name":"docker-socket","hostPath":{"path":"/var/run/docker.sock"}},{"name":"inspector-admin-secret-inspector-admin-dockercfg-r7j75","secret":{"secretName":"inspector-admin-dockercfg-r7j75"}}]}}] [----] E, [2019-09-11T07:03:08.353744 #27446:d7f118] ERROR -- : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::Scanning::Job#start) pod creation for [management-infra/manageiq-img-scan-d5e2e] failed: [HTTP status code 422, Pod "manageiq-img-scan-d5e2e" is invalid: spec.hostname: Invalid value: "mcpaasatdxctechnologytestcvopenshiftrhocpinstregopenshift3oa...": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')] What I don't know yet is whether these `...` truncations are purely an artifact of logging, or whether we actually sent `"hostname":"mcpaasatdxctechnologytestcvopenshiftrhocpinstregopenshift3oa..."` to kubernetes. If it's not the length, another reason that might be a problem here is the hostname contains underscores, which are not technically valid in DNS, though defacto used in some networks: mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy looking into code... https://github.com/ManageIQ/manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager/scanning/job.rb#L422-L423
Aha, the "hostname" here is not a name of any real host. It's a fake value we set to "smuggle" at least an approximate image name into the OpenSCAP report: https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/248 > If it's not the length, another reason that might be a problem here is the hostname contains underscores, which are not technically valid in DNS, though defacto used in some networks: OK, underscores are not the problem. The code drop underscores and other punctuation: mcpaasatdxctechnology-test-cv_openshift-rhocpinstreg-openshift3_oauth-proxy => mcpaasatdxctechnologytestcvopenshiftrhocpinstregopenshift3oa... The resulting length, including the ellipsis, is exactly 63, to this must be the result of the `.truncate(63)` we do. Indeed: https://guides.rubyonrails.org/active_support_core_extensions.html#truncate Working on fix to suppress ellipsis...
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/351
Should be fixed by https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/351 This was a regression introduced by enhancement https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/248 for bz 1554250. The bug was backported to gaprindashvili-3 (though for some reason I don't see it merged in any 5.9.*.* tag), so I propose backporting the fix to 5.9, 5.10, 5.11.
New commit detected on ManageIQ/manageiq-providers-kubernetes/master: https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/f47caa78871b1cd9e4786abefac9893fcbc17bf1 commit f47caa78871b1cd9e4786abefac9893fcbc17bf1 Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Wed Sep 18 06:02:56 2019 -0400 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Wed Sep 18 06:02:56 2019 -0400 Fix scanning pod fake "hostname" for long image names Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1751267 Setting "hostname" was an enhancement in https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/248 to improve image names but it introduced a regression: Pod spec could be invalid for long image names, causing scanning to not run at all. (Possibly also for weird characters in name, not sure if realistic but made code more robust to that too.) Expanded comments to clarify lowercase requirement is by Kubernetes not the DNS RFC (https://github.com/kubernetes/kubernetes/pull/39675). The RFC also says: > Host software MUST handle host names of up to 63 characters and > SHOULD handle host names of up to 255 characters so a bit ambiguous what's "valid DNS label" means but anyway Kubernetes chose 63 (some history on https://github.com/kubernetes/kubernetes/issues/4825) app/models/manageiq/providers/kubernetes/container_manager/scanning/job.rb | 8 +- spec/models/manageiq/providers/kubernetes/container_manager/scanning/job_spec.rb | 13 +- 2 files changed, 16 insertions(+), 5 deletions(-)