Can you provide a `podman inspect` for the container created by the test script? Also, your `/etc/containers/containers.conf` or `/etc/containers/libpod.conf` would be helpful (if they exist)

Podman should be popping up that message if `podman ps` cannot find the kata-runtime binary. It looks like it's specified as `--runtime=/usr/bin/kata-runtime` in the test script (absolute path, not name). I suspect it's not in the configuration files, either.

I think we expect this error if a runtime is manually specified by path, and not included in the configuration files (subsequent Podman invocations won't be able to find the runtime unless it's handed to them via `--runtime=/usr/bin/kata-runtime`). This is probably not a necessary error, though. We can have Podman search $PATH for the given runtime and instantiate it if we try and access a container using it. Question would be if this is confusing behavior to the user, as we just pulled a runtime they did not explicitly specify via configuration or command line into their Podman command.

# podman inspect 7badedb40528
ERRO[0000] OCI Runtime kata-runtime is in use by a container, but is not available (not in configuration file or not installed) 
[
    {
        "Id": "7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f",
        "Created": "2020-07-23T17:18:40.107825641-04:00",
        "Path": "bash",
        "Args": [
            "bash"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 85385,
            "ConmonPid": 85187,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-07-23T17:18:57.601869553-04:00",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "a1f8c969978652a6d1b2dfb265ae0c6c346da69000160cd3ecd5f619e26fa9f3",
        "ImageName": "registry.access.redhat.com/ubi8:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/var/run/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/resolv.conf",
        "HostnamePath": "/var/run/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/hostname",
        "HostsPath": "/var/run/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/config.json",
        "OCIRuntime": "kata-runtime",
        "LogPath": "/var/lib/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/ctr.log",
        "LogTag": "",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f/userdata/conmon.pid",
        "Name": "crazy_hellman",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c103,c1003",
        "ProcessLabel": "system_u:system_r:container_kvm_t:s0:c103,c1003",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "BoundingCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/1da86dd5dbd9e4d52991b5846320796527b223a8bc5c770caa644281a94fd828/diff:/var/lib/containers/storage/overlay/226bfaae015f1d5712cfced3b5b628206618eaacf72f4a44d0e4084071996319/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/21df3bb6a8cf806694d8fb0ecfa991a342e9d8f2dfe1943559602dc441301fb2/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/21df3bb6a8cf806694d8fb0ecfa991a342e9d8f2dfe1943559602dc441301fb2/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/21df3bb6a8cf806694d8fb0ecfa991a342e9d8f2dfe1943559602dc441301fb2/work"
            }
        },
        "Mounts": [
            {
                "Type": "bind",
                "Name": "",
                "Source": "/usr/bin/trinity",
                "Destination": "/usr/bin/trinity",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Name": "",
                "Source": "/dev/kmsg",
                "Destination": "/dev/kmsg",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Name": "",
                "Source": "/opt/ltp",
                "Destination": "/opt/ltp",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "10.88.0.1",
            "IPAddress": "10.88.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "2a:45:2c:58:2d:8a",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": [],
            "SandboxKey": "/var/run/netns/cni-3baa67e3-1201-58e5-e41d-fb7d78ff5ed6"
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "error",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "/usr/bin/kata-runtime",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mountopt=nodev,metacopy=on",
            "--events-backend",
            "file",
            "container",
            "cleanup",
            "7badedb40528d6d8ca75f962148a82b9dd052d22ac525ed7b724d898d2aef58f"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "7badedb40528",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=podman",
                "HOSTNAME=7badedb40528",
                "HOME=/root"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "registry.access.redhat.com/ubi8:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": "",
            "OnBuild": null,
            "Labels": {
                "architecture": "x86_64",
                "build-date": "2020-07-22T12:11:33.547097",
                "com.redhat.build-host": "cpt-1007.osbs.prod.upshift.rdu2.redhat.com",
                "com.redhat.component": "ubi8-container",
                "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
                "description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
                "distribution-scope": "public",
                "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
                "io.k8s.display-name": "Red Hat Universal Base Image 8",
                "io.openshift.expose-services": "",
                "io.openshift.tags": "base rhel8",
                "maintainer": "Red Hat, Inc.",
                "name": "ubi8",
                "release": "343",
                "summary": "Provides the latest release of Red Hat Universal Base Image 8.",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.2-343",
                "vcs-ref": "0fede9a6ae85a36962206913c66bfe9965c49940",
                "vcs-type": "git",
                "vendor": "Red Hat, Inc.",
                "version": "8.2"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2020-07-23T17:18:40.107825641-04:00",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "0"
            },
            "StopSignal": 0,
            "CreateCommand": [
                "podman",
                "--runtime=/usr/bin/kata-runtime",
                "run",
                "--cpus",
                "16",
                "-m",
                "16g",
                "-v",
                "/dev/kmsg:/dev/kmsg",
                "-v",
                "/opt/ltp:/opt/ltp",
                "-v",
                "/usr/bin/trinity:/usr/bin/trinity",
                "-it",
                "ubi8",
                "bash"
            ]
        },
        "HostConfig": {
            "Binds": [
                "/usr/bin/trinity:/usr/bin/trinity:rw,rprivate,rbind",
                "/dev/kmsg:/dev/kmsg:rw,rprivate,nosuid,rbind",
                "/opt/ltp:/opt/ltp:rw,rprivate,rbind"
            ],
            "CgroupMode": "host",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 17179869184,
            "NanoCpus": 16000000000,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 100000,
            "CpuQuota": 1600000,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 34359738368,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 4096,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        }
    }
]

Neither /etc/containers/containers.conf nor /etc/containers/libpod.conf exist.

This looks like we are verifying the configuration of running containers?

	// Once for each missing runtime, we want to error.
	logrus.Errorf("OCI Runtime %s is in use by a container, but is not available (not in configuration file or not installed)", name)
This should definitely be a warning or maybe lesser.

func (s *BoltState) getContainerFromDB(id []byte, ctr *Container, ctrsBkt *bolt.Bucket) error {
Calls
			ociRuntime = getMissingRuntime(runtimeName, s.runtime)

When you specify a container with a --runc flag, which prints this error.  I do not see why this is an error, or even a warning.
Should be info or maybe debug.

https://github.com/containers/podman/pull/7077

Assigning to Matt Heon as he's taken over the PR noted in https://bugzilla.redhat.com/show_bug.cgi?id=1860176#c6

https://github.com/containers/podman/pull/7126 is a new fix.

This is getting closer

This is merged upstream. It is too late to make 8.3.1, but will make it in for 8.4.

Test with podman-3.0.0-0.38rc2.module+el8.4.0+9804+5385893b.x86_64 and the message is gone. So set this to verified. Details:
# podman --runtime /usr/bin/kata-runtime run -d registry.access.redhat.com/ubi8  sleep 99
bf33fd8a910de53c55dc585bcd2ffe10d7f0f534897a80fe38eaeeb789e36f69
# podman ps
CONTAINER ID  IMAGE                            COMMAND   CREATED         STATUS            PORTS   NAMES
bf33fd8a910d  registry.access.redhat.com/ubi8  sleep 99  11 seconds ago  Up 7 seconds ago          gallant_haibt

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1796