Bug 2009264

Summary: Cannot get logs with --follow
Product: Red Hat Enterprise Linux 8 Reporter: Matej Marušák <mmarusak>
Component: containers-commonAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Yuhui Jiang <yujiang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.6CC: bbaude, dwalsh, jligon, jnovy, lsm5, mheon, mpitt, pthomas, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: containers-common-1-23.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-08 09:14:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matej Marušák 2021-09-30 09:15:50 UTC
Description of problem:
In Cockpit CI we updated to new podman 4.0 in rhel-8.6 and now it is not possible to get logs from container.


Version-Release number of selected component (if applicable):
podman-4.0.0-0.10.module+el8.6.0+12734+7d3a79c8.x86_64
conmon-2.0.30-1.module+el8.6.0+12727+cc05c9e2.x86_64
containers-common-1-4.module+el8.6.0+12734+7d3a79c8.noarch
criu-3.15-3.module+el8.6.0+12698+b6644727.x86_64
systemd-239-50.el8.x86_64
kernel-4.18.0-345.1.el8.x86_64


How reproducible:
# podman run -dit --name foo busybox
48192c8a90aacaf8e651113b448b67227ac746cf108991b9298b868a75d8fb47
# podman logs --follow foo
Error: using --follow with the journald --log-driver but without the journald --events-backend (file) is not supported

Actual results:
Fails to show logs. If I omit `--follow` it works.

Comment 1 Matthew Heon 2021-09-30 12:16:37 UTC
Please provide the output of `podman info`. I assume that you're running the stock configuration (no changes to containers.conf)?

Comment 2 Matej Marušák 2021-09-30 12:46:20 UTC
> I assume that you're running the stock configuration (no changes to containers.conf)?

Yes, this is stock configuration.

```
# podman info
host:
  arch: amd64
  buildahVersion: 1.23.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.30-1.module+el8.6.0+12727+cc05c9e2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: 5530285273de12c68c8432bdba3abf2ed4de3ad3'
  cpus: 1
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  hostname: rhel-8-6-127-0-0-2-2201
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-345.1.el8.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 483303424
  memTotal: 980512768
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.6.0+12698+b6644727.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.6.0+12698+b6644727.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 51.99s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1632477228
  BuiltTime: Fri Sep 24 05:53:48 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 4.0.0-dev
```

Comment 3 Matthew Heon 2021-09-30 14:01:03 UTC
OK, we're shipping a buggy containers.conf on 8.6. We should match settings to what's in 8.5.0.

Comment 4 Matthew Heon 2021-09-30 14:45:44 UTC
The key difference is that 8.5.0 uses the `file` log driver - we'll need to patch `containers.conf` on 8.6.0 to override the default.

Adding `log_driver = "k8s-file"` to containers.conf should be sufficient.

Assigning to Jindrich.

Comment 5 Eduardo Minguez 2022-03-15 16:16:48 UTC
(In reply to Matthew Heon from comment #4)
> The key difference is that 8.5.0 uses the `file` log driver - we'll need to
> patch `containers.conf` on 8.6.0 to override the default.
> 
> Adding `log_driver = "k8s-file"` to containers.conf should be sufficient.
> 
> Assigning to Jindrich.

I faced this issue and the workaround provided by Matthew Heon works. In my case:

```
mkdir -p ~/.config/containers/
echo -e "[containers]\nlog_driver = \"k8s-file\"" >> ~/.config/containers/containers.conf
podman restart --all
```

Comment 7 Jindrich Novy 2022-03-15 19:30:12 UTC
This change is already present in 8.6.0 (committed 1st Dec 2021) but targeting this at 8.7.0 for QE to verify.

Can we get qa ack please?

Comment 14 errata-xmlrpc 2022-11-08 09:14:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7457