Hide Forgot
Description of problem: I noticed slow performance in fedora silverblue 35 (fresh install) when comparing execution time of SQL scripts against a MYSQL database started via podman or rootless docker against one started in docker for Windows (in WSL2). Actually it‘s multiple times slower than the Windows version. After starting strace in the mysql container I saw that all of the time is spent restart_syscall. So there seem to be lots of timeouts. The performance was bad with both btrfs and xfs file systems. How reproducible: Build the following docker image via podman or via (rootless) docker: --- FROM mysql:8 ENV MYSQL_ROOT_PASSWORD=someExamplePassword RUN apt-get update \ && apt-get install -y procps \ && apt-get install -y strace \ && echo 'CREATE DATABASE IF NOT EXISTS sakila;' > /docker-entrypoint-initdb.d/init.sql ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh","--character-set-server","utf8","--collation-server","utf8_general_ci"] --- Download the word_x example database script from https://dev.mysql.com/doc/index-other.html (https://downloads.mysql.com/docs/world_x-db.zip) Start the image via podman or rootless docker, the extra params are needed for strace: - Podman: podman run --cap-add=SYS_PTRACE -p3306:3306 <Name of the build docker image> - Docker: docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -p3306:3306 <Name of the build docker image> Open a bash in the container via exec -it and start strace: strace -c -p $(pgrep -x mysqld) Start importing the world_x example database schema into the mysql db (User root, password someExamplePassword). On my notebook it takes forever, so cancel strace. The output should look like this: root@6b1c83ccb6ab:/# strace -c -p $(pgrep -x mysqld) strace: Process 1 attached ^Cstrace: Process 1 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.69 0.004498 4498 1 restart_syscall 0.18 0.000008 8 1 accept 0.13 0.000006 6 1 futex ------ ----------- ----------- --------- --------- ---------------- 100.00 0.004512 3 total It happens with both podman and docker. The file system btfs or xfs has no effect. I tried it with a fresh install of fedora silverblue 35. So there seems to be a "deeper" problem.
Just to verify it, I just did a clean reinstall of fedora workstation (using the default btrfs partioning). The performance is as bad as with silverblue. So it's not silverblue specific. Again, using strace you'll see that most of the time is spent in restart_syscall.
These are the podman / docker versions installed on my newly setup fedora workstation: ##### Kernel [mschwartau@fedora ~]$ uname -r 5.16.12-200.fc35.x86_64 ##### podman: [mschwartau@fedora ~]$ podman info host: arch: amd64 buildahVersion: 1.23.1 cgroupControllers: - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.0-2.fc35.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.0, commit: ' cpus: 16 distribution: distribution: fedora variant: workstation version: "35" eventLogger: journald hostname: fedora idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.16.12-200.fc35.x86_64 linkmode: dynamic logDriver: journald memFree: 117969846272 memTotal: 134887301120 ociRuntime: name: crun package: crun-1.4.2-1.fc35.x86_64 path: /usr/bin/crun version: |- crun version 1.4.2 commit: f6fbc8f840df1a414f31a60953ae514fa497c748 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.12-2.fc35.x86_64 version: |- slirp4netns version 1.1.12 commit: 7a104a101aa3278a2152351a082a6df71f57c9a3 libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.3 swapFree: 8589930496 swapTotal: 8589930496 uptime: 27m 45.05s plugins: log: - k8s-file - none - journald network: - bridge - macvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /home/mschwartau/.config/containers/storage.conf containerStore: number: 4 paused: 0 running: 1 stopped: 3 graphDriverName: overlay graphOptions: {} graphRoot: /home/mschwartau/.local/share/containers/storage graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageStore: number: 5 runRoot: /run/user/1000/containers volumePath: /home/mschwartau/.local/share/containers/storage/volumes version: APIVersion: 3.4.4 Built: 1638999907 BuiltTime: Wed Dec 8 22:45:07 2021 GitCommit: "" GoVersion: go1.16.8 OsArch: linux/amd64 Version: 3.4.4 ##### rootless docker: [mschwartau@fedora ~]$ docker info Client: Context: default Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 20.10.12 Storage Driver: btrfs Build Version: Btrfs v4.20.1 Library Version: 102 Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: v1.0.2-0-g52b36a2d init version: de40ad0 Security Options: seccomp Profile: default rootless cgroupns Kernel Version: 5.16.12-200.fc35.x86_64 Operating System: Fedora Linux 35 (Workstation Edition) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 125.6GiB Name: fedora ID: JMRH:7ZUY:PYWI:RWL3:VTXI:UCGH:6VEJ:PPLC:MOZO:KFVE:MXW5:II2T Docker Root Dir: /home/mschwartau/.local/share/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: No cpuset support WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled
Okay, I give up. Installed PopOS on my notebook (Lenovo P15 Gen1 with internal samsung 980 PRO SSD). The performance is as bad as with fedora. Output of docker: --- mschwartau@pop-os:~$ docker info Client: Context: default Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 20.10.12 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: false userxattr: true Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: v1.0.2-0-g52b36a2d init version: de40ad0 Security Options: seccomp Profile: default rootless cgroupns Kernel Version: 5.16.11-76051611-generic Operating System: Pop!_OS 21.10 OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 125.6GiB Name: pop-os ID: RSYT:QPUK:ONMO:WBR5:7E3N:ISS3:OE7E:Y2VB:CHYA:7Q7K:QQ2Z:UW56 Docker Root Dir: /home/mschwartau/.local/share/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: No cpu cfs quota support WARNING: No cpu cfs period support WARNING: No cpu shares support WARNING: No cpuset support WARNING: No io.weight support WARNING: No io.weight (per device) support WARNING: No io.max (rbps) support WARNING: No io.max (wbps) support WARNING: No io.max (riops) support WARNING: No io.max (wiops) support WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled --- So maybe the Samsung SSD doesn't like linux (I use windows on another internal ssd with which I got the notebook). Or I have some wrong bios settings. Or rootless docker and podman are slower than the windows version. But I'll switch back to Windows for now :-(. Regards Meinert
This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.