Bug 2062008
| Summary: | Slow mysql performance in rootless docker / podman due to many timeouts | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Meinert Schwartau <m.schwartau> |
| Component: | podman | Assignee: | Matthew Heon <mheon> |
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 35 | CC: | acui, bbaude, container-sig, debarshir, dwalsh, jnovy, lsm5, mheon, patrick, pehunt, rh.container.bot, santiago, steven.daniel.webb |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | --- | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-12-13 16:56:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Just to verify it, I just did a clean reinstall of fedora workstation (using the default btrfs partioning). The performance is as bad as with silverblue. So it's not silverblue specific. Again, using strace you'll see that most of the time is spent in restart_syscall. These are the podman / docker versions installed on my newly setup fedora workstation:
##### Kernel
[mschwartau@fedora ~]$ uname -r
5.16.12-200.fc35.x86_64
##### podman:
[mschwartau@fedora ~]$ podman info
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.0-2.fc35.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: '
cpus: 16
distribution:
distribution: fedora
variant: workstation
version: "35"
eventLogger: journald
hostname: fedora
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.16.12-200.fc35.x86_64
linkmode: dynamic
logDriver: journald
memFree: 117969846272
memTotal: 134887301120
ociRuntime:
name: crun
package: crun-1.4.2-1.fc35.x86_64
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.12-2.fc35.x86_64
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 8589930496
swapTotal: 8589930496
uptime: 27m 45.05s
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
- quay.io
store:
configFile: /home/mschwartau/.config/containers/storage.conf
containerStore:
number: 4
paused: 0
running: 1
stopped: 3
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/mschwartau/.local/share/containers/storage
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 5
runRoot: /run/user/1000/containers
volumePath: /home/mschwartau/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1638999907
BuiltTime: Wed Dec 8 22:45:07 2021
GitCommit: ""
GoVersion: go1.16.8
OsArch: linux/amd64
Version: 3.4.4
##### rootless docker:
[mschwartau@fedora ~]$ docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.12
Storage Driver: btrfs
Build Version: Btrfs v4.20.1
Library Version: 102
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2d
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
cgroupns
Kernel Version: 5.16.12-200.fc35.x86_64
Operating System: Fedora Linux 35 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 125.6GiB
Name: fedora
ID: JMRH:7ZUY:PYWI:RWL3:VTXI:UCGH:6VEJ:PPLC:MOZO:KFVE:MXW5:II2T
Docker Root Dir: /home/mschwartau/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No cpuset support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Okay, I give up. Installed PopOS on my notebook (Lenovo P15 Gen1 with internal samsung 980 PRO SSD). The performance is as bad as with fedora. Output of docker: --- mschwartau@pop-os:~$ docker info Client: Context: default Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 20.10.12 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: false userxattr: true Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: v1.0.2-0-g52b36a2d init version: de40ad0 Security Options: seccomp Profile: default rootless cgroupns Kernel Version: 5.16.11-76051611-generic Operating System: Pop!_OS 21.10 OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 125.6GiB Name: pop-os ID: RSYT:QPUK:ONMO:WBR5:7E3N:ISS3:OE7E:Y2VB:CHYA:7Q7K:QQ2Z:UW56 Docker Root Dir: /home/mschwartau/.local/share/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: No cpu cfs quota support WARNING: No cpu cfs period support WARNING: No cpu shares support WARNING: No cpuset support WARNING: No io.weight support WARNING: No io.weight (per device) support WARNING: No io.max (rbps) support WARNING: No io.max (wbps) support WARNING: No io.max (riops) support WARNING: No io.max (wiops) support WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled --- So maybe the Samsung SSD doesn't like linux (I use windows on another internal ssd with which I got the notebook). Or I have some wrong bios settings. Or rootless docker and podman are slower than the windows version. But I'll switch back to Windows for now :-(. Regards Meinert This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed. Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed. |
Description of problem: I noticed slow performance in fedora silverblue 35 (fresh install) when comparing execution time of SQL scripts against a MYSQL database started via podman or rootless docker against one started in docker for Windows (in WSL2). Actually it‘s multiple times slower than the Windows version. After starting strace in the mysql container I saw that all of the time is spent restart_syscall. So there seem to be lots of timeouts. The performance was bad with both btrfs and xfs file systems. How reproducible: Build the following docker image via podman or via (rootless) docker: --- FROM mysql:8 ENV MYSQL_ROOT_PASSWORD=someExamplePassword RUN apt-get update \ && apt-get install -y procps \ && apt-get install -y strace \ && echo 'CREATE DATABASE IF NOT EXISTS sakila;' > /docker-entrypoint-initdb.d/init.sql ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh","--character-set-server","utf8","--collation-server","utf8_general_ci"] --- Download the word_x example database script from https://dev.mysql.com/doc/index-other.html (https://downloads.mysql.com/docs/world_x-db.zip) Start the image via podman or rootless docker, the extra params are needed for strace: - Podman: podman run --cap-add=SYS_PTRACE -p3306:3306 <Name of the build docker image> - Docker: docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -p3306:3306 <Name of the build docker image> Open a bash in the container via exec -it and start strace: strace -c -p $(pgrep -x mysqld) Start importing the world_x example database schema into the mysql db (User root, password someExamplePassword). On my notebook it takes forever, so cancel strace. The output should look like this: root@6b1c83ccb6ab:/# strace -c -p $(pgrep -x mysqld) strace: Process 1 attached ^Cstrace: Process 1 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.69 0.004498 4498 1 restart_syscall 0.18 0.000008 8 1 accept 0.13 0.000006 6 1 futex ------ ----------- ----------- --------- --------- ---------------- 100.00 0.004512 3 total It happens with both podman and docker. The file system btfs or xfs has no effect. I tried it with a fresh install of fedora silverblue 35. So there seems to be a "deeper" problem.