Description of problem: Unused storage directories from old container which are no longer in use is filling up the filesystem space. Version-Release number of selected component (if applicable): [root@rgw-6 storage]# podman version Version: 1.6.4 RemoteAPI Version: 1 Go Version: go1.13.4 OS/Arch: linux/amd64 [root@rgw-6 storage]# How reproducible: Always ( experiencing on 6 nodes ) Steps to Reproduce: 1. deploy RHCS 4.0 2. purge rhcs4.0 3. re-deploy rhcs4.0 4. you will see old container directories consuming space Actual results: Expected results: As soon as containers are removed, their corresponding directories should be removed. Additional info: [root@rgw-6 storage]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 94G 0 94G 0% /dev tmpfs 94G 84K 94G 1% /dev/shm tmpfs 94G 4.2G 90G 5% /run tmpfs 94G 0 94G 0% /sys/fs/cgroup /dev/mapper/rhel-root 112G 57G 56G 51% / /dev/sda2 1014M 175M 840M 18% /boot /dev/sda1 599M 6.8M 593M 2% /boot/efi tmpfs 19G 16K 19G 1% /run/user/42 shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/a0c6118c34996b7a22ab247fd47cd931ee7b3fd56285b1e2d7beafe68da1aa08/userdata/shm overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/a4897d3b8eb76751b5f59500de45369dd32fa8a52ee63725fabaf949c17498ba/merged shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/270d4d29801cf780c992dec85a45bc81beb1b924d6247e1dba8872e0ba9a2e8d/userdata/shm overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/338c2db84503c68ddc0007d3bc14f53c9f003eedceaa6c1b6a03b0d2cda6b044/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/a7d6b631ab7d0c206ab2a71f870f4d3e235767217f248237a52a9bb52760b491/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/134fce9b04d93a52fafee555e324f8571944fbb47e58e6aa18901a897f431dd8/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/658763008cb19dd94122e0fb84129b9796f6848f1006b17357daac11840fbed7/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/f6bbbc215999daae8c3a88dffd9d1b00065ef6b6d19362aff6092db3bf4a6e8d/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/66b37d04974989f6c58193c841360442c7fb98546b839dea77559550422a743f/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/d175d8ee859f85f8a5c007731daf3bd362dbb457dc781f49fee0df0dbea02c21/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/5f66e95400e84d284dc761192edaf4f9c9ee4eb5d8be3aeac19d6d95a31e7933/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/f44a7696cc5978485f5b167fe2e858b71f1868731eea6ee89ed4aa4246de8273/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/b9882144e4be25a233521b988f80dc1cd24e35cfea7eaa708134570e175265bf/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/6f22bf5c13bf3aa39d1183e1e0027b79e91f2f1bbf04f88e8c13da46e4bbccd6/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/02b61d6002bbef89d0870356165cbb61c92b89fa823a24a07df8ac9b896db5b9/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/7119ca771dff7749dc78072d9444960607f9b67f3e53cccf78ce59d076061421/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/caf05fb3ce8f72b0f5a385fd215769802103390199efcf9de82b232616c19bdb/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/56ecbcc9c9b2b4827663c5075dd6d951b4fe4f3efaf13c228a879a029fde1a9e/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/511cf1a39c74953b07bad6aa0a818d69f993da8e130ce2b3cb378c8a523f645a/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/7c38139982c713e7efa764e029145b7fc8701ca6762a3a7c5b37ce7906801180/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/12b6edfc50bb2ef5bfb4bcbf2be1468a78e1e653ca278bcf51ef03d427877504/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/cf5399c5da31fef0f7fcf5826bef558c58ca55a2217682d56d402439527f5fc5/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/7de7d89a980d95be3ba69b338ee959745965b5b53c5caf688745e3d304e63274/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/1b26ac5cec6d6e4b401acb0930dca5dd5346c83e42925e41d0418c75ea76b212/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/e24c2ee45fb12b626561e1d12da17767a1493b630e308bd0a508d5f539c7b452/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/61846adeeeba54aac776898c8574c72f604c028be33080f212524700e86ddc76/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/4c9922c9df114211249f56d23d80e0773e5e07a57d2ac6e3a97ba70f7deba1d2/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/e338c3e90a34ff9ddcb4aac36aa237f5d953ffea992fb6947d24a789d18522b0/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/ddf1df0d6156e7958b20d1a5ad3a6090b0b88387f8f11fcf7d804f7aaea6d520/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/49bd4de8c481bb135b82b17c6bb6ac84c2e97de4de80dea633f372f5ebeb0467/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/77a9f0160e7e7f680ea029397c0c8de8e5992b08939c710c1bd969734d41f624/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/3c806f5eb0a32f7e5203fcec970a447a4351eb46e010869cbb3990c487058c62/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/4dbbb178e8a48e03a9387474d42bfc074dbd9945ee831405ce2cd8b7a428c896/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/76203f630338a6602f1ac2320ad864ffb8ca4c0ea201dd00a7b8d556ca12b2c1/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/11f24eb06082789a04a03f43bf5e35418b0be06dfc32dd1767ab4a0061c537fb/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/f35eb9b4da3099ec173367365b706ea2b2c66119e931fe6fa33f9cb73f6c615f/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/b37aaa009998044c72c2863d0b156d8e0768d641f7e79a65903bfc1657e7132b/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/92254ce97fcb59def4347d3c8b1bb0f56c28f268b4f1861f90e54429f3023a48/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/51d5a3adbe044dfa7583e58c59598e12f5d194ac2a075de20d42b578eac9a12a/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/485fe4bf82b15d52205672317df540c62563231eae7129814e6363239d82fc6b/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/023bf9f0e6dfbedefd7860f0b2399bfc3d11af7ab25cb3ab0e74ace93cd32487/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/67fb72861b49a6f9437a812c88656fe0e57ecac73a76fb48e7d1c598fd9fff8a/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/54b7f4ccd56930469d5b9a00dbd93a128d0871090e651721e0025ded0c08820a/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/c2ad413a4f804445d8c3af9c36b75d98d90c5f5b6f81217297d8639a7b4dd23f/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/2cabc6fdace1f3acaa9a46d7ff0e8669a46fae6941fa8135bfbbb66a65c527e5/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/d054126b19221bb82b56eec853231825dda478d7bd09abdcd9e7a841435b3ff1/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/acf72e66458e91cb6357c4adcd28cbf541116777ad37270b1aed92d506111b36/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/d4d0858c074f0fbd522efdc77c5cf65a6c4fa615c70cddf4cf0caf158909d9ef/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/2e7473fd45d5756e2037b720ea91f52ec0d8f04eb8b6185178fa52c587423de4/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/9d024d133ec8e6caeda339333256a7b558757301d4c01c09052cc5d19a609192/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/fd878d68fa22b8ea7f624b6bb088f4426ba90b9c0373a31a2eb9c49abbbf0b54/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/56c4ee3861d21eb08aa64f93777359a206c914ee3b61a634b3fa6f01d33cc5d2/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/b9dac4319495a4eb4ff32526c6dcf926fc03b687c8a6a23f0a52c610d5bb4f6b/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/5de45c6b18999c61116834abc7088a95c49a61a16262181cdae4e189ed101a32/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/9a1549c6cdde79294e8c0c1a00e9ef0b383d4a5df6a9d2b20f8adb86e5bfdd40/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/e9bf6a4695fd7a2d923583ce1c31454ea2387d3ce4f461dd23599e6419c27e1a/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/9e910b819502047b28832acc5a7d7a1eb4a1fd62e617e888ff466d2b521e134e/merged shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/7fd487bd4b1b76c86556659ce9960b79f24618979f593050ef1a0f3392f0590c/userdata/shm overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/784cdb2a8ccd28ed7309d4aa75ec852cc083c5a219ff3929ba15f164847d0d58/merged shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/b262e9e208524579e4f3ff9e11099bf933e10cf69655b9d91d7cdf7c28b850c1/userdata/shm overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/a760fc8aa7f865bbd095a54788c446b5940cb0fe03b69baff44391fb168a54e0/merged overlay 112G 57G 56G 51% /var/lib/containers/storage/overlay/da41cc69949ceee58b2670f778cf5cce3faf43f3ae51150ea16019b488afed62/merged [root@rgw-6 storage]# ## Running/Stopped containers count=59 [root@rgw-6 storage]# podman ps -a | wc -l 59 [root@rgw-6 storage]# ## Container with overlay storage = 58 [root@rgw-6 storage]# df -h | grep -v shm | grep -i containers | wc -l 58 [root@rgw-6 storage]# ## [root@rgw-6 storage]# df -h /var Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 112G 49G 64G 44% / [root@rgw-6 storage]# [root@rgw-6 storage]# du -sh /var/lib/containers/ 57G /var/lib/containers/ [root@rgw-6 storage]# drwx------. 76 root root 8192 May 12 13:09 overlay-containers drwx------. 4 root root 188 May 4 00:15 overlay-images drwx------. 2 root root 4096 May 12 13:09 overlay-layers -rw-------. 1 root root 64 May 3 11:00 storage.lock drwx------. 2 root root 6 Apr 29 01:26 tmp [root@rgw-6 storage]# du -sh * 13M libpod 0 mounts 57G overlay 5.2M overlay-containers 56K overlay-images 3.9M overlay-layers 4.0K storage.lock 0 tmp [root@rgw-6 storage]# [root@rgw-6 overlay]# ls -l total 4 drwx------. 5 root root 69 May 9 11:22 023bf9f0e6dfbedefd7860f0b2399bfc3d11af7ab25cb3ab0e74ace93cd32487 drwx------. 5 root root 69 May 9 11:20 02b61d6002bbef89d0870356165cbb61c92b89fa823a24a07df8ac9b896db5b9 drwx------. 5 root root 69 May 9 11:22 11f24eb06082789a04a03f43bf5e35418b0be06dfc32dd1767ab4a0061c537fb drwx------. 5 root root 69 May 9 11:21 12b6edfc50bb2ef5bfb4bcbf2be1468a78e1e653ca278bcf51ef03d427877504 drwx------. 5 root root 69 May 9 11:20 134fce9b04d93a52fafee555e324f8571944fbb47e58e6aa18901a897f431dd8 drwx------. 5 root root 69 Apr 29 07:21 14694d06ab05beda84e7a2f9403ce202fbd32078bf52553b4949c54101947954 drwx------. 5 root root 69 May 9 11:21 1b26ac5cec6d6e4b401acb0930dca5dd5346c83e42925e41d0418c75ea76b212 drwx------. 5 root root 69 May 9 11:22 2cabc6fdace1f3acaa9a46d7ff0e8669a46fae6941fa8135bfbbb66a65c527e5 drwx------. 5 root root 69 May 9 11:22 2e7473fd45d5756e2037b720ea91f52ec0d8f04eb8b6185178fa52c587423de4 drwx------. 5 root root 69 May 4 00:15 338c2db84503c68ddc0007d3bc14f53c9f003eedceaa6c1b6a03b0d2cda6b044 drwx------. 5 root root 69 Apr 29 07:21 34eac3d447522c5f96dfe2488d96820a4a65c23c1710a13605cae9f145e158b7 drwx------. 6 root root 69 May 4 00:15 35b7a5c4e1b4a84fb05d9c6658572c2b7a9925a270e8f7860c0ae30671c0a57c drwx------. 5 root root 69 May 9 11:21 3c806f5eb0a32f7e5203fcec970a447a4351eb46e010869cbb3990c487058c62 drwx------. 5 root root 69 May 9 11:22 485fe4bf82b15d52205672317df540c62563231eae7129814e6363239d82fc6b drwx------. 5 root root 69 May 9 11:21 49bd4de8c481bb135b82b17c6bb6ac84c2e97de4de80dea633f372f5ebeb0467 drwx------. 5 root root 69 May 9 11:21 4c9922c9df114211249f56d23d80e0773e5e07a57d2ac6e3a97ba70f7deba1d2 drwx------. 5 root root 69 May 9 11:21 4dbbb178e8a48e03a9387474d42bfc074dbd9945ee831405ce2cd8b7a428c896 drwx------. 5 root root 69 May 9 11:20 511cf1a39c74953b07bad6aa0a818d69f993da8e130ce2b3cb378c8a523f645a drwx------. 5 root root 69 May 9 11:22 51d5a3adbe044dfa7583e58c59598e12f5d194ac2a075de20d42b578eac9a12a drwx------. 5 root root 69 May 9 11:22 54b7f4ccd56930469d5b9a00dbd93a128d0871090e651721e0025ded0c08820a drwx------. 5 root root 69 May 9 11:23 56c4ee3861d21eb08aa64f93777359a206c914ee3b61a634b3fa6f01d33cc5d2 drwx------. 5 root root 69 May 9 11:20 56ecbcc9c9b2b4827663c5075dd6d951b4fe4f3efaf13c228a879a029fde1a9e drwx------. 5 root root 69 May 9 11:23 5de45c6b18999c61116834abc7088a95c49a61a16262181cdae4e189ed101a32 drwx------. 5 root root 69 May 9 11:20 5f66e95400e84d284dc761192edaf4f9c9ee4eb5d8be3aeac19d6d95a31e7933 drwx------. 5 root root 69 May 9 11:21 61846adeeeba54aac776898c8574c72f604c028be33080f212524700e86ddc76 drwx------. 5 root root 69 May 9 11:20 658763008cb19dd94122e0fb84129b9796f6848f1006b17357daac11840fbed7 drwx------. 5 root root 69 May 9 11:20 66b37d04974989f6c58193c841360442c7fb98546b839dea77559550422a743f drwx------. 5 root root 69 May 9 11:22 67fb72861b49a6f9437a812c88656fe0e57ecac73a76fb48e7d1c598fd9fff8a drwx------. 5 root root 69 May 9 11:20 6f22bf5c13bf3aa39d1183e1e0027b79e91f2f1bbf04f88e8c13da46e4bbccd6 drwx------. 5 root root 69 May 9 11:20 7119ca771dff7749dc78072d9444960607f9b67f3e53cccf78ce59d076061421 drwx------. 5 root root 69 May 4 00:15 732b22cd2991d5c4122df1b761a330af125f49c5574229df76df21f16b65441a drwx------. 5 root root 69 May 9 11:21 76203f630338a6602f1ac2320ad864ffb8ca4c0ea201dd00a7b8d556ca12b2c1 drwx------. 5 root root 69 May 9 11:21 77a9f0160e7e7f680ea029397c0c8de8e5992b08939c710c1bd969734d41f624 drwx------. 5 root root 69 May 11 09:28 784cdb2a8ccd28ed7309d4aa75ec852cc083c5a219ff3929ba15f164847d0d58 drwx------. 5 root root 69 May 9 11:21 7c38139982c713e7efa764e029145b7fc8701ca6762a3a7c5b37ce7906801180 drwx------. 5 root root 69 May 9 11:21 7de7d89a980d95be3ba69b338ee959745965b5b53c5caf688745e3d304e63274 drwx------. 5 root root 69 May 9 11:22 92254ce97fcb59def4347d3c8b1bb0f56c28f268b4f1861f90e54429f3023a48 drwx------. 6 root root 69 Apr 29 07:21 92b864bfcfaa5486b735fd9bfe48cf59636532ce9873532bef2cf9988570c858 drwx------. 5 root root 69 May 9 11:23 9a1549c6cdde79294e8c0c1a00e9ef0b383d4a5df6a9d2b20f8adb86e5bfdd40 drwx------. 5 root root 69 May 9 11:23 9d024d133ec8e6caeda339333256a7b558757301d4c01c09052cc5d19a609192 drwx------. 5 root root 69 May 9 11:23 9e910b819502047b28832acc5a7d7a1eb4a1fd62e617e888ff466d2b521e134e drwx------. 5 root root 69 May 4 00:13 a4897d3b8eb76751b5f59500de45369dd32fa8a52ee63725fabaf949c17498ba drwx------. 5 root root 69 May 11 09:28 a760fc8aa7f865bbd095a54788c446b5940cb0fe03b69baff44391fb168a54e0 drwx------. 5 root root 69 May 9 11:20 a7d6b631ab7d0c206ab2a71f870f4d3e235767217f248237a52a9bb52760b491 drwx------. 5 root root 69 May 9 11:22 acf72e66458e91cb6357c4adcd28cbf541116777ad37270b1aed92d506111b36 drwx------. 5 root root 69 May 9 11:22 b37aaa009998044c72c2863d0b156d8e0768d641f7e79a65903bfc1657e7132b drwx------. 5 root root 69 May 9 11:20 b9882144e4be25a233521b988f80dc1cd24e35cfea7eaa708134570e175265bf drwx------. 5 root root 69 May 9 11:23 b9dac4319495a4eb4ff32526c6dcf926fc03b687c8a6a23f0a52c610d5bb4f6b brw-------. 1 root root 253, 0 May 12 13:09 backingFsBlockDev drwx------. 5 root root 69 May 9 11:22 c2ad413a4f804445d8c3af9c36b75d98d90c5f5b6f81217297d8639a7b4dd23f drwx------. 5 root root 69 May 9 11:20 caf05fb3ce8f72b0f5a385fd215769802103390199efcf9de82b232616c19bdb drwx------. 5 root root 69 May 9 11:21 cf5399c5da31fef0f7fcf5826bef558c58ca55a2217682d56d402439527f5fc5 drwx------. 5 root root 69 May 9 11:22 d054126b19221bb82b56eec853231825dda478d7bd09abdcd9e7a841435b3ff1 drwx------. 5 root root 69 May 9 11:20 d175d8ee859f85f8a5c007731daf3bd362dbb457dc781f49fee0df0dbea02c21 drwx------. 5 root root 69 May 12 13:09 d238e48b6d90c8047b3568ea986fc8583b81e55b8b7da960b83c389b963104bd drwx------. 5 root root 69 May 9 11:22 d4d0858c074f0fbd522efdc77c5cf65a6c4fa615c70cddf4cf0caf158909d9ef drwx------. 5 root root 69 May 4 00:15 d966b6410ba70fde02aee54b065aea105196483a2f0fb4ff6ddffd10199b4e3c drwx------. 5 root root 69 May 9 11:21 ddf1df0d6156e7958b20d1a5ad3a6090b0b88387f8f11fcf7d804f7aaea6d520 drwx------. 5 root root 69 May 9 11:21 e24c2ee45fb12b626561e1d12da17767a1493b630e308bd0a508d5f539c7b452 drwx------. 5 root root 69 May 9 11:21 e338c3e90a34ff9ddcb4aac36aa237f5d953ffea992fb6947d24a789d18522b0 drwx------. 5 root root 69 May 9 11:23 e9bf6a4695fd7a2d923583ce1c31454ea2387d3ce4f461dd23599e6419c27e1a drwx------. 5 root root 69 May 9 11:22 f35eb9b4da3099ec173367365b706ea2b2c66119e931fe6fa33f9cb73f6c615f drwx------. 5 root root 69 May 4 00:15 f38558f603890918b58c46ad4d836489bea77d2be08e9460abe7012e5d9a28fd drwx------. 5 root root 69 May 9 11:20 f44a7696cc5978485f5b167fe2e858b71f1868731eea6ee89ed4aa4246de8273 drwx------. 5 root root 69 May 9 11:20 f6bbbc215999daae8c3a88dffd9d1b00065ef6b6d19362aff6092db3bf4a6e8d drwx------. 5 root root 69 May 9 11:23 fd878d68fa22b8ea7f624b6bb088f4426ba90b9c0373a31a2eb9c49abbbf0b54 drwx------. 2 root root 4096 May 12 13:09 l [root@rgw-6 overlay]# [root@rgw-6 overlay]# ls -l | wc -l 68 [root@rgw-6 overlay]# The problem is there are 10 directories under overlay which is eating up ~10GB of space. I have same issue on other nodes as well Setting the severity to urgent as because if the filesystem gets filled up, i have not way to save my cluster. So would appretiate any help here.
How is Podman being invoked to remove the containers? Is Podman being used at all? It seems like there are still active overlay mounts - it seems like the containers in question didn't exit cleanly / weren't removed by `podman rm`? I think we need a lot more information about how RHCS invokes Podman (and tears it down when being removed) to be able to diagnose this.
Assigning to Matt now, but we likely will need input from RHCS.
As an example, if you see this service file, you will find answers to your questions 1. podman is being managed using service files 2. start == run 3. stop == rm (pre-requisite to start) [root@rgw-5 ~]# cat /etc/systemd/system/ceph-radosgw@.service [Unit] Description=Ceph RGW After=network.target [Service] EnvironmentFile=/var/lib/ceph/radosgw/ceph-%i/EnvironmentFile ExecStartPre=-/usr/bin/podman stop ceph-rgw-rgw-5-${INST_NAME} ExecStartPre=-/usr/bin/podman rm ceph-rgw-rgw-5-${INST_NAME} ExecStart=/usr/bin/podman run --rm --net=host \ --memory=191620m \ --cpus=8 \ -v /var/lib/ceph:/var/lib/ceph:z \ -v /etc/ceph:/etc/ceph:z \ -v /var/run/ceph:/var/run/ceph:z \ -v /etc/localtime:/etc/localtime:ro \ -v /var/log/ceph:/var/log/ceph:z \ -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:z \ -e CEPH_DAEMON=RGW \ -e CLUSTER=ceph \ -e RGW_NAME=rgw-5.${INST_NAME} \ -e RGW_CIVETWEB_PORT=${INST_PORT} \ -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \ --name=ceph-rgw-rgw-5-${INST_NAME} \ \ registry.redhat.io/rhceph/rhceph-4-rhel8:latest ExecStopPost=-/usr/bin/podman stop ceph-rgw-rgw-5-${INST_NAME} Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi-user.target [root@rgw-5 ~]# Per my output above, i do not see mounted container overlay directories if the container is not running (which is good), however, i do see extra container overlay directories under /var/lib/containers/overlay . These directories are old leftovers from earlier container runs. I even tried podman system prune --all --volumes but it was not useful.
It doesn't look like you're using unit files from `podman generate systemd` - and that has produced a number of problems. This looks like a simple unit not managed by PID files. This is not a supported configuration for running Podman under systemd. Podman containers are not direct children of the Podman process; Podman starts conmon, a lightweight container monitor daemon, which double-forks to daemonize and reparent on init, and the container is started as a child of Conmon. At that point, the Podman process is entirely superfluous; killing Podman, as systemd would do to stop the unit, is by no means guaranteed to kill the actual *container*. Please investigate moving to unit files generated by `podman generate systemd --new`. These are Type=forking and use PID files to ensure that Conmon and the container are properly recognized and managed by systemd, amongst other fixes to ensure that Systemd can properly manage the container.
Hi Guillaume / Seb Per comment-4 from Matthew Heon, can you guys chime into this discussion. Are we aware that managing podman under systems without PID files is not a supported configuration? We have been using this all over the place in RHCS 4. I am not sure in RHCS 4.1, we have plans to move to podman generate systems (Type=forking) and PID files, if that's the recommended way to run Podman on RHEL. (if you are not the right person for this discussion, pls add the right guy in this bz, so that we can come to a conclusion)
FYI there is one more BZ along the same lines, checkout https://bugzilla.redhat.com/show_bug.cgi?id=1807440
I don't have much experience with podman, I'm adding Dimitri as well. Dimitri, any idea?
Dan Walsh, can you comment on https://bugzilla.redhat.com/show_bug.cgi?id=1834974#c5 please?
Does `podman image prune` exist and do anything? `podman container prune` The issue seems to be the way your systemd unit file is configured. Valenetin could you advise?
Verified with Ceph Version: 14.2.8-79.el8cp Ceph Ansible Version: 4.0.25-1.el8cp.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3003
Seems to have introduced https://bugzilla.redhat.com/show_bug.cgi?id=1858865
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days