Hi The customer is waiting for an update, do you need any additional information ? Regards Alicia
Heal just finished: ``` sh-4.2# gluster v status vol_7cedfda39e63fbb59c2998233091cb10 Status of volume: vol_7cedfda39e63fbb59c2998233091cb10 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.28.139.73:/var/lib/heketi/mounts/v g_e48e0dbb3dd65dd250ce3c997effe892/brick_34 934b39c2999e4f55a0c89aecab5718/brick 49161 0 Y 18510 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_329c82fd567da85c0b62c5dda1c342f3/brick_5d 1fcdf29e210fbf066c910126d43524/brick 49159 0 Y 28716 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_f2e34e863e3a99b21011b4d8344ec848/brick_4 cd6d96d7b40cd17b59c4950b28cf3bc/brick 49159 0 Y 50806 Brick 10.28.139.73:/var/lib/heketi/mounts/v g_c7be81950c8c69dbb33fbbea58d5d3a5/brick_78 9c14aa897f8d1a75d0954ff2dd0227/brick 49161 0 Y 18510 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_042c5a884998313285e3d00c6c59470c/brick_4 65fda0ee5c13e5d78e3279461e1fe99/brick 49159 0 Y 50806 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_2b36d9ace572cb665ea1cb28ebe45081/brick_f5 7189b014ed23f279d80db687b89ad5/brick 49159 0 Y 28716 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_2b36d9ace572cb665ea1cb28ebe45081/brick_d8 a46b7c8bd752f2031577ea3be69112/brick 49159 0 Y 28716 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7 677d50724618caa89055fdd921bb501/brick 49159 0 Y 50806 Brick 10.28.139.73:/var/lib/heketi/mounts/v g_d54f514570ec9f63a8d1e7b578b03059/brick_00 cd7128c612491996944fafb80bc8f9/brick 49161 0 Y 18510 Brick 10.28.139.73:/var/lib/heketi/mounts/v g_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b 00db39119354483417a28b5d55aaf1/brick 49161 0 Y 18510 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_042c5a884998313285e3d00c6c59470c/brick_7 076b52b2f6219162e8973b338859c4e/brick 49159 0 Y 50806 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_2b36d9ace572cb665ea1cb28ebe45081/brick_99 7a09c1ac081ddc7e805ff759b7f29a/brick 49159 0 Y 28716 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_042c5a884998313285e3d00c6c59470c/brick_9 bd4899a61d0998ef986e5d714a0df4f/brick 49159 0 Y 50806 Brick 10.28.139.73:/var/lib/heketi/mounts/v g_d54f514570ec9f63a8d1e7b578b03059/brick_b6 dd1cbab3c28cb84a56ce1d65093935/brick 49161 0 Y 18510 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_6eb35f532caca1f72b88a54475950a7d/brick_92 d7d2aca1a7d8b5c02b94f61c807120/brick 49159 0 Y 28716 Brick 10.28.139.166:/var/lib/heketi/mounts/ vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c 5a962f47423c1232d31e2c901d5de08/brick 49159 0 Y 50806 Brick 10.28.139.73:/var/lib/heketi/mounts/v g_d54f514570ec9f63a8d1e7b578b03059/brick_2b d672a66df2636fa91cc255cf4accf9/brick 49161 0 Y 18510 Brick 10.28.139.13:/var/lib/heketi/mounts/v g_6eb35f532caca1f72b88a54475950a7d/brick_5f 2f7653e2732cdf542b8ac934e05352/brick 49159 0 Y 28716 Self-heal Daemon on localhost N/A N/A Y 40089 Self-heal Daemon on storage02.appcanvas.net N/A N/A Y 97772 Self-heal Daemon on storage01.appcanvas.net N/A N/A Y 125437 Task Status of Volume vol_7cedfda39e63fbb59c2998233091cb10 ------------------------------------------------------------------------------ There are no active volume tasks sh-4.2# gluster v heal vol_7cedfda39e63fbb59c2998233091cb10 info Brick 10.28.139.73:/var/lib/heketi/mounts/vg_e48e0dbb3dd65dd250ce3c997effe892/brick_34934b39c2999e4f55a0c89aecab5718/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_329c82fd567da85c0b62c5dda1c342f3/brick_5d1fcdf29e210fbf066c910126d43524/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_f2e34e863e3a99b21011b4d8344ec848/brick_4cd6d96d7b40cd17b59c4950b28cf3bc/brick Status: Connected Number of entries: 0 Brick 10.28.139.73:/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_789c14aa897f8d1a75d0954ff2dd0227/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_465fda0ee5c13e5d78e3279461e1fe99/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_f57189b014ed23f279d80db687b89ad5/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_d8a46b7c8bd752f2031577ea3be69112/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7677d50724618caa89055fdd921bb501/brick Status: Connected Number of entries: 0 Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_00cd7128c612491996944fafb80bc8f9/brick Status: Connected Number of entries: 0 Brick 10.28.139.73:/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b00db39119354483417a28b5d55aaf1/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_7076b52b2f6219162e8973b338859c4e/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_997a09c1ac081ddc7e805ff759b7f29a/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_9bd4899a61d0998ef986e5d714a0df4f/brick Status: Connected Number of entries: 0 Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_b6dd1cbab3c28cb84a56ce1d65093935/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_92d7d2aca1a7d8b5c02b94f61c807120/brick Status: Connected Number of entries: 0 Brick 10.28.139.166:/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c5a962f47423c1232d31e2c901d5de08/brick Status: Connected Number of entries: 0 Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_2bd672a66df2636fa91cc255cf4accf9/brick Status: Connected Number of entries: 0 Brick 10.28.139.13:/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_5f2f7653e2732cdf542b8ac934e05352/brick Status: Connected Number of entries: 0 ``` It shows correct here now, but only after killing 03 and forcing a heal: ``` [root@storage01 docker_registry]# ls -l docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 total 2 -rw-r--r--. 1 1000030000 2000 1992 Feb 7 12:02 data [root@storage01 docker_registry]# ``` But not sure if there are any other files that could have been missed, curious if we can force a re-heal/rebalance or have a different way for OCS to double check everything within the volume. Even though the split-brain for the data dir has gone, still issues as its now an empty file: ``` WMac5159685:~ c09010a$ docker pull docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 Error response from daemon: manifest for docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 not found WMac5159685:~ c09010a$ ``` File pretty much empty via FUSE: ``` [root@storage01 docker_registry]# cd docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 [root@storage01 61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2]# du -shc * 2.0K data 2.0K total [root@storage01 61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2]# ``` storage01: ``` sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.13 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done 4.0K /var/lib/heketi/mounts/vg_329c82fd567da85c0b62c5dda1c342f3/brick_5d1fcdf29e210fbf066c910126d43524/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 4.0K total 0 /var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_f57189b014ed23f279d80db687b89ad5/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_d8a46b7c8bd752f2031577ea3be69112/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_997a09c1ac081ddc7e805ff759b7f29a/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_92d7d2aca1a7d8b5c02b94f61c807120/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_5f2f7653e2732cdf542b8ac934e05352/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total sh-4.2# ``` storage02: ``` sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.73 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done 4.0K /var/lib/heketi/mounts/vg_e48e0dbb3dd65dd250ce3c997effe892/brick_34934b39c2999e4f55a0c89aecab5718/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 4.0K total 0 /var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_789c14aa897f8d1a75d0954ff2dd0227/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_00cd7128c612491996944fafb80bc8f9/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b00db39119354483417a28b5d55aaf1/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_b6dd1cbab3c28cb84a56ce1d65093935/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_2bd672a66df2636fa91cc255cf4accf9/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total sh-4.2# ``` storage03: ``` sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.166 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done 4.0K /var/lib/heketi/mounts/vg_f2e34e863e3a99b21011b4d8344ec848/brick_4cd6d96d7b40cd17b59c4950b28cf3bc/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 4.0K total 0 /var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_465fda0ee5c13e5d78e3279461e1fe99/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7677d50724618caa89055fdd921bb501/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_7076b52b2f6219162e8973b338859c4e/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_9bd4899a61d0998ef986e5d714a0df4f/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total 0 /var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c5a962f47423c1232d31e2c901d5de08/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 0 total sh-4.2# ``` Yesterday however, I could pull the image: ``` $ time docker pull docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas:saas-rc-190128-201902071645@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2: Pulling from da-saas-images/software-app-saas c325120ebc8d: Pull complete c9d123037991: Pull complete 521a1d24b010: Pull complete 0e82bb9f4fba: Pull complete aee3da753a34: Pull complete f8dee6aae8b9: Pull complete c147fa80b11a: Pull complete a60ea87d6b32: Pull complete Digest: sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 Status: Downloaded newer image for docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 real 5m2.891s user 0m0.136s sys 0m0.095s $ I suppose this happened again after adding storage03, what could we need to have a better idea ?
Csaba, What's the state of the patch to resolve the df issue discussed by Sonal in https://bugzilla.redhat.com/show_bug.cgi?id=1684823#c17 ? I'm just the TAM, not an engineer maintaining the code. Dan
> What's the state of the patch to resolve the df issue discussed by Sonal in https://bugzilla.redhat.com/show_bug.cgi?id=1684823#c17 ? Dan, this issue is resolved in version RHGS 3.4.1 (Batch Update 1). Is there anything more we should be looking at in this bug?
Closing with WORKSFORME (as per comment#17 etc). Please reopen if that is not the case.