Description of problem: As of now rebalance status just indicate the count of failed and skipped entries. This information should be more informative and should even give the list of files so that list can be used to manually operate and complete rebalancing. Version-Release number of selected component (if applicable): RHGS-3.2 Good to have list of files in a separate log file. Preferable would be to have file path and not gfid. We should log in such a way that it can be consumed easily even if the list is long. With this fix we should even create a way/script to consume the list and make the life of Admin easier. As of now doing it manually is a tough job. So here we have 3 requirements : 1) Get the list of files failed or skipped with error messages 2) Automatically consume those file list to complete rebalance( This can be triggered manually) 3) If not able to consume list, list the file with some reason of failure
*** Bug 1286183 has been marked as a duplicate of this bug. ***
Skipped files are now logged in the rebalance log : 2017-08-11 05:33:01.464620] I [MSGID: 109126] [dht-rebalance.c:2681:gf_defrag_migrate_single_file] 0-vol1-dht: File migration skipped for /NEWFILE-1. Users can search for the msgid [109126] for the list of skipped files. Upstream master: https://review.gluster.org/#/c/18021/
Verified this BZ on glusterfs version: 3.12.2-6.el7rhgs. With this fix, now in rebalance logs we are logging skipped files with full file path and users can search for the msgid [109126] in the rebalance logs for the list of skipped files. [root@dhcp~]# gluster v rebalance distrepx3 status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 317 1.1MB 3484 0 155 completed 0:02:38 node2 0 0Bytes 0 0 0 completed 0:02:23 node3 234 737.8KB 3350 0 257 completed 0:02:25 node4 3 14.6KB 71 0 6 completed 0:00:02 volume rebalance: distrepx3: success [root@dhcp ~]# grep -i 109126 /var/log/glusterfs/distrepx3-rebalance.log [2018-03-15 09:14:30.203393] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-fs-orangefs. [2018-03-15 09:14:31.262969] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-devices. [2018-03-15 09:14:31.842631] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-devices-system-cpu. [2018-03-15 09:14:33.733728] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/testing/sysfs-bus-fcoe. [2018-03-15 09:14:35.576404] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/testing/sysfs-bus-iio-frequency-ad9523. [2018-03-15 09:14:43.378480] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/DocBook/kgdb.tmpl. [2018-03-15 09:14:44.511774] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/DocBook/uio-howto.tmpl. [2018-03-15 09:14:44.534501] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/DocBook/usb.tmpl. [2018-03-15 09:14:44.780687] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/PCI/pci-error-recovery.txt. [2018-03-15 09:14:45.771193] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/RCU/NMI-RCU.txt. [2018-03-15 09:14:46.820019] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/RCU/UP.txt. [2018-03-15 09:14:48.607502] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/OMAP/omap_pm. [2018-03-15 09:14:48.965584] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/SA1100/serial_UART. [2018-03-15 09:14:49.844030] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/Samsung-S3C24XX/CPUfreq.txt. [2018-03-15 09:14:50.749548] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/pxa/mfp.txt. [2018-03-15 09:14:50.964903] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/stm32/overview.txt. [2018-03-15 09:14:51.207807] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/arm/IXP4xx. [2018-03-15 09:14:54.071332] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/block/capability.txt. [2018-03-15 09:14:56.791724] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/connector/connector.txt. [2018-03-15 09:14:59.671225] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/development-process/1.Intro.rst. [2018-03-15 09:15:02.314768] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/bcm/brcm,brcmstb.txt. [2018-03-15 09:15:04.795408] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/keystone/keystone.txt. [2018-03-15 09:15:05.238280] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/marvell/armada-cpu-reset.txt. [2018-03-15 09:15:05.619350] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/mediatek/mediatek,vdecsys.txt. [2018-03-15 09:15:06.619131] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/msm/ssbi.txt. [2018-03-15 09:15:10.884336] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/calxeda.txt. [2018-03-15 09:15:11.502137] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/pmu.txt. [2018-03-15 09:15:11.648964] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/fw-cfg.txt. [2018-03-15 09:15:11.940964] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/axxia.txt. [2018-03-15 09:15:12.102155] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/vt8500.txt. [2018-03-15 09:15:12.310015] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/oxnas.txt. [2018-03-15 09:15:12.580028] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/arm/olimex.txt. [2018-03-15 09:15:13.285985] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/ata/atmel-at91_cf.txt. [2018-03-15 09:15:13.599076] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt. [2018-03-15 09:15:14.425139] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/clock/st/st,clkgen-mux.txt. [2018-03-15 09:15:17.662556] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/clock/imx1-clock.txt. [2018-03-15 09:15:20.038224] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/clock/sunxi.txt. [2018-03-15 09:15:20.884185] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/bridge/ptn3460.txt. [2018-03-15 09:15:21.356088] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/exynos/exynos_dp.txt. [2018-03-15 09:15:21.429809] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/exynos/exynos7-decon.txt. [2018-03-15 09:15:21.525531] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/exynos/exynos_hdmi.txt. [2018-03-15 09:15:23.591882] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/auo,b101aw03.txt. [2018-03-15 09:15:24.233732] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/hannstar,hsd070pww1.txt. [2018-03-15 09:15:24.349799] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/lg,lp097qx1-spa1.txt. [2018-03-15 09:15:24.408820] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/lg,lp129qe.txt. [2018-03-15 09:15:24.609363] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/startek,startek-kd050c.txt. [2018-03-15 09:15:24.632719] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/panel/tpo,td043mtea1.txt. [2018-03-15 09:15:25.483129] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/display/ti/ti,omap5-dss.txt. [2018-03-15 09:15:27.028087] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/dma/xilinx/xilinx_dma.txt. [2018-03-15 09:15:27.158374] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/dma/atmel-xdma.txt. [2018-03-15 09:15:27.467756] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/dma/apm-xgene-dma.txt. [2018-03-15 09:15:30.518608] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/gpio/brcm,bcm6345-gpio.txt. [2018-03-15 09:15:32.714125] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/hwmon/nsa320-mcu.txt. [2018-03-15 09:15:35.486040] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/iio/adc/ti-adc128s052.txt. [2018-03-15 09:15:35.670893] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/iio/chemical/atlas,orp-sm.txt. [2018-03-15 09:15:40.486616] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/input/touchscreen/chipone_icn8318.txt. [2018-03-15 09:15:41.555065] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/input/nvidia,tegra20-kbc.txt. [2018-03-15 09:15:43.806259] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/leds/register-bit-led.txt. [2018-03-15 09:15:45.586354] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/i2c/adv7604.txt. [2018-03-15 09:15:46.233297] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/i2c/mt9m111.txt. [2018-03-15 09:15:46.829037] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/i2c/ths8200.txt. [2018-03-15 09:15:47.036949] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/i2c/nokia,smia.txt. [2018-03-15 09:15:47.261112] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/st,st-hva.txt. [2018-03-15 09:15:47.593924] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/media/s5p-cec.txt. [2018-03-15 09:15:48.806993] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/memory-controllers/ti-aemif.txt. [2018-03-15 09:15:49.644305] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mfd/atmel-smc.txt. [2018-03-15 09:15:49.678812] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mfd/88pm860x.txt. [2018-03-15 09:15:49.879130] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mfd/arizona.txt. [2018-03-15 09:15:50.544087] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mfd/tps65086.txt. [2018-03-15 09:15:50.638048] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mfd/stmpe.txt. [2018-03-15 09:15:52.279080] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/misc/ifm-csi.txt. [2018-03-15 09:15:53.601077] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mmc/mmci.txt. [2018-03-15 09:15:53.624783] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/mmc/sdhci-fujitsu.txt. [2018-03-15 09:15:54.947159] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/net/can/sja1000.txt. [2018-03-15 09:15:55.090450] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/net/can/sun4i_can.txt. [2018-03-15 09:15:55.832544] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/net/nfc/st21nfca.txt. [2018-03-15 09:15:57.929418] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/net/ralink,rt3050-esw.txt. [2018-03-15 09:16:00.358166] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt. [2018-03-15 09:16:02.184177] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/phy/berlin-sata-phy.txt. [2018-03-15 09:16:02.447930] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/phy/phy-miphy365x.txt. [2018-03-15 09:16:03.747017] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/phy/dm816x-phy.txt. [2018-03-15 09:16:03.808227] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/phy/rockchip-usb-phy.txt. [2018-03-15 09:16:03.911572] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/phy/rockchip-emmc-phy.txt. [2018-03-15 09:16:04.041112] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/brcm,nsp-pinmux.txt. [2018-03-15 09:16:04.687793] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/fsl,imx27-pinctrl.txt. [2018-03-15 09:16:05.466979] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/fsl,imx51-pinctrl.txt. [2018-03-15 09:16:07.250172] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/lantiq,pinctrl-xway.txt. [2018-03-15 09:16:07.340536] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/marvell,dove-pinctrl.txt. [2018-03-15 09:16:07.394108] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/pinctrl-atlas7.txt. [2018-03-15 09:16:07.603733] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pinctrl/nxp,lpc1850-scu.txt. [2018-03-15 09:16:07.867572] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/power/supply/isp1704.txt. [2018-03-15 09:16:08.750788] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/powerpc/4xx/reboot.txt. [2018-03-15 09:16:09.663339] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/powerpc/nintendo/wii.txt. [2018-03-15 09:16:09.792557] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pwm/cirrus,clps711x-pwm.txt. [2018-03-15 09:16:10.589563] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pwm/vt8500-pwm.txt. [2018-03-15 09:16:10.701808] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/pwm/pwm-mtk-disp.txt. [2018-03-15 09:16:11.441813] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/regulator/mt6397-regulator.txt. [2018-03-15 09:16:14.377303] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/epson,rx6110.txt. [2018-03-15 09:16:15.627127] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/rtc-omap.txt. [2018-03-15 09:16:15.802177] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/maxim-ds1302.txt. [2018-03-15 09:16:15.937771] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/rtc-palmas.txt. [2018-03-15 09:16:16.201088] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/olpc-xo1-rtc.txt. [2018-03-15 09:16:16.408046] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/rtc-st-lpc.txt. [2018-03-15 09:16:16.637521] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/rtc/xgene-rtc.txt. [2018-03-15 09:16:16.700194] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/scsi/hisilicon-sas.txt. [2018-03-15 09:16:16.786688] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/serial/arm,mps2-uart.txt. [2018-03-15 09:16:17.437383] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/serial/axis,etraxfs-uart.txt. [2018-03-15 09:16:18.268910] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/soc/fsl/cpm_qe/qe/ucc.txt. [2018-03-15 09:16:18.433346] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/soc/fsl/bman-portals.txt. [2018-03-15 09:16:19.476136] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/arndale.txt. [2018-03-15 09:16:19.538944] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/adi,adau17x1.txt. [2018-03-15 09:16:19.821721] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/cs42l73.txt. [2018-03-15 09:16:21.665767] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/ingenic,jz4740-i2s.txt. [2018-03-15 09:16:21.895564] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/max98095.txt. [2018-03-15 09:16:23.546962] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sound/rt5659.txt. [2018-03-15 09:16:24.758491] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/spi/spi_oc_tiny.txt. [2018-03-15 09:16:24.909817] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/spi/spi-sc18is602.txt. [2018-03-15 09:16:25.174912] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/spi/ti_qspi.txt. [2018-03-15 09:16:25.388017] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sram/sram.txt. [2018-03-15 09:16:25.565772] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/sram/rockchip-pmu-sram.txt. [2018-03-15 09:16:25.825209] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/staging/ion/hi6220-ion.txt. [2018-03-15 09:16:26.757935] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/timer/st,stih407-lpc. [2018-03-15 09:16:26.984016] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/timer/mediatek,mtk-timer.txt. [2018-03-15 09:16:27.427168] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/timer/marvell,orion-timer.txt. [2018-03-15 09:16:27.593070] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/timer/snps,archs-rtc.txt. [2018-03-15 09:16:29.609497] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/usb/ux500-usb.txt. [2018-03-15 09:16:31.675180] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/bindings/x86/timer.txt. [2018-03-15 09:16:31.679991] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/devicetree/dynamic-resolution-notes.txt. [2018-03-15 09:16:32.211632] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/driver-api/spi.rst. [2018-03-15 09:16:33.735904] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/fb/sm712fb.txt. [2018-03-15 09:16:33.866925] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/fb/sstfb.txt. [2018-03-15 09:16:51.281967] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/pohmelfs/design_notes.txt. [2018-03-15 09:16:51.445773] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/00-INDEX. [2018-03-15 09:16:51.450412] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/adfs.txt. [2018-03-15 09:16:51.540154] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/inotify.txt. [2018-03-15 09:16:51.641516] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/orangefs.txt. [2018-03-15 09:16:51.741710] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/bfs.txt. [2018-03-15 09:16:51.859166] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/devpts.txt. [2018-03-15 09:16:52.342070] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/udf.txt. [2018-03-15 09:16:52.443847] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/path-lookup.md. [2018-03-15 09:16:53.162221] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/xfs-self-describing-metadata.txt. [2018-03-15 09:16:53.282323] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/filesystems/romfs.txt. [2018-03-15 09:16:53.418249] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/fpga/fpga-mgr.txt. [2018-03-15 09:16:53.459040] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/frv/features.txt. [2018-03-15 09:16:53.865713] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/hid/hiddev.txt. [2018-03-15 09:16:54.499115] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/hwmon/pmbus. [2018-03-15 09:16:54.608149] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/i2c/busses/i2c-ali1535. [2018-03-15 09:16:54.757835] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/i2c/busses/i2c-ocores. [2018-03-15 09:16:55.396088] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ide/ChangeLog.ide-floppy.1996-2002. [2018-03-15 09:16:55.400299] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ide/ide-tape.txt. [2018-03-15 09:16:55.876282] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/infiniband/user_mad.txt. [2018-03-15 09:16:56.332186] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/ioctl/ioctl-number.txt. [2018-03-15 09:17:03.745924] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/DMA-API.txt. [2018-03-15 09:17:04.087889] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/Documentation/cachetlb.txt. [2018-03-15 09:17:04.281477] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-distrepx3-dht: File migration skipped for /linux-4.9.27/.cocciconfig. [root@dhcp ~]# grep -i 109126 /var/log/glusterfs/distrepx3-rebalance.log | wc -l 155 Moving this BZ to Verified state.
Nithya, How can we capture other failures too. If We are increasing the failed file count, we should be able to log the files for which this failed. For a user failed file count will matter, and they should be able to find all such files. As of now, it looks we can only search for a subset of it i.e which failed during migration. Please let us know your opinion.
(In reply to Bipin Kunal from comment #12) > Nithya, How can we capture other failures too. If We are increasing the > failed file count, we should be able to log the files for which this failed. > For a user failed file count will matter, and they should be able to find > all such files. As of now, it looks we can only search for a subset of it > i.e which failed during migration. Please let us know your opinion. I'm not sure I understand - when else can the file fail?
(In reply to Nithya Balachandran from comment #13) > (In reply to Bipin Kunal from comment #12) > > Nithya, How can we capture other failures too. If We are increasing the > > failed file count, we should be able to log the files for which this failed. > > For a user failed file count will matter, and they should be able to find > > all such files. As of now, it looks we can only search for a subset of it > > i.e which failed during migration. Please let us know your opinion. > > I'm not sure I understand - when else can the file fail? I was referring to your mail which is added in comment #10 Snippet: ================================================ That sounds like it is because this includes all the possible failures (not just file migration) such as fix layout errors because the bricks were killed. A little misleading , yes, but this is the current behaviour. ===============================================
(In reply to Bipin Kunal from comment #14) > (In reply to Nithya Balachandran from comment #13) > > (In reply to Bipin Kunal from comment #12) > > > Nithya, How can we capture other failures too. If We are increasing the > > > failed file count, we should be able to log the files for which this failed. > > > For a user failed file count will matter, and they should be able to find > > > all such files. As of now, it looks we can only search for a subset of it > > > i.e which failed during migration. Please let us know your opinion. > > > > I'm not sure I understand - when else can the file fail? > > I was referring to your mail which is added in comment #10 > > Snippet: > > ================================================ > That sounds like it is because this includes all the possible failures (not > just file migration) such as fix layout errors because the bricks were > killed. A little misleading , yes, but this is the current behaviour. > > =============================================== Ok, let me clarify this. Currently the Rebalance failure count includes * File migration failures - the file was picked up for migration but one of the steps failed (maybe we could not lock or a write failed). These do not cause a rebalance process failure. - Msg logged: "migrate-data failed for <filename>" * Skipped files only in case of remove-brick. Does not cause a rebalance process failure. * Directory fix-layout failures. These cause the entire rebalance process to fail as well. - Msg logged : Fix layout failed for <dirname> - This will abort the rebalance process and set the status to failed. - Does not consider ENOENT or ESTALE a failure which can cause issues with renames. What is not captured: In most cases, ENOENT and ESTALE are not considered failures. This can cause issues with renames. What is not captured at the moment:
Updated the doc text. Kindly review.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607