Attachment 1481699 Details for Bug 1624527 – Hotfix_1624527_report

Hotfix_1624527_report

Hotfix_1624527_report (text/plain), 24.19 KB, created by Ramakrishnan Periyasamy on 2018-09-08 05:41:59 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ramakrishnan Periyasamy

Created: 2018-09-08 05:41:59 UTC

Size: 24.19 KB

patch

obsolete

>Hotfix_1624527 Report:
>-------------------------
>
>Configured cluster with 3.0z5 GA based build and able to reproduce the issue. Console output and logs added below.
>
>Note: Client IO was running continuously during this entire test execution
>IO: for i in {1..100};do sudo cp -av /usr /mnt/fuse/fuse/dir4/.; done
>
>Cluster verison before upgrade:
>-------------------------------
>[ubuntu@magna051 ~]$ ceph -v
>ceph version 12.2.4-42.el7cp (f73642baacccbf2a3c254d1fb5f0317b933b28cf) luminous (stable)
>[ubuntu@magna051 ~]$ 
>[ubuntu@magna051 ~]$ sudo ceph -s
>  cluster:
>    id:     366f0928-0367-4fb0-ab6b-939700f59f40
>    health: HEALTH_OK
> 
>  services:
>    mon: 1 daemons, quorum magna051
>    mgr: magna051(active)
>    mds: cephfs-2/2/2 up  {0=magna092=up:active,1=magna113=up:active}, 1 up:standby
>    osd: 9 osds: 9 up, 9 in
> 
>  data:
>    pools:   2 pools, 128 pgs
>    objects: 244k objects, 37363 MB
>    usage:   114 GB used, 8220 GB / 8334 GB avail
>    pgs:     128 active+clean
> 
>  io:
>    client:   683 kB/s wr, 0 op/s rd, 162 op/s wr
>
>[ubuntu@magna092 ~]$ sudo ss -tnp | grep ceph-mds
>ESTAB      0      0      10.8.128.92:60148              10.8.128.68:6808                users:(("ceph-mds",pid=3430,fd=31))
>ESTAB      0      0      10.8.128.92:32838              10.8.128.56:6808                users:(("ceph-mds",pid=3430,fd=30))
>ESTAB      0      0      10.8.128.92:44122              10.8.128.51:6789                users:(("ceph-mds",pid=3430,fd=17))
>ESTAB      0      0      10.8.128.92:47418              10.8.128.56:6800                users:(("ceph-mds",pid=3430,fd=34))
>ESTAB      0      0      10.8.128.92:37116              10.8.128.68:6804                users:(("ceph-mds",pid=3430,fd=28))
>ESTAB      0      0      10.8.128.92:56194              10.8.128.51:6800                users:(("ceph-mds",pid=3430,fd=26))
>ESTAB      0      0      10.8.128.92:6800               10.8.128.70:43614               users:(("ceph-mds",pid=3430,fd=37))
>ESTAB      0      0      10.8.128.92:47384              10.8.128.52:6808                users:(("ceph-mds",pid=3430,fd=35))
>ESTAB      0      0      10.8.128.92:53148              10.8.128.56:6804                users:(("ceph-mds",pid=3430,fd=29))
>ESTAB      0      0      10.8.128.92:55922              10.8.128.68:6800                users:(("ceph-mds",pid=3430,fd=27))
>ESTAB      0      0      10.8.128.92:60522              10.8.128.52:6804                users:(("ceph-mds",pid=3430,fd=32))
>ESTAB      0      0      10.8.128.92:48830              10.8.128.52:6800                users:(("ceph-mds",pid=3430,fd=33))
>(reverse-i-search)`d': sudo ss -tnp | grep ceph-m^C
>[ubuntu@magna092 ~]$ sudo iptables -I INPUT -p tcp --sport 44122 --dport 6789 -j DROP 
>
>Mds log:
>----------------
>2018-09-08 04:47:21.867619 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15782
>2018-09-08 04:47:25.867723 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15783
>2018-09-08 04:47:28.867744 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.000240 > 15 since last acked beacon
>2018-09-08 04:47:28.869684 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002181 > 15 since last acked beacon
>2018-09-08 04:47:28.869698 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002194 > 15 since last acked beacon
>2018-09-08 04:47:28.869700 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002197 > 15 since last acked beacon
>2018-09-08 04:47:29.280596 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413092 > 15 since last acked beacon
>2018-09-08 04:47:29.280619 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413115 > 15 since last acked beacon
>2018-09-08 04:47:29.280622 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413118 > 15 since last acked beacon
>2018-09-08 04:47:29.280640 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413136 > 15 since last acked beacon
>2018-09-08 04:47:29.280649 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413145 > 15 since last acked beacon
>2018-09-08 04:47:29.280651 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413147 > 15 since last acked beacon
>2018-09-08 04:47:29.280665 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413161 > 15 since last acked beacon
>2018-09-08 04:47:29.280674 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413170 > 15 since last acked beacon
>2018-09-08 04:47:29.280676 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413173 > 15 since last acked beacon
>2018-09-08 04:47:29.280690 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413186 > 15 since last acked beacon
>2018-09-08 04:47:29.280698 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413194 > 15 since last acked beacon
>2018-09-08 04:47:29.280701 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413197 > 15 since last acked beacon
>2018-09-08 04:47:29.280732 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413228 > 15 since last acked beacon
>2018-09-08 04:47:29.280747 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413243 > 15 since last acked beacon
>2018-09-08 04:47:29.280750 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413246 > 15 since last acked beacon
>2018-09-08 04:47:29.280808 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413304 > 15 since last acked beacon
>2018-09-08 04:47:29.280819 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413316 > 15 since last acked beacon
>2018-09-08 04:47:29.280822 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413318 > 15 since last acked beacon
>2018-09-08 04:47:29.280905 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413401 > 15 since last acked beacon
>2018-09-08 04:47:29.280917 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413413 > 15 since last acked beacon
>2018-09-08 04:47:29.280919 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413416 > 15 since last acked beacon
>2018-09-08 04:47:29.280991 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413487 > 15 since last acked beacon
>2018-09-08 04:47:29.281008 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413504 > 15 since last acked beacon
>2018-09-08 04:47:29.281012 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413508 > 15 since last acked beacon
>2018-09-08 04:47:29.281069 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413565 > 15 since last acked beacon
>2018-09-08 04:47:29.281085 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413581 > 15 since last acked beacon
>2018-09-08 04:47:29.281088 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413584 > 15 since last acked beacon
>2018-09-08 04:47:29.281103 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413599 > 15 since last acked beacon
>2018-09-08 04:47:29.281112 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413608 > 15 since last acked beacon
>2018-09-08 04:47:29.281115 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413611 > 15 since last acked beacon
>2018-09-08 04:47:29.281161 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413657 > 15 since last acked beacon
>2018-09-08 04:47:29.281175 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413671 > 15 since last acked beacon
>2018-09-08 04:47:29.281178 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413674 > 15 since last acked beacon
>2018-09-08 04:47:29.281214 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413711 > 15 since last acked beacon
>2018-09-08 04:47:29.281226 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413722 > 15 since last acked beacon
>2018-09-08 04:47:29.281229 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413725 > 15 since last acked beacon
>2018-09-08 04:47:29.281290 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413786 > 15 since last acked beacon
>2018-09-08 04:47:29.281304 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413800 > 15 since last acked beacon
>2018-09-08 04:47:29.281306 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413803 > 15 since last acked beacon
>2018-09-08 04:47:29.281344 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413840 > 15 since last acked beacon
>2018-09-08 04:47:29.281356 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413852 > 15 since last acked beacon
>2018-09-08 04:47:29.281359 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413855 > 15 since last acked beacon
>2018-09-08 04:47:29.281407 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413903 > 15 since last acked beacon
>2018-09-08 04:47:29.281418 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413914 > 15 since last acked beacon
>2018-09-08 04:47:29.281421 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413917 > 15 since last acked beacon
>2018-09-08 04:47:29.867832 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15784
>2018-09-08 04:47:30.280741 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413237 > 15 since last acked beacon
>2018-09-08 04:47:30.280759 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413255 > 15 since last acked beacon
>2018-09-08 04:47:30.280762 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413259 > 15 since last acked beacon
>2018-09-08 04:47:30.280781 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413277 > 15 since last acked beacon
>2018-09-08 04:47:30.280792 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413289 > 15 since last acked beacon
>2018-09-08 04:47:30.280796 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413292 > 15 since last acked beacon
>2018-09-08 04:47:30.280813 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413309 > 15 since last acked beacon
>2018-09-08 04:47:30.280824 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413320 > 15 since last acked beacon
>2018-09-08 04:47:30.280827 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413323 > 15 since last acked beacon
>2018-09-08 04:47:30.280839 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413335 > 15 since last acked beacon
>2018-09-08 04:47:30.280847 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413343 > 15 since last acked beacon
>2018-09-08 04:47:30.280850 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413346 > 15 since last acked beacon
>2018-09-08 04:47:30.280862 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413358 > 15 since last acked beacon
>2018-09-08 04:47:30.280870 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413366 > 15 since last acked beacon
>2018-09-08 04:47:30.280872 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413369 > 15 since last acked beacon
>2018-09-08 04:47:30.280885 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413382 > 15 since last acked beacon
>2018-09-08 04:47:30.280894 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413390 > 15 since last acked beacon
>2018-09-08 04:47:30.280896 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413393 > 15 since last acked beacon
>2018-09-08 04:47:30.280954 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413451 > 15 since last acked beacon
>2018-09-08 04:47:30.280964 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413460 > 15 since last acked beacon
>2018-09-08 04:47:30.280966 7fbe315ae700  5 mds.beacon.magna092 is_laggy 16.413462 > 15 since last acked beacon
>  -163> 2018-09-08 04:47:28.867744 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.000240 > 15 since last acked beacon
>  -160> 2018-09-08 04:47:28.869684 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002181 > 15 since last acked beacon
>  -158> 2018-09-08 04:47:28.869698 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002194 > 15 since last acked beacon
>  -157> 2018-09-08 04:47:28.869700 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.002197 > 15 since last acked beacon
>  -153> 2018-09-08 04:47:29.280596 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413092 > 15 since last acked beacon
>  -151> 2018-09-08 04:47:29.280619 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413115 > 15 since last acked beacon
>  -150> 2018-09-08 04:47:29.280622 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413118 > 15 since last acked beacon
>  -147> 2018-09-08 04:47:29.280640 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413136 > 15 since last acked beacon
>  -145> 2018-09-08 04:47:29.280649 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413145 > 15 since last acked beacon
>  -144> 2018-09-08 04:47:29.280651 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413147 > 15 since last acked beacon
>  -142> 2018-09-08 04:47:29.280665 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413161 > 15 since last acked beacon
>  -139> 2018-09-08 04:47:29.280674 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413170 > 15 since last acked beacon
>  -138> 2018-09-08 04:47:29.280676 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413173 > 15 since last acked beacon
>  -136> 2018-09-08 04:47:29.280690 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413186 > 15 since last acked beacon
>  -134> 2018-09-08 04:47:29.280698 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413194 > 15 since last acked beacon
>  -133> 2018-09-08 04:47:29.280701 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413197 > 15 since last acked beacon
>  -130> 2018-09-08 04:47:29.280732 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413228 > 15 since last acked beacon
>  -128> 2018-09-08 04:47:29.280747 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413243 > 15 since last acked beacon
>  -127> 2018-09-08 04:47:29.280750 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413246 > 15 since last acked beacon
>  -124> 2018-09-08 04:47:29.280808 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413304 > 15 since last acked beacon
>  -122> 2018-09-08 04:47:29.280819 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413316 > 15 since last acked beacon
>  -121> 2018-09-08 04:47:29.280822 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413318 > 15 since last acked beacon
>  -118> 2018-09-08 04:47:29.280905 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413401 > 15 since last acked beacon
>  -116> 2018-09-08 04:47:29.280917 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413413 > 15 since last acked beacon
>  -115> 2018-09-08 04:47:29.280919 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413416 > 15 since last acked beacon
>  -112> 2018-09-08 04:47:29.280991 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413487 > 15 since last acked beacon
>  -110> 2018-09-08 04:47:29.281008 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413504 > 15 since last acked beacon
>  -109> 2018-09-08 04:47:29.281012 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413508 > 15 since last acked beacon
>  -105> 2018-09-08 04:47:29.281069 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413565 > 15 since last acked beacon
>  -103> 2018-09-08 04:47:29.281085 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413581 > 15 since last acked beacon
>  -102> 2018-09-08 04:47:29.281088 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413584 > 15 since last acked beacon
>  -100> 2018-09-08 04:47:29.281103 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413599 > 15 since last acked beacon
>   -98> 2018-09-08 04:47:29.281112 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413608 > 15 since last acked beacon
>   -97> 2018-09-08 04:47:29.281115 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413611 > 15 since last acked beacon
>   -94> 2018-09-08 04:47:29.281161 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413657 > 15 since last acked beacon
>   -92> 2018-09-08 04:47:29.281175 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413671 > 15 since last acked beacon
>   -91> 2018-09-08 04:47:29.281178 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413674 > 15 since last acked beacon
>   -88> 2018-09-08 04:47:29.281214 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413711 > 15 since last acked beacon
>   -86> 2018-09-08 04:47:29.281226 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413722 > 15 since last acked beacon
>   -85> 2018-09-08 04:47:29.281229 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413725 > 15 since last acked beacon
>   -82> 2018-09-08 04:47:29.281290 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413786 > 15 since last acked beacon
>   -80> 2018-09-08 04:47:29.281304 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413800 > 15 since last acked beacon
>   -79> 2018-09-08 04:47:29.281306 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413803 > 15 since last acked beacon
>   -76> 2018-09-08 04:47:29.281344 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413840 > 15 since last acked beacon
>   -74> 2018-09-08 04:47:29.281356 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413852 > 15 since last acked beacon
>   -73> 2018-09-08 04:47:29.281359 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413855 > 15 since last acked beacon
>   -70> 2018-09-08 04:47:29.281407 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413903 > 15 since last acked beacon
>   -68> 2018-09-08 04:47:29.281418 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413914 > 15 since last acked beacon
>   -67> 2018-09-08 04:47:29.281421 7fbe315ae700  5 mds.beacon.magna092 is_laggy 15.413917 > 15 since last acked beacon
>   -66> 2018-09-08 04:47:29.867832 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15784
>   -64> 2018-09-08 04:47:29.867853 7fbe2e5a8700  1 -- 10.8.128.92:6800/4096239442 --> 10.8.128.51:6789/0 -- mdsbeacon(14183/magna092 up:active seq 15784 v194) v7 -- 0x558cf182e000 con 0
>
>
>Cluster Upgrade:
>-----------------------
>Upgraded Cluster to hotfix build with IO from client. 
>Followed MDS upgrade procedures https://github.com/ceph/ceph/blob/luminous/doc/cephfs/upgrading.rst
>
>After upgrade
>------------------------
>[ubuntu@magna051 ~]$ ceph -v
>ceph version 12.2.4-42.1.hotfix.nvidia.el7cp (4a72ecd06cdc5a049945b166073ce39fbe631308) luminous (stable)
>[ubuntu@magna051 ~]$ sudo ceph -s
>  cluster:
>    id:     366f0928-0367-4fb0-ab6b-939700f59f40
>    health: HEALTH_OK
> 
>  services:
>    mon: 1 daemons, quorum magna051
>    mgr: magna051(active)
>    mds: cephfs-2/2/2 up  {0=magna113=up:active,1=magna092=up:active}, 1 up:standby
>    osd: 9 osds: 9 up, 9 in
> 
>  data:
>    pools:   2 pools, 128 pgs
>    objects: 249k objects, 37274 MB
>    usage:   113 GB used, 8220 GB / 8334 GB avail
>    pgs:     128 active+clean
> 
>  io:
>    client:   511 B/s rd, 209 kB/s wr, 0 op/s rd, 76 op/s wr
> 
>[ubuntu@magna051 ~]$ sudo ceph fs status
>cephfs - 1 clients
>======
>+------+--------+----------+---------------+-------+-------+
>| Rank | State  |   MDS    |    Activity   |  dns  |  inos |
>+------+--------+----------+---------------+-------+-------+
>|  0   | active | magna113 | Reqs:    0 /s | 36.9k | 32.8k |
>|  1   | active | magna114 | Reqs:    0 /s | 33.2k | 29.4k |
>+------+--------+----------+---------------+-------+-------+
>+-----------------+----------+-------+-------+
>|       Pool      |   type   |  used | avail |
>+-----------------+----------+-------+-------+
>| cephfs_metadata | metadata |  377M | 2591G |
>|   cephfs_data   |   data   | 36.0G | 2591G |
>+-----------------+----------+-------+-------+
>
>+-------------+
>| Standby MDS |
>+-------------+
>|   magna092  |
>+-------------+
>MDS version: ceph version 12.2.4-42.1.hotfix.nvidia.el7cp (4a72ecd06cdc5a049945b166073ce39fbe631308) luminous (stable)
> 
>[ubuntu@magna092 ~]$ sudo ss -tnp | grep ceph-mds
>ESTAB      0      0      10.8.128.92:60256              10.8.128.68:6808                users:(("ceph-mds",pid=6214,fd=26))
>ESTAB      0      0      10.8.128.92:56036              10.8.128.68:6800                users:(("ceph-mds",pid=6214,fd=27))
>ESTAB      0      0      10.8.128.92:37242              10.8.128.68:6804                users:(("ceph-mds",pid=6214,fd=34))
>ESTAB      0      0      10.8.128.92:32972              10.8.128.56:6808                users:(("ceph-mds",pid=6214,fd=33))
>ESTAB      0      0      10.8.128.92:48956              10.8.128.52:6800                users:(("ceph-mds",pid=6214,fd=30))
>ESTAB      0      0      10.8.128.92:44240              10.8.128.51:6789                users:(("ceph-mds",pid=6214,fd=17))
>ESTAB      0      0      10.8.128.92:47544              10.8.128.56:6800                users:(("ceph-mds",pid=6214,fd=31))
>ESTAB      0      0      10.8.128.92:53276              10.8.128.56:6804                users:(("ceph-mds",pid=6214,fd=29))
>ESTAB      0      0      10.8.128.92:56338              10.8.128.51:6800                users:(("ceph-mds",pid=6214,fd=35))
>ESTAB      0      0      10.8.128.92:6800               10.8.128.114:35452               users:(("ceph-mds",pid=6214,fd=38))
>ESTAB      0      0      10.8.128.92:60650              10.8.128.52:6804                users:(("ceph-mds",pid=6214,fd=28))
>ESTAB      0      0      10.8.128.92:47510              10.8.128.52:6808                users:(("ceph-mds",pid=6214,fd=32))
>ESTAB      0      0      10.8.128.92:6800               10.8.128.70:44872               users:(("ceph-mds",pid=6214,fd=36))
>[ubuntu@magna092 ~]$ sudo iptables -I OUTPUT -p tcp --sport 44240 --dport 6789 -j DROP
>
>MDS logs:
>------------------------
>2018-09-08 05:24:52.828060 7f8564705700  5 mds.beacon.magna092 handle_mds_beacon up:active seq 200 rtt 0.000579
>2018-09-08 05:24:56.827548 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 201
>2018-09-08 05:25:00.827659 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 202
>2018-09-08 05:25:04.827741 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 203
>2018-09-08 05:25:07.851715 7f8561f85700  1 mds.beacon.magna092 MDS connection to Monitors appears to be laggy; 15.024234s since last acked beacon
>2018-09-08 05:25:08.827859 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 204
> -9265> 2018-09-08 05:24:48.827390 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 199
> -9263> 2018-09-08 05:24:48.827409 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 199 v239) v7 -- 0x557acb654380 con 0
> -9262> 2018-09-08 05:24:48.827800 7f8564705700  5 -- 10.8.128.92:6800/3178541370 >> 10.8.128.51:6789/0 conn(0x557ac9445000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=124 cs=1 l=1). rx mon.0 seq 217 0x557acb654380 mdsbeacon(24206/magna092 up:active seq 199 v239) v7
> -9261> 2018-09-08 05:24:48.827818 7f8564705700  1 -- 10.8.128.92:6800/3178541370 <== mon.0 10.8.128.51:6789/0 217 ==== mdsbeacon(24206/magna092 up:active seq 199 v239) v7 ==== 133+0+0 (3166973937 0 0) 0x557acb654380 con 0x557ac9445000
> -9260> 2018-09-08 05:24:48.827827 7f8564705700  5 mds.beacon.magna092 handle_mds_beacon up:active seq 199 rtt 0.000428
> -7056> 2018-09-08 05:24:52.827470 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 200
> -7054> 2018-09-08 05:24:52.827490 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 200 v239) v7 -- 0x557acb655180 con 0
> -7053> 2018-09-08 05:24:52.828035 7f8564705700  5 -- 10.8.128.92:6800/3178541370 >> 10.8.128.51:6789/0 conn(0x557ac9445000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=124 cs=1 l=1). rx mon.0 seq 218 0x557acb655180 mdsbeacon(24206/magna092 up:active seq 200 v239) v7
> -7052> 2018-09-08 05:24:52.828052 7f8564705700  1 -- 10.8.128.92:6800/3178541370 <== mon.0 10.8.128.51:6789/0 218 ==== mdsbeacon(24206/magna092 up:active seq 200 v239) v7 ==== 133+0+0 (4218276460 0 0) 0x557acb655180 con 0x557ac9445000
> -7051> 2018-09-08 05:24:52.828060 7f8564705700  5 mds.beacon.magna092 handle_mds_beacon up:active seq 200 rtt 0.000579
> -6856> 2018-09-08 05:24:56.827548 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 201
> -6854> 2018-09-08 05:24:56.827571 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 201 v239) v7 -- 0x557acb655500 con 0
> -3585> 2018-09-08 05:25:00.827659 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 202
> -3583> 2018-09-08 05:25:00.827677 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 202 v239) v7 -- 0x557ad1bd0380 con 0
> -2129> 2018-09-08 05:25:04.827741 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 203
> -2127> 2018-09-08 05:25:04.827760 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 203 v239) v7 -- 0x557ad1bd0700 con 0
>   -32> 2018-09-08 05:25:07.851715 7f8561f85700  1 mds.beacon.magna092 MDS connection to Monitors appears to be laggy; 15.024234s since last acked beacon
>   -30> 2018-09-08 05:25:08.827859 7f855ef7f700  5 mds.beacon.magna092 _send up:active seq 204
>   -28> 2018-09-08 05:25:08.827878 7f855ef7f700  1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 204 v239) v7 -- 0x557ad1bd1180 con 0
>2018-09-08 05:25:11.143245 7f0591e38240  5 mds.beacon.magna092 _send up:boot seq 1
>2018-09-08 05:25:15.143436 7f0587f6a700  5 mds.beacon.magna092 _send up:boot seq 2
>2018-09-08 05:25:15.553912 7f058af70700  5 mds.beacon.magna092 set_want_state: up:boot -> up:standby
>2018-09-08 05:25:15.554028 7f058d6f0700  5 mds.beacon.magna092 handle_mds_beacon up:boot seq 2 rtt 0.410583
>2018-09-08 05:25:19.143638 7f0587f6a700  5 mds.beacon.magna092 _send up:standby seq 3
>2018-09-08 05:25:19.144232 7f058d6f0700  5 mds.beacon.magna092 handle_mds_beacon up:standby seq 3 rtt 0.000582
>
>

Actions: View

Attachments on bug 1624527: 1481699