Bug 1594205

We are having the following setup:
#1 Host with IP 10.101.1.218/24
This sever is hosting docker registry v2_registry_1:
# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                  PORTS                     NAMES
49889b385ae7        ssp12/ssp/registry:v2   "/usr/local/sbin/run-"   19 hours ago        Up 19 hours (healthy)   0.0.0.0:49652->5000/tcp   v2_registry_1
e81318714566        ssp12/ssp/inst_srv:v3   "bash /start.sh"         19 hours ago        Up 19 hours                                       v3_inst_srv_1

v2_registry_1 is using docker default network. This container IP ADDR is: 172.17.0.2
Please see v2_registry_1_inspect.txt file for details
# docker inspect v2_registry_1 > v2_registry_1_inspect.txt

Container v3_inst_srv_1 is using host networking as it requires layer two access to the network.
So the only container in docker network is v2_registry_1

#2 Host with IP 10.101.1.216/24
This is server with docker client. We wish to pull images from registry hosted by #1 to it.


Issue description:
When we try to connect to docker registry locally from 10.101.1.218 it is working fine.
I'm using curl with -k in examples to eliminate certificates.

On #1:
ad):(root) 04:33:19 CDT DESU-LAUNCHPAD-07.00.00.80-15
# curl -k -X GET https://10.101.1.218:49652/v2/_catalog
{"repositories":["ssp11/ssp/bare","ssp11/ssp/base","ssp11/ssp/registry","ssp11/ssp/site_mgr","ssp11/ssp/yum.repo","ssp12/ssp/bare","ssp12/ssp/base","ssp12/ssp/registry","ssp12/ssp/site_mgr","ssp12/ssp/yum.repo"]}

(launchpad):(root) 04:43:29 CDT DESU-LAUNCHPAD-07.00.00.80-15
# curl -k -X GET https://127.0.0.1:49652/v2/_catalog
{"repositories":["ssp11/ssp/bare","ssp11/ssp/base","ssp11/ssp/registry","ssp11/ssp/site_mgr","ssp11/ssp/yum.repo","ssp12/ssp/bare","ssp12/ssp/base","ssp12/ssp/registry","ssp12/ssp/site_mgr","ssp12/ssp/yum.repo"]}


On #2
[root@z001s001ssp01 ~]#  curl -k --connect-timeout 7 -X GET https://10.101.1.218:49652/v2/_catalog
curl: (28) Connection timed out after 7001 milliseconds

I have added --connect-timeout parameter only to shorten time I need to wait to get failure.


I have tried to find out where the frames are dropped.
Please see below taken steps and logs.

On #1
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:17:4e:e7 brd ff:ff:ff:ff:ff:ff
    inet 100.100.100.10/24 brd 100.100.100.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:17:4e:f1 brd ff:ff:ff:ff:ff:ff
    inet 10.101.1.218/24 brd 10.101.1.255 scope global eth1:1
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:4c:d1:81:89 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
108: vetha449c56@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
    link/ether 36:cf:9f:cf:92:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0

vetha449c56 - this is v2_registry_1 net interface.

I have run tcpdump on eth1, docker0, vetha449c56.
# tcpdump -i eth1 -w  ./tcpdump_eth1.cap
# tcpdump -i docker0 -w ./tcpdump_docker0.cap
# tcpdump -i  vetha449c56 -w ./tcpdump_registry.cap
Dumps are attached.

# iptables --zero
# iptables-save -c > before.txt

On #2
[root@z001s001ssp01 ~]#  curl -k --connect-timeout 7 -X GET https://10.101.1.218:49652/v2/_catalog
curl: (28) Connection timed out after 7001 millisecond

On #1
# iptables-save -c > after.txt

# conntrack -L > conntrack.txt
conntrack v1.4.4 (conntrack-tools): 10 flow entries have been shown.



Now I take logs for comparison when trying to connect to registry from local hosts
# iptables --zero
# iptables-save -c > before_local.txt

# curl -k --connect-timeout 5 -X GET https://10.101.1.218:49652/v2/_catalog
{"repositories":["ssp11/ssp/bare","ssp11/ssp/base","ssp11/ssp/registry","ssp11/ssp/site_mgr","ssp11/ssp/yum.repo","ssp12/ssp/bare","ssp12/ssp/base","ssp12/ssp/registry","ssp12/ssp/site_mgr","ssp12/ssp/yum.repo"]}
So if calling command from local PC it works.

# iptables-save -c > after_local.txt
# conntrack -L > conntrack_local.txt



So when trying to connect from external PC, tcpdump started on registry container interface did not catch any frame.
At the same time iptables is showing that 3 frames were send to 172.17.0.2

before.txt: [253:14592] -A DOCKER ! -i docker0 -p tcp -m tcp --dport 49652 -j DNAT --to-destination 172.17.0.2:5000
vs
after.txt: [256:14772] -A DOCKER ! -i docker0 -p tcp -m tcp --dport 49652 -j DNAT --to-destination 172.17.0.2:5000mv 

In test above I have stopped tcpdump before calling "iptables-save -c ". I have repated the test doing this in reverse order, to make sure that frames were not send after I have stopped tcpdump. The result was exactly the same.

Why frames are not seen inside the container even iptables is showing there were forwarded to it?
Why issue is impacting only traffic from external world?

Issue can be fixed by stopping/starting registry container. 
Issue has been seen only twice so far. This is around 5 months time period. During each day many tests with registry are done. In each registry is created from scratch.