Bug 1269191

Summary: Skydns cannot bind to port 53
Product: [Fedora] Fedora Reporter: Victor Costan <costan>
Component: golang-github-skynetservices-skydnsAssignee: FridolĂ­n PokornĂ˝ <fpokorny>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: costan, fpokorny, golang-updates, jchaloup, lsm5
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc24 golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc23 golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-04 18:53:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Victor Costan 2015-10-06 15:19:17 UTC
Description of problem:
By default, skydns runs under its own user, so it cannot bind to port 53.

Version-Release number of selected component (if applicable): 0.7.git6c94cbe.fc22
This is the version in updates-testing for Fedora 22. It seems like it needs more karma to be promoted to stable.

How reproducible: always


Steps to Reproduce:
1. dnf install skydns
2. systemctl start skydns
3. systemctl status skydns
4. dig @127.0.0.1 skydns.local

Actual results:
skydns errors out because it can't bind to port 53.
I can't issue DNS queries.

Expected results: I should be able to issue DNS queries.

Additional info:

I worked around this problem by creating a systemd socket file, and using socket activation. skydns supports this. I'll be happy to provide additional details, if it helps. I think this is a good solution, as it avoids having to run skydns as root.

Comment 1 Jan Chaloupka 2015-10-07 14:12:31 UTC
Hi Victor,

thank you for reporting the issue. The workaround is appreciated, can you share?

Jan

Comment 2 Victor Costan 2015-10-07 18:38:20 UTC
My workaround is documented below. After following these steps, skydns works perfectly for me. Thank you very much for packaging it!

1) I wrote the following in /etc/systemd/system/skydns.socket (the package would presumably use /usr/lib/systemd/system/skydns.socket instead):

[Socket]
ListenStream=0.0.0.0:53
ListenDatagram=0.0.0.0:53

[Install]
WantedBy=sockets.target

2) I set systemd to true in skydns's etcd config:

etcdctl set /skydns/config \
    '{"ttl": 3600, "systemd": true, "domain": "skydns.local."}'

I also submitted a PR to skydns to make it possible to set an environment variable instead, which could be done in skydns.conf.

https://github.com/skynetservices/skydns/pull/229

The skydns package could also interpolate an env var like OTHER_OPTIONS in the skydns command line, along the lines of what the docker package does. Specifically, in skydns.service

ExecStart=/usr/bin/skydns $OTHER_OPTIONS

This way, one could set OTHER_OPTIONS="--systemd" in skydns.conf

3) I manually enabled and started the socket. I'm not sure that's necessary.

systemctl enable skydns.socket
systemctl start skydns.socket

Comment 3 Jan Chaloupka 2015-10-21 12:42:27 UTC
At the current state of skydns's dependencies, I can rebuilt skydns. What version of skydns would you like to have in Fedora? The latest is 2.4.0c. Not sure how stable it is. Have you had a change to play with upstream build?

[1] https://github.com/skynetservices/skydns/releases/tag/2.4.0c

Comment 4 Victor Costan 2015-10-23 06:18:41 UTC
I haven't had a chance to play with the upstream build. I'd be happy to do so, if it helps!

For my needs, the version in Fedora has been working out flawlessly, modulo this bug. 

I'd wait until we have a resolution on my PR. If the author decides to merge it, I think it's worth updating straight to the relevant git revision in the repository. 

Socket-activation aside, it seems to me that there aren't many improvements between 2.4.0a and 2.4.0c, so updating skydns might not be worth the testing effort if we can't get the extra socket activation option.

Comment 5 Jan Chaloupka 2015-10-26 09:45:43 UTC
My typo, it meant the current skydns can not be rebuilt as its dependencies have been updated to the state they break backward compatibility with the skydns. The only way how to patch it is to built one of the latest releases. At least in rawhide. If it will work for you, it can be promoted to f23 and f22.

Once all deps are updated and packaged, it is fairly straightforward to update it to newer version. Though, not aware how much skydns changes its set of deps.

Comment 6 Victor Costan 2015-10-27 01:27:04 UTC
Sorry for misunderstanding! In that case, should I wait for an updated skydns package to test, or would you prefer that I build 2.4.0c from upstream on my own and try it out?

I'll be happy to help out with testing either way!

Comment 7 Jan Chaloupka 2015-10-27 19:51:14 UTC
The build is ready for testing [1]. Good luck and thanks :).

[1] http://koji.fedoraproject.org/koji/buildinfo?buildID=694593

Comment 8 Victor Costan 2015-10-28 00:03:58 UTC
It worked for me!

I uninstalled the old skydns and installed this one, then replaced /etc/skydns/skydns.conf with my version. Skydns is answering queries just like before, and my cluster is stable. This is ready to ship, as far as I'm concerned.

Thank you very much for all the changes!

Comment 9 Victor Costan 2015-10-28 00:05:39 UTC
I'm ready to +1 the update on bodhi, if it helps.

Comment 10 Fedora Update System 2015-10-28 15:27:04 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-aadc2e2de8

Comment 11 Fedora Update System 2015-10-28 15:27:04 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-14c8299f15

Comment 12 Victor Costan 2015-10-31 02:46:28 UTC
I tested both the fc22 and the fc23 versions and +1ed them in Bodhi.

Can you please release the update to testing or stable, so I can have my deployment scripts get it straight from dnf?

Sorry if I'm referencing the wrong procedure. I looked at this page [1], because Bodhi lists the updates as "pending".

Thank you very very much for the fix!

Context: I worked at Google around 2009, and I've been really missing the Borg DNS service. Skydns brings me the same convenience, and it's super-easy to set up now, thanks to your fix!

[1] https://fedoraproject.org/wiki/Bodhi#Pending

Comment 13 Jan Chaloupka 2015-10-31 10:42:11 UTC
Thanks Victor.

It takes some time before the update gets into updates-testing repository. Nothing I can do about it right now.

Do you have any tips how to test skydns? I would like to run some integration tests for each update of it to see if it is working or not. 

Glad to hear the effort to package and maintain skydns is not in vain :).

Cheers
Jan

Comment 14 Fedora Update System 2015-11-01 06:58:58 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update golang-github-skynetservices-skydns'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-14c8299f15

Comment 15 Fedora Update System 2015-11-02 00:27:39 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update golang-github-skynetservices-skydns'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-aadc2e2de8

Comment 16 Victor Costan 2015-11-02 00:56:05 UTC
I severely underestimated the lag between "pending" and "testing". Sorry for the spam!

I have a Vagrantfile that spawns a bunch of VMs and uses Ansible to deploy a bunch of things. I can get you a trimmed down list of Ansible roles that bring up etcd and skydns on a single machine. Alternatively, I can get you a shell script that does the same thing.

I can imagine using something like bats [1] the way docker-swarm uses it [2]. If both you and the skydns author are interested, I could look into writing a few tests and contributing them to the repository.

Last, my PR got merged into skydns [3]. If this doesn't translate into a large amount of work, would it be possible for you to package the git tree for that commit (or a more recent one) into fedora?

The commit I referenced would honor a "SKYDNS_SYSTEMD=true" line in /etc/skydns/skydns.conf, reducing the manual configuration overhead by a tiny bit. Most importantly, The etcd configuration for skydns would be relieved of a "systemd: true" key, which really looks like an implementation detail to me.

What do you think of all the things I mentioned here?

[1] https://github.com/sstephenson/bats
[2] https://github.com/docker/swarm/tree/master/test/integration
[3] https://github.com/skynetservices/skydns/commit/2ed66bcd0ec7d50cc89dfad1a68a5670313b662b

Comment 17 Fedora Update System 2016-01-04 18:53:07 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2016-01-04 19:57:32 UTC
golang-github-skynetservices-skydns-2.5.3-0.1.a.git8688008.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.