Attachment 1519220 Details for Bug 1625597 – Setup instructions for NBD over TLS

Setup instructions for NBD over TLS

Setup-for-NBD-and-migration-streams-over-TLS.rst.txt (text/plain), 11.42 KB, created by Kashyap Chamarthy on 2019-01-08 15:13:23 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Kashyap Chamarthy

Created: 2019-01-08 15:13:23 UTC

Size: 11.42 KB

patch

obsolete

>Full virtual setup for NBD and migration streams over TLS
>=========================================================
>
>(1) Setup two guest hypervisors::
>
>    $ virt-builder fedora-28 -o guestHyp1.qcow2 --update \
>        --format qcow2 --selinux-relabel --size 80G \
>        --root-password password:123456 --install \
>        "libguestfs-tools-c libvirt-daemon-kvm virt-install \
>        libvirt-daemon-config-network libvirt-daemon-config-nwfilter "
>
>    $ virt-builder fedora-28 -o guestHyp2.qcow2 --update \
>        --format qcow2 --selinux-relabel --size 80G \
>        --root-password password:123456 --install \
>        "libguestfs-tools-c libvirt-daemon-kvm virt-install \
>        libvirt-daemon-config-network libvirt-daemon-config-nwfilter "
>
>(2) Import them into libvirt::
>
>    $ virt-install --name guestHyp1 --ram 8096 \
>      --disk path=./guestHyp2.qcow2,format=qcow2 \
>      --machine q35 --os-variant fedora27 \
>      --cpu host-passthrough --nographics --import
>
>    $ virt-install --name guestHyp1 --ram 8096 \
>      --disk path=./guestHyp2.qcow2,format=qcow2 \
>      --machine q35 --os-variant fedora27 \
>      --cpu host-passthrough --nographics --import
>
>(3) From guestHyp1, start the nested guest::
>
>    $ virt-builder fedora-28 -o nGuest1 --update \
>        --format qcow2 --selinux-relabel --size 20G  \
>        --root-password password:fedora
>
>    $ virt-install --name nGuest1 --ram 2048 \
>        --disk path=./nGuest1.qcow2,format=qcow2 --machine q35 \
>        --os-variant fedora27 --cpu host-passthrough --nographics \
>        --import --print-xml > /tmp/nGuest1.xml
>
>    $ virsh define /tmp/nGuest1.xml
>
>   .. note:: The above approach of: `virt-install --print-xml`, followed
>             by `virsh define` will let us script it.)
>
>
>(4) Setup password-less SSH on both guest hypervisors, guestHyp1 and
>    guestHyp2::
>
>    $ ssh-keygen -t rsa
>    $ ssh-copy-id root@guestHyp$X
>    $ ssh root@guestHyp$X
>
>(5) Stop ``firewalld`` on both guestHyp1 and guestHyp2:: (otherwise you
>    get: ``error: internal error: unable to execute QEMU command
>    'drive-mirror': Failed to connect socket: No route to host``) ::
>
>    $ systemctl stop firewalld
>
>(6) Migrate 'nGuest1' from guestHyp1 to guestHyp2, and vice-versa;
>    *without* TLS)::
>
>    $ virsh migrate --verbose --copy-storage-all \
>        --live nguest1 qemu+ssh://guestHyp2/system
>
>    $ virsh migrate --verbose --copy-storage-all \
>        --live nGuest1 qemu+ssh://guestHyp1/system
>
>(7) Setup TLS.
>
>(7.a) On _both_ hosts::
>
>    $ mkdir -p /etc/pki/qemu
>    $ mkdir -p /etc/pki/libvirt/private
>    $ mkdir -p /etc/pki/CA
>
>(7.b) On guestHyp1, create the CA cert and SCP it to guestHyp2::
>
>    $ cd /home
>    $ mkdir pki && cd pki
>
>    $ cat ca.info
>    cn = JAF Migration TLS Test
>    ca
>    cert_signing_key
>    expiration_days = 700
>
>    $ certtool --generate-privkey > cakey.pem
>
>    $ certtool --generate-self-signed --load-privkey cakey.pem \
>      --template ca.info --outfile cacert.pem
>    $ cp cacert.pem /etc/pki/CA/
>    $ scp cacert.pem guestHyp2:/etc/pki/CA/
>
>(7.b) On guestHyp1, generate server certificates for both, guestHyp1 and
>      guestHyp2 and SCP the guestHyp2 cert to its host::
>
>    $ cat guestHyp1-server.info
>    organization = TLS Migration Test
>    cn = guestHyp1
>    tls_www_server
>    encryption_key
>    signing_key
>
>    $ cat guestHyp2-server.info
>    organization = TLS Migration Test
>    cn = guestHyp2
>    tls_www_server
>    encryption_key
>    signing_key
>
>    $ certtool --generate-privkey > guestHyp1-serverkey.pem
>    Generating a 3072 bit RSA private key...
>
>    $ certtool --generate-privkey > guestHyp2-serverkey.pem
>    Generating a 3072 bit RSA private key...
>
>    $ certtool --generate-certificate --template guestHyp1-server.info \
>           --load-privkey guestHyp1-serverkey.pem \
>           --load-ca-certificate cacert.pem \
>           --load-ca-privkey cakey.pem \
>           --outfile guestHyp1-servercert.pem
>
>    $ certtool --generate-certificate --template guestHyp2-server.info \
>           --load-privkey guestHyp2-serverkey.pem \
>           --load-ca-certificate cacert.pem \
>           --load-ca-privkey cakey.pem \
>           --outfile guestHyp2-servercert.pem
>
>    $ cp guestHyp1-servercert.pem /etc/pki/libvirt/servercert.pem
>    $ cp guestHyp1-serverkey.pem /etc/pki/libvirt/private/serverkey.pem
>    $ scp guestHyp2-servercert.pem guestHyp2:/etc/pki/libvirt/servercert.pem
>    $ scp guestHyp2-serverkey.pem guestHyp2:/etc/pki/libvirt/private/serverkey.pem
>
>(7.c) On guestHyp1::
>
>    $ chown root:qemu /etc/pki/libvirt
>    $ chown root:qemu /etc/pki/libvirt/*
>    $ chown root:qemu /etc/pki/libvirt/private/*
>
>(7.d) On *both* hosts, guestHyp1 and guestHyp2, adjust permissoins for
>      their corresponding server certificates::
>
>    $ chmod 440 /etc/pki/libvirt/servercert.pem
>    $ chmod 750 /etc/pki/libvirt/private/
>    $ chmod 600 /etc/pki/libvirt/private/serverkey.pem
>
>(7.e) On guestHyp1, generate client certificates for both, guestHyp1 and
>      guestHyp2 and SCP the guestHyp2 client cert to its host::
>
>    $ certtool --generate-privkey > guestHyp1-clientkey.pem
>    $ certtool --generate-privkey > guestHyp2-clientkey.pem
>
>    $ cat guestHyp1-client.info
>    country = BE
>    state = EF
>    locality = Belgium
>    organization = TLS Migration Test
>    cn = guestHyp1
>    tls_www_client
>    encryption_key
>    signing_key
>
>    $ cat guestHyp2-client.info
>    country = BE
>    state = EF
>    locality = Belgium
>    organization = TLS Migration Test
>    cn = guestHyp1
>    tls_www_client
>    encryption_key
>    signing_key
>
>    $ certtool --generate-certificate \
>             --template guestHyp1-client.info \
>             --load-privkey guestHyp1-clientkey.pem \
>             --load-ca-certificate cacert.pem \
>             --load-ca-privkey cakey.pem \
>             --outfile guestHyp1-clientcert.pem
>
>    $ certtool --generate-certificate \
>             --template guestHyp2-client.info \
>             --load-privkey guestHyp2-clientkey.pem \
>             --load-ca-certificate cacert.pem \
>             --load-ca-privkey cakey.pem \
>             --outfile guestHyp2-clientcert.pem
>
>    $ cp guestHyp1-clientcert.pem /etc/pki/libvirt/clientcert.pem
>    $ cp guestHyp1-clientkey.pem /etc/pki/libvirt/private/clientkey.pem
>    $ scp guestHyp2-clientcert.pem guestHyp2:/etc/pki/libvirt/clientcert.pem
>    $ scp guestHyp2-clientkey.pem guestHyp2:/etc/pki/libvirt/private/clientkey.pem
>
>(7.f) On *both* hosts, guestHyp1 and guestHyp2::
>
>    $ chmod 400 /etc/pki/libvirt/clientcert.pem
>    $ chmod 644 /etc/pki/libvirt/private/clientkey.pem
>
>(7.g) On *both*, because QEMU requires the `/etc/pki/qemu` directory
>      contents to be slightly different::
>
>    $ cp /etc/pki/CA/cacert.pem /etc/pki/qemu/ca-cert.pem
>    $ cp /etc/pki/libvirt/servercert.pem /etc/pki/qemu/server-cert.pem
>    $ cp /etc/pki/libvirt/private/serverkey.pem /etc/pki/qemu/server-key.pem
>    $ cp /etc/pki/libvirt/clientcert.pem /etc/pki/qemu/client-cert.pem
>    $ cp /etc/pki/libvirt/private/clientkey.pem /etc/pki/qemu/client-key.pem
>
>(7.h) On *both*, guestHyp1, and guestHyp2, update 'x509' config options
>      in /etc/libvirt/qemu.conf::
>
>        default_tls_x509_cert_dir = "/etc/pki/qemu"
>        default_tls_x509_verify = 1
>
>      And modify /etc/sysconfig/libvirtd on both (guestHyp1 &
>      guestHyp2)::
>
>        LIBVIRTD_ARGS="--listen"
>
>      And restart libvirt daemon (also on both hosts)::
>
>        $ systemctl restart libvirtd
>
>(8) Run `virt-pki-validate` on guestHyp1 and guestHyp2::
>
>    $ virt-pki-validate
>    Found /usr/bin/certtool
>    Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test
>    Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp1
>    Found client private key /etc/pki/libvirt/private/clientkey.pem
>    Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp1
>    Found server private key /etc/pki/libvirt/private/serverkey.pem
>    Make sure /etc/sysconfig/libvirtd is setup to listen to
>    TCP/IP connections and restart the libvirtd service
>
>    $ virt-pki-validate
>    Found /usr/bin/certtool
>    Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test
>    Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp2
>    Found client private key /etc/pki/libvirt/private/clientkey.pem
>    Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp2
>    Found server private key /etc/pki/libvirt/private/serverkey.pem
>    Make sure /etc/sysconfig/libvirtd is setup to listen to
>    TCP/IP connections and restart the libvirtd service
>
>(9) **IMPORTANT**: *Ensure* the permissions of certificate files and keys
>    in ``/etc/pki/qemu/*`` directory on both source _and_ destination to be
>    the following::
>
>    [root@guestHyp1 qemu]$ ls -lasrtZ /etc/pki/qemu/                                                                                                                                              total 32
>    0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0      110 Dec 10 10:39 ..
>    4 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:08 ca-cert.pem
>    4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:08 server-cert.pem
>    4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:09 client-cert.pem
>    8 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:09 client-key.pem
>    4 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 2459 Dec 11 05:32 server-key-stripped.pem
>    8 -rw----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8177 Dec 11 05:35 server-key.pem
>    0 drwxr-xr-x.  2 root root unconfined_u:object_r:cert_t:s0  146 Dec 11 06:01 .
>
>    [root@guestHyp2 ~]# ls -lasrtZ /etc/pki/qemu/
>    total 28
>    0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0      110 Dec 10 10:39 ..
>    4 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:10 ca-cert.pem
>    4 -r--r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:10 server-cert.pem
>    8 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8170 Dec 10 11:10 server-key.pem
>    4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:10 client-cert.pem
>    8 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:10 client-key.pem
>    0 drwxr-xr-x.  2 root root unconfined_u:object_r:cert_t:s0  115 Dec 10 11:10 .
>
>
>(10) Migrate 'nGuest1' from guestHyp1 to guestHyp2 *with* TLS::
>
>    $ virsh migrate --tls --verbose --copy-storage-all \
>        --migrate-disks vda nGuest1 --live qemu+ssh://guestHyp2/system
>
>
>Other notes
>-----------
>
>There are in total _nine_ TLS-related config options in
>/etc/libvirt/qemu.conf:
>
>  default_tls_x509_cert_dir
>  default_tls_x509_verify
>  nbd_tls
>  nbd_tls_x509_cert_dir
>  migrate_tls_x509_cert_dir
>
>  vnc_tls_x509_cert_dir
>  spice_tls_x509_cert_dir
>  vxhs_tls_x509_cert_dir
>  chardev_tls_x509_cert_dir
>
>
>See how they fits in with ``default_tls_x509_cert_dir`` and
>``default_tls_x509_verify``.
>
>Answer (thanks: Dan BerrangÃ©, for the IRC chat):
>
>If you set the both ``default_*`` parameters for all certificates, then
>no need to specify any of the other ``*_tls*`` config options.
>
>The intention of the upstream libvirt is that you can just use the
>`default_*` (that way you don't need to set any other ``*_tls*``
>parameters, _unless_ you need different certificates for some services.
>The rationale for that is that some services (e.g. migration / NBD)  are
>only exposed to internal infrastructure; while some sevices (VNC, Spice)
>might be exposed publically, so might need different certificates.
>
>For OpenStack this doesn't matter, though.  So we're good
>

Actions: View

Attachments on bug 1625597: 1519220