I think it is effectively escaping the single quotes. I ran it myself with the nickname quoted via bash -x and got:

++ grep NSSNickname /etc/httpd/conf.d/nss.conf
++ cut -f 2 -d ' '
+ http_cert_nick=''\''Server-Cert'\'''
+ certutil -M -n ''\''Server-Cert'\''' -d /etc/httpd/alias -f /etc/httpd/alias/pwdfile.txt -t Pu,u,u
certutil: could not find certificate named "'Server-Cert'": SEC_ERROR_BAD_DATABASE: security library: bad database.
+ '[' 255 -ne 0 ']'
+ echo 'Can not set trust flags on HTTP certificate'
Can not set trust flags on HTTP certificate
+ exit 1

The easy answer is to sed those single quotes out but given it can be in a subject I suspect something like this is needed at the end of the grep:

| sed "s/^'//" | sed "s/'$//"

to drop leading and trailing single quotes.

Note that double-quotes around the nickname will cause similar issues.

Upstream ticket:
https://pagure.io/freeipa/issue/7706

Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/6e2bd184d60894dae18d08c214403251ee7e26ad


Note: fixed in ipa-4-6 only as ipa-4-7 is using mod_ssl instead of mod_nss and issue does not happen on ipa-4-7.

IPA: ipa-server-4.6.5-10.el7.x86_64


Verified the bug on the basis of following steps:

Setup Script to create ext certs:
------------------------------------
#!/bin/bash

DBDIR="/tmp/ipa/ext_nssdb" # will be removed if exists!
PWDFILE="$DBDIR/pwdfile.txt"
NOISE="$DBDIR/noise.txt"
PASSWORD="Secret123"
DOMAIN="TESTRELM.TEST"
SERVER="qe-blade-07.testrelm.test"

if [ $EUID -ne 0 ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

# Remove previous NSS database if it exists
if [ -e "$DBDIR" ]; then
    rm -rf "$DBDIR"
fi

# Get Subject Key Identifiers for the root and IPA CAs
ROOT_KEY_ID=0x$(dd if=/dev/urandom bs=20 count=1 | xxd -p)

# Prepare a new NSS database to serve us as an external CA
mkdir -p "$DBDIR"
echo "$PASSWORD" > "$PWDFILE"
# create noise file
dd count=10 bs=1024 if=/dev/random of="$NOISE" 2>/dev/null
certutil -N -d "$DBDIR" -f "$PWDFILE"

# Generate a CA certificate
echo -e "0\n1\n5\n6\n9\ny\ny\n\ny\n${ROOT_KEY_ID}\nn\n" \
    | certutil -d "$DBDIR" -S -s "CN=Cert Auth,O=Flo4Auth" -n ca -t C,C,C -x \
-1 -2 --extSKID -f "$PWDFILE" -z "$NOISE"

# Generate a key for the server cert
openssl genrsa -aes256 -out $DBDIR/server.key  -passout pass:$PASSWORD 2048

# Generate a CSR
openssl req -key $DBDIR/server.key -new -sha256 -outform der -out $DBDIR/server.csr -subj /O=${DOMAIN}/CN=${SERVER} -passin pass:$PASSWORD

# Sign the CSR
echo -e "0\n1\n2\n3\n9\ny\n${ROOT_KEY_ID}\n" \
     | certutil -C -d "$DBDIR" -m 1001 -i "$DBDIR/server.csr" \
       -o "$DBDIR/server.cer" -c ca \
-1 --extSKID -f "$PWDFILE" -z "$NOISE"

openssl x509 -inform der -in "$DBDIR/server.cer" -out "$DBDIR/server.pem"

# Export the NSS CA certificate and add it to a chain file
certutil -L -n ca -d "$DBDIR" -a > "$DBDIR/ca.crt"
cat "$DBDIR/server.pem" "$DBDIR/ca.crt" > "$DBDIR/chain.crt"

echo "Please use $DBDIR/ca.crt $DBDIR/server.pem and $DBDIR/server.key"


Steps:
--------------------------
1. Copy the above shell script to IPA MASTER. (I named it as /tmp/test.sh)
2. Run the script # bash -x test.sh
3. Browse to '/tmp/ipa/ext_nssdb' and run following commands

[root@qe-blade-07 ext_nssdb]# rpm -q ipa-server
ipa-server-4.6.5-10.el7.x86_64
[root@qe-blade-07 ext_nssdb]# ipa-server-certinstall --http server.key server.pem ca.crt
Directory Manager password: <Secret123> 

Enter private key unlock password: <Secret123>

Please restart ipa services after installing certificate (ipactl restart)
The ipa-server-certinstall command was successful

[root@qe-blade-07 ext_nssdb]# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

[root@qe-blade-07 ext_nssdb]# cat /etc/httpd/conf.d/nss.conf | grep NSSNickname
NSSNickname 'CN=qe-blade-07.testrelm.test,O=TESTRELM.TEST'

[root@qe-blade-07 ext_nssdb]# kinit admin
Password for admin: 

[root@qe-blade-07 ext_nssdb]# mkdir -pv /tmp/IPA_Stuff/SmartCard_CA-new/
mkdir: created directory ‘/tmp/IPA_Stuff/SmartCard_CA-new/’

[root@qe-blade-07 ext_nssdb]# ipa-advise config-server-for-smart-card-auth > /tmp/IPA_Stuff/SmartCard_CA-new/server_smart_card_script.sh
trying https://qe-blade-07.testrelm.test/ipa/json

[root@qe-blade-07 ext_nssdb]# ls -l /tmp/IPA_Stuff/SmartCard_CA-new/server_smart_card_script.sh
-rw-r--r--. 1 root root 4022 Jun 27 04:47 /tmp/IPA_Stuff/SmartCard_CA-new/server_smart_card_script.sh


Thus on the basis of above observations, the issue mentioned in the bug is no more observed and 'server_smart_card_script.sh' is created successfully, thus marking the status of bug to 'VERIFIED'.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2241