/usr/lib64/libdialog.so.14.0.0 in dialog-1.3-11.20170509.fc28.x86_64 was not linked with the standard Fedora linker flags (LDFLAGS) from redhat-rpm-config: libtool: link: gcc -shared -fPIC -DPIC .libs/trace.o .libs/rc.o .libs/calendar.o .libs/fselect.o .libs/timebox.o .libs/buildlist.o .libs/rangebox.o .libs/treeview.o .libs/formbox.o .libs/guage.o .libs/pause.o .libs/prgbox.o .libs/progressbox.o .libs/tailbox.o .libs/mixedform.o .libs/mixedgauge.o .libs/argv.o .libs/arrows.o .libs/buttons.o .libs/checklist.o .libs/columns.o .libs/dlg_keys.o .libs/editbox.o .libs/help.o .libs/inputbox.o .libs/inputstr.o .libs/menubox.o .libs/mouse.o .libs/mousewget.o .libs/msgbox.o .libs/textbox.o .libs/ui_getc.o .libs/util.o .libs/version.o .libs/yesno.o -lncursesw -ltinfo -lm -Wl,-z -Wl,relro -Wl,-soname -Wl,libdialog.so.14 -o .libs/libdialog.so.14.0.0 -specs=/usr/lib/rpm/redhat/redhat-hardened-ld is missing. It was still there when LDFLAGS was set at the beginning of the %build section. So this failure looks libtool-related.
dialog uses the system libtool, which doesn't include the hardening hack. However, the configure script has an option to specify options for libtool. This seems to help: --with-libtool-opts="-Wc,%{?_hardened_ldflags}" From the build log: libtool: link: gcc -shared -fPIC -DPIC .libs/trace.o .libs/rc.o .libs/calendar.o .libs/fselect.o .libs/timebox.o .libs/buildlist.o .libs/rangebox.o .libs/treeview.o .libs/formbox.o .libs/guage.o .libs/pause.o .libs/prgbox.o .libs/progressbox.o .libs/tailbox.o .libs/mixedform.o .libs/mixedgauge.o .libs/argv.o .libs/arrows.o .libs/buttons.o .libs/checklist.o .libs/columns.o .libs/dlg_keys.o .libs/editbox.o .libs/help.o .libs/inputbox.o .libs/inputstr.o .libs/menubox.o .libs/mouse.o .libs/mousewget.o .libs/msgbox.o .libs/textbox.o .libs/ui_getc.o .libs/util.o .libs/version.o .libs/yesno.o -lncursesw -ltinfo -lm -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-z -Wl,relro -Wl,-soname -Wl,libdialog.so.14 -o .libs/libdialog.so.14.0.0
It seems this stopped working in rawhide. Now the command is mangled to: libtool: link: gcc -shared -fPIC -DPIC .libs/trace.o .libs/rc.o .libs/calendar.o .libs/fselect.o .libs/timebox.o .libs/buildlist.o .libs/rangebox.o .libs/treeview.o .libs/formbox.o .libs/guage.o .libs/pause.o .libs/prgbox.o .libs/progressbox.o .libs/tailbox.o .libs/mixedform.o .libs/mixedgauge.o .libs/argv.o .libs/arrows.o .libs/buttons.o .libs/checklist.o .libs/columns.o .libs/dlg_keys.o .libs/editbox.o .libs/help.o .libs/inputbox.o .libs/inputstr.o .libs/menubox.o .libs/mouse.o .libs/mousewget.o .libs/msgbox.o .libs/textbox.o .libs/ui_getc.o .libs/util.o .libs/version.o .libs/yesno.o -lncursesw -ltinfo -lm -Wl -z now -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-soname -Wl,libdialog.so.14 -o .libs/libdialog.so.14.0.0 gcc: error: unrecognized command line option '-Wl'; did you mean '-W'?
The problem was with _hardened_flags including a new linker option. The spec was fixed to specify the options separately using multiple -Xcompiler options of libtool (although only the -spec option has to be specified like that).