An issue was discovered in lib\cdt\dttree.c in libcdt.a in graphviz 2.40.1. Stack consumption occurs because of recursive agclose calls in lib\cgraph\graph.c in libcgraph.a, related to agfstsubg in lib\cgraph\subg.c. Reference: https://gitlab.com/graphviz/graphviz/issues/1512 https://research.loginsoft.com/vulnerability/stack-buffer-overflow-in-function-agclose-graphviz/
Created graphviz tracking bugs for this issue: Affects: fedora-all [bug 1691719]
I doubt it has security implications. From what I have seen it's about overflowing the stack by "over 24,000 nested graphs" processing which is quite a lot for graphviz. IMHO this can happen with every recursive processor if there is no artificial barrier for the recursion. Also IMHO DoS for graphviz is not much an issue - there are many reproducers which can crash it and they never got CVE. Also it's not clear to me how it is related to the pdftops the researchers are talking about in their report. Just my two cents.
Statement: This issue affects the versions of graphviz as shipped with Red Hat Enterprise Linux 6 and 7. Red Hat Enterprise Linux 6 is now in Maintenance Support 2 Phase of the support and maintenance life cycle. This has been rated as having a security impact of Low, and is not currently planned to be addressed in future updates. For additional information, refer to the Red Hat Enterprise Linux Life Cycle: https://access.redhat.com/support/policy/updates/errata/.