Linking anything with -ffast-math adds code that runs on load that reprograms the FPU for reduced accuracy. This applies even to shared libraries, which means that merely loading a shared library linked with -ffast-math can break programs.
Please consider disallowing any shared library linked with -ffast-math in rpmlint.
See, for example, bug 1127544.
This is more a policy-decision (ie, something for FESCo to consider), not something to be enforceable via any individual (ie, maintainer of rpmlint)