Generally speaking, detailed changelog files do not belong to %doc of the package.
%doc should prefer NEWS file, which should concisely describe user visible changes of the package.
It is possible to make an exception, if some of the following conditions are met:
- the changelog is short,
- the changelog is written in a user-geared style,
- the project fails to deliver a news file.

Speaking specifically about Perl:
It meets the last condition only, and I do not think it is sufficient for such an exception.
The change logs might be important to dig out when a particular feature was introduced or bug fixed.  But to satisfy this advanced user need, it is enough to pack the changelog collection in perl-devel.