Are you sure that it is definately working with 0.98.7? Because neither the
changelog nor the release notes of 0.99 mention any milter-related changes.
May you try a downgrade please to see if it works again with 0.98.7?

100% percent sure

the milter service stopped directly after the update, also after a reboot it crashed with the permission errors and since downgrade all is runnign as before


Dec 02 18:08:08 INFO Downgraded: clamav-lib-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-lib-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-filesystem-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-filesystem-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-server-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-server-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-server-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-server-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-scanner-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-scanner-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-scanner-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-scanner-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-milter-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-milter-systemd-0.98.7-3.fc23.noarch
Dec 02 18:08:08 INFO Downgraded: clamav-milter-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-milter-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-update-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Downgraded: clamav-update-0.98.7-3.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-scanner-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-scanner-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-scanner-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-scanner-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-update-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-update-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-server-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-server-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-server-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-server-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-milter-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-milter-systemd-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-milter-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-milter-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-filesystem-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-filesystem-0.99-1.fc23.noarch
Dec 02 18:08:08 INFO Erased: clamav-lib-0.99-1.fc23.x86_64
Dec 02 18:08:08 INFO Erased: clamav-lib-0.99-1.fc23.x86_64

[root@testserver:~]$ systemctl status clamav-milter.service
? clamav-milter.service - ClamAV Postfix-Milter
   Loaded: loaded (/etc/systemd/system/clamav-milter.service; enabled; vendor preset: disabled)
   Active: active (running) since Mi 2015-12-02 19:44:13 CET; 1h 1min ago
 Main PID: 1391 (clamav-milter)
   CGroup: /system.slice/clamav-milter.service
           ??1391 /usr/sbin/clamav-milter -c /etc/mail/clamav-milter.conf --nofork=yes

Dez 02 19:44:13 testserver.rhsoft.net systemd[1]: Started ClamAV Postfix-Milter.
Dez 02 19:44:13 testserver.rhsoft.net systemd[1]: Starting ClamAV Postfix-Milter...
Dez 02 19:44:13 testserver.rhsoft.net clamav-milter[1391]: +++ Started at Wed Dec  2 19:44:13 2015

Is it possible, that this change caused this?

commit daca419a0ae02d5503d0adf97b9e711a7d5623f8
Author: Robert Scheck <robert>
Date:   Tue Jun 30 02:46:50 2015 +0200

    Move /etc/tmpfiles.d/ to /usr/lib/tmpfiles.d/ (#1126595)

No, tmpfiles is only relevant due boot to recreate the folders but the daemon failed straight after the update and automatic restart - i have posted the folder permissions and they are identical on our production server running 0.98 on f22

there is no reason that the daemon pretends it has no permissions

maybe there is something wrong with drop privileges and they are dropped to nobody instead the user in the configuration?

Have you been able to exclude that this is a SELinux policy issue? Does it
change anything if you run it with "setenforce 0"? By the way, could you try
to use also the original shipped systemd targets rather your own ones (at
least for testing)?

SELinux is *not* enabled on any of my machines

it's "CapabilityBoundingSet=CAP_KILL CAP_SETGID CAP_SETUID CAP_SYS_CHROOT" but i am unable to figure out *what* it needs and why the hell it does now

tried add "CAP_DAC_OVERRIDE CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_NET_ADMIN"

Dec  6 15:28:26 testserver clamav-milter: ERROR: Failed to change socket ownership to user clamiltestserver systemd: clamav-milter.service: Failed with result 'exit-code'
_________________________

frankly why can't that damend service not just support get startet like spamass-milter directly with User/Group and without any capabilities like below


User=sa-milt
Group=sa-milt
PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
CapabilityBoundingSet=CAP_KILL

The error sounds like you need CAP_CHOWN - given this has been tracked down
to a custom CapabilityBoundingSet, this is anyway not really a packaging bug.

"CapabilityBoundingSet=CAP_KILL CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE" works - looks like some monkey upstream borked the ordering of operations due create the socket with the correct permissions, otherwise "CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE" would not be needed now when it's not needed with exactly the same pemissions below /run with the downgraded version

honestly there is no excuse for require CAP_DAC_OVERRIDE for a daemon which only needs to talk with another daemon over a socket nor should it need to get started as root with later drop privileges

would the milter-service support to get started with the low-privileged user/group and create the socket with the correct flags it would even not need to chown/chmod

"given this has been tracked down to a custom CapabilityBoundingSet, this is anyway not really a packaging bug" - well, in a perfect world the Fedora systemd units would use more secure defaults at all

at least this 4 directives should be part of nearly any service

PrivateTmp=yes
PrivateDevices=yes
ReadOnlyDirectories=/etc
ReadOnlyDirectories=/usr

Harald, may I kindly ask you first to stop this rude behaviour on this report
immediately before we continue? Calling e.g. upstream people "monkeys" shows 
from my point of view, that you don't honor their work at all.

i honor the work, honestly

what i do not honor is a regression in a trivial part like create a socket while the whole purpose of /run/clamav-milter/ onwed by "clamilt" is that the whole socket creation happens *after* drop privileges which would automatically lead in ownership "clamilt:clamilt" as well as the file-creation mode should be 0660 instead chown it later

doing that correctly would even allow "User=clamilt" and "Group=clamilt" in the systemd-unit, the dameon would only need to skip the complete drop-privileges/chmod/chown if it notcies that it's not running as root from the begin

https://bugzilla.clamav.net/show_bug.cgi?id=10731 is likely the cause for
the current situation.

But that (forget Capabilities for now) don't explain "MilterSocketMode 0660" in "/etc/mail/clamav-milter.conf" results in which makes the whole 0660 meaningless until somebody is crazy enough running his MTA as root :-)

[root@testserver:~]$ stat /run/clamav-milter/clamav-milter.socket
  File: '/run/clamav-milter/clamav-milter.socket'
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 13h/19d Inode: 253755      Links: 1
Access: (0660/srw-rw----)  Uid: (  109/ clamilt)   Gid: (    0/    root)
Access: 2015-12-06 15:59:38.861766729 +0100
Modify: 2015-12-06 15:59:38.861766729 +0100
Change: 2015-12-06 15:59:38.861766729 +0100
 Birth: -

interesting - when you change the systemd-unit to start it directly as "clamilt" the socket permissions are correct and you don't need any capabilities - shows how much a bug "Unable to stat() /run/clamav-milter/clamav-milter.socket" is when started in fact as root

may i propose the following systemd-unit?

that solves all problems at once, avoids touching anything as root and restrcits the milter as much as possible, there is no business to write below /var/lib because it's only the glue between MTA/ClamAV and /usr as well as /etc should be read-only in general
_______________________________________________________

[Unit]
Description = Milter module for the Clam Antivirus scanner
After = syslog.target nss-lookup.target network.target
Before = sendmail.service
Before = postfix.service

[Service]
Type = simple
ExecStart = /usr/sbin/clamav-milter -c /etc/mail/clamav-milter.conf --nofork=yes
Restart = on-failure

User=clamilt
Group=clamilt

PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
CapabilityBoundingSet=CAP_KILL

ReadOnlyDirectories=/etc
ReadOnlyDirectories=/usr
ReadOnlyDirectories=/var/lib

[Install]
WantedBy = multi-user.target
_______________________________________________________

Dec  6 17:12:52 testserver clamav-milter[17180]: +++ Started at Sun Dec  6 17:12:52 2015
Dec  6 17:13:11 testserver postfix/submission/smtpd[17188]: connect from unknown[192.168.196.2]
Dec  6 17:13:11 testserver postfix/submission/smtpd[17188]: Anonymous TLS connection established from unknown[192.168.196.2]: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)
Dec  6 17:13:11 testserver postfix/submission/smtpd[17188]: 3pDCVb5GkYz295Z: client=unknown[192.168.196.2], sasl_method=CRAM-MD5, sasl_username=test.net
Dec  6 17:13:11 testserver postfix/cleanup[17194]: 3pDCVb5GkYz295Z: info: header Subject: Test from unknown[192.168.196.2]; from=<test.net> to=<rhsoft> proto=ESMTP helo=<srv-rhsoft.rhsoft.net>
Dec  6 17:13:11 testserver postfix/cleanup[17194]: 3pDCVb5GkYz295Z: message-id=<56645E97.8040201.net>
Dec  6 17:13:11 testserver spamd[1296]: spamd: got connection over /run/spamassassin/spamassassin.sock
Dec  6 17:13:11 testserver spamd[1296]: spamd: processing message <56645E97.8040201.net> for sa-milt:189
Dec  6 17:13:13 testserver spamd[1296]: pyzor: check failed: internal error, python traceback seen in response
Dec  6 17:13:13 testserver spamd[1296]: spamd: clean message (0.6/5.5) for sa-milt:189 in 1.5 seconds, 1368 bytes.
Dec  6 17:13:13 testserver spamd[1296]: spamd: result: . 0 - ALL_TRUSTED,BAYES_50,TVD_SPACE_RATIO scantime=1.5,size=1368,user=sa-milt,uid=189,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=/run/spamassassin/spamassassin.sock,mid=<56645E97.8040201.net>,bayes=0.499785,autolearn=disabled
Dec  6 17:13:13 testserver postfix/cleanup[17194]: 3pDCVb5GkYz295Z: milter-reject: END-OF-MESSAGE from unknown[192.168.196.2]: 5.7.1 Virus found: "Eicar-Test-Signature"; from=<test.net> to=<rhsoft> proto=ESMTP helo=<srv-rhsoft.rhsoft.net>
Dec  6 17:13:14 testserver postfix/submission/smtpd[17188]: disconnect from unknown[192.168.196.2] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1 data=0/1 quit=1 commands=7/8

BTW - i deployed 0.99 on F22 now on our production server with the proposed unit-changes, interesting is that clamd seems not to be affected by the socket troubles, until now unchanged unit, i will give it also i try to get started as restricted user - but that's off-topic for this bugreport

[root@mail-gw:~]$ cat /etc/systemd/system/clamd.service
[Unit]
Description=ClamAV Scanner Daemon

[Service]
Type=forking
Environment="TMPDIR=/tmp"
Environment="LANG=en_GB.UTF-8"
ExecStart=/usr/sbin/clamd -c /etc/clamd.d/scan.conf
Restart=always
RestartSec=1

PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
CapabilityBoundingSet=CAP_KILL CAP_SETGID CAP_SETUID CAP_SYS_CHROOT
RestrictAddressFamilies=~AF_APPLETALK AF_ATMPVC AF_AX25 AF_IPX AF_NETLINK AF_PACKET AF_X25
SystemCallArchitectures=x86-64

ReadOnlyDirectories=/etc
ReadOnlyDirectories=/usr
ReadOnlyDirectories=/var/lib
ReadWriteDirectories=/var/lib/clamav-quarantine
InaccessibleDirectories=-/var/lib/spamass-milter

just for the record: clamd works also started as unrestricted user using "clamilt" as group, not sure how that all is intended to work, as i installed the systema year ago i need to put postfix into the "clamilt" to get all 3 pieces work together without making the sockets chmod 0666 and especially open the folder permissions of /run/clamav-milter

however, i am happy with running both as non-root from the begin
__________________________________________________________

[root@mail-gw:~]$ cat /etc/systemd/system/clamd.service
[Unit]
Description=ClamAV Scanner Daemon

[Service]
Type=forking
Environment="TMPDIR=/tmp"
Environment="LANG=en_GB.UTF-8"
ExecStart=/usr/sbin/clamd -c /etc/clamd.d/scan.conf
Restart=always
RestartSec=1
Nice=5

User=clamscan
Group=clamilt

PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
CapabilityBoundingSet=CAP_KILL CAP_SETGID CAP_SETUID CAP_SYS_CHROOT
RestrictAddressFamilies=~AF_APPLETALK AF_ATMPVC AF_AX25 AF_IPX AF_NETLINK AF_PACKET AF_X25
SystemCallArchitectures=x86-64

ReadOnlyDirectories=/etc
ReadOnlyDirectories=/usr
ReadOnlyDirectories=/var/lib
InaccessibleDirectories=-/var/lib/spamass-milter

InaccessibleDirectories=-/boot
InaccessibleDirectories=-/home
InaccessibleDirectories=-/media
InaccessibleDirectories=-/root
InaccessibleDirectories=-/etc/dbus-1
InaccessibleDirectories=-/etc/modprobe.d
InaccessibleDirectories=-/etc/modules-load.d
InaccessibleDirectories=-/etc/postfix
InaccessibleDirectories=-/etc/ssh
InaccessibleDirectories=-/etc/sysctl.d
InaccessibleDirectories=-/run/console
InaccessibleDirectories=-/run/dbus
InaccessibleDirectories=-/run/lock
InaccessibleDirectories=-/run/mount
InaccessibleDirectories=-/run/systemd/generator
InaccessibleDirectories=-/run/systemd/system
InaccessibleDirectories=-/run/systemd/users
InaccessibleDirectories=-/run/udev
InaccessibleDirectories=-/run/user
InaccessibleDirectories=-/usr/lib64/dbus-1
InaccessibleDirectories=-/usr/lib64/xtables
InaccessibleDirectories=-/usr/lib/dracut
InaccessibleDirectories=-/usr/libexec/iptables
InaccessibleDirectories=-/usr/libexec/openssh
InaccessibleDirectories=-/usr/libexec/postfix
InaccessibleDirectories=-/usr/lib/grub
InaccessibleDirectories=-/usr/lib/kernel
InaccessibleDirectories=-/usr/lib/modprobe.d
InaccessibleDirectories=-/usr/lib/modules
InaccessibleDirectories=-/usr/lib/modules-load.d
InaccessibleDirectories=-/usr/lib/rpm
InaccessibleDirectories=-/usr/lib/sysctl.d
InaccessibleDirectories=-/usr/lib/udev
InaccessibleDirectories=-/usr/local/scripts
InaccessibleDirectories=-/var/db
InaccessibleDirectories=-/var/lib/dbus
InaccessibleDirectories=-/var/lib/rpm
InaccessibleDirectories=-/var/lib/systemd
InaccessibleDirectories=-/var/lib/yum
InaccessibleDirectories=-/var/spool

[Install]
WantedBy=multi-user.target
__________________________________________________________

[root@mail-gw:~]$ cat /etc/clamd.d/scan.conf
User clamscan
AllowSupplementaryGroups yes

PidFile /run/clamd.scan/clamd.pid
TemporaryDirectory /tmp
DatabaseDirectory /var/lib/clamav
OfficialDatabaseOnly no
LocalSocket /run/clamd/clamd.sock 
LocalSocketMode 0666
MaxConnectionQueueLength 100
StreamMaxLength 35M
StreamMinPort 30000
StreamMaxPort 32000
MaxThreads 20
MaxQueue 50
ReadTimeout 120
CommandReadTimeout 5
SendBufTimeout 200
IdleTimeout 60
ExcludePath ^/proc/
ExcludePath ^/sys/
MaxDirectoryRecursion 20
FollowDirectorySymlinks no
FollowFileSymlinks no
CrossFilesystems yes
SelfCheck 1200
ExitOnOOM yes
Foreground no
Debug no
LeaveTemporaryFiles no
AllowAllMatchScan yes
DetectPUA no
AlgorithmicDetection yes
DisableCache no

ScanPE yes
DisableCertCheck yes
ScanELF yes
DetectBrokenExecutables yes
ScanOLE2 yes
OLE2BlockMacros no
ScanPDF yes
ScanSWF yes
ScanMail yes
ScanPartialMessages no
PhishingSignatures yes
PhishingScanURLs yes
PhishingAlwaysBlockSSLMismatch no
PhishingAlwaysBlockCloak no
PartitionIntersection no
HeuristicScanPrecedence yes
StructuredDataDetection no
ScanHTML yes
ScanArchive yes
ArchiveBlockEncrypted no
MaxScanSize 50M
MaxFileSize 50M
MaxRecursion 15
MaxFiles 10000
MaxEmbeddedPE 10M
MaxHTMLNormalize 10M
MaxHTMLNoTags 2M
MaxScriptNormalize 5M
MaxZipTypeRcg 1M
MaxPartitions 50
MaxIconsPE 100
ScanOnAccess no
Bytecode yes
BytecodeSecurity TrustSigned
BytecodeTimeout 1000
StatsEnabled no
StatsPEDisabled yes

Did I get you right, that the proposed systemd unit from comment #15 should
solve the issues introduced by the update to clamav-0.99-1.fc2x?

yes, when starting not as root at all the chown/chgrp/chmod code doesn't happen and the socket has the group "clamilt" instead "root" which makes "MilterSocketMode 0660" working properly

User=clamilt
Group=clamilt

however ending the socket as group "root" without User/Group in the systemd-unit is still a upstream bug but the above is preferred anyways because there is no point running code as root when it's not really needed

clamav-0.99-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-a81b2c2212

clamav-0.99-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-a81b2c2212

clamav-0.99-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-13661ecb72

clamav-0.99-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-13661ecb72

clamav-0.99-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-7b36c1ca5a

clamav-0.99-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-7b36c1ca5a

clamav-0.99-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update clamav'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-a81b2c2212

clamav-0.99-2.fc23 dansguardian-2.10.1.1-17.fc23 klamav-0.46-18.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-a81b2c2212

clamav-0.99-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update clamav'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-13661ecb72

clamav-0.99-2.fc22 dansguardian-2.10.1.1-17.fc22 klamav-0.46-18.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-13661ecb72

clamav-0.99-2.fc22, dansguardian-2.10.1.1-17.fc22, klamav-0.46-18.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update dansguardian klamav clamav'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-13661ecb72

clamav-0.99-2.fc23, dansguardian-2.10.1.1-17.fc23, klamav-0.46-18.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update dansguardian klamav clamav'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-a81b2c2212

clamav-0.99-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'yum --enablerepo=epel-testing update clamav'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-7b36c1ca5a

clamav-0.99-2.fc22, dansguardian-2.10.1.1-17.fc22, klamav-0.46-18.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

clamav-0.99-2.fc23, dansguardian-2.10.1.1-17.fc23, klamav-0.46-18.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

clamav-0.99-2.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.