The problem:
The variable 'j' in this routine is the number of matching (not different) characters between the old and new passwords.
As such, the direction of the inequality 'j >= opt->diff_ok' is wrong.  You want something like 'i-j >= opt->diff_ok'.

To reproduce the problem:
Anyone using MD5 passwords should be able to reproduce the problem by changing (say) 'the quick brown' to 'the quick brow'.
For crypt passwords, try setting difok=5.  Create a test user and set its password to 'st3y7r8'.  Now change the password to 'st3y7r9'.
Since 6 characters were unchanged and 6 >=5, this is accepted.  Now try changing the password to 'st3y1b2'.  
Since only 4 characters are the same (4<5) and fewer than half the characters are different, this will be rejected.

MNSHO:
Frankly, the calculation of 'j' seems rather odd.  
It's the number of not-necessarily-distinct characters in the old password that appear in the new password.
And if the new password is shorter, the old password is effectively truncated to the length of the new password before this calculation is done.
I would be interested in understanding why this was done in light of the comments above the function.