On most machines I now have both RSA and DSA keys, for connecting to SSHv1 or SSHv2 hosts respectively. I tend to give the keys the same passphrase - I'm sure many people do the same. When adding both RSA and DSA keys, I believe ssh-add should attempt to re-use the first passphrase given for the second key, and only prompt for a second time if it fails.
After a passphrase is read in and ssh-add attempts to use it to decrypt a key file, it takes care to clear the memory used. This leads me to believe that reusing passphrases in this manner would be a Bad Thing.
Remembering the passphrase for the duration of a single invocation of ssh-add wouldn't open any new vulnerability. If you're paranoid about an attacker being able to control/crash ssh-add and read the passphrase then the same applies to getting decrypted keys from ssh-agent and you shouldn't be using them at all.
Created attachment 7207 [details] patch to provide requested functionality