oauth2 support should be implemented in future releases of fetchmail-7.x.y (avaliable now as alpha version).
Upstream expects it to be released in 2021 and he doesn't plan to backport oauth2 support into fetchmail-6.x.y:
https://sourceforge.net/p/fetchmail/mailman/fetchmail-users/thread/DM6PR02MB52281958D09878A8CDBFCDD9FA0E0%40DM6PR02MB5228.namprd02.prod.outlook.com/#msg37121286

I've prepared srpm which can be used for building test rpm of fetchmail-7 on RHEL 8:
https://vcrhonek.fedorapeople.org/fetchmail-test/fetchmail-7.0.0-0.1.alpha6.el8.src.rpm

New alpha version of fetchmail-7.x.x branch has been released recently by upstream.

Again, I have prepared srpm useful for building test rpm on RHEL8:
https://vcrhonek.fedorapeople.org/fetchmail-test/fetchmail-7.0.0-0.1.alpha7.el8.src.rpm

Release notes of alpha7 explicitly mentions initial OAUTH2 support:
...
# FEATURES ADDED
* fetchmail has initial support for OAUTH2, courtesy of Matthew M. Ogilvie.  
  This requires a helper script (in Python) that ships in the contrib/ section.
...

It seems that users are already trying this feature:
https://sourceforge.net/p/fetchmail/mailman/fetchmail-devel/thread/CA%2BOGZjM1H_TKajqOV9YKU1fNt31L_zT7fEkAOtVPgyT77S4CAg%40mail.gmail.com/#msg37148276

I would be great if customer could test it with Exchange/Office 365.

Customer here: I'm happy to give the SRPM a try on RHEL8 as I have a vested interest in this working before October.

I tried to rebuild on RHEL8 and it failed as follows so no RPM binary is built and fetchmail is not installed

Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/fetchmail-7.0.0-0.1.alpha7.el8.x86_64
error: Installed (but unpackaged) file(s) found:
   /usr/lib/python3.8/site-packages/__pycache__/fetchmailconf.cpython-38.opt-1.pyc
   /usr/lib/python3.8/site-packages/__pycache__/fetchmailconf.cpython-38.pyc
   /usr/lib/python3.8/site-packages/fetchmailconf.py


RPM build errors:
    Installed (but unpackaged) file(s) found:
   /usr/lib/python3.8/site-packages/__pycache__/fetchmailconf.cpython-38.opt-1.pyc
   /usr/lib/python3.8/site-packages/__pycache__/fetchmailconf.cpython-38.pyc
   /usr/lib/python3.8/site-packages/fetchmailconf.py

Hi Greg,

Thanks for testing the package!

This works for me on clean RHEL-8.6 machine:

# wget https://vcrhonek.fedorapeople.org/fetchmail-test/fetchmail-7.0.0-0.1.alpha7.el8.src.rpm
# rpm -i fetchmail-7.0.0-0.1.alpha7.el8.src.rpm
# dnf -y builddep fetchmail
# dnf -y install rpmdevtools
# cd rpmbuild/SPECS/
# rpmbuild -ba ./fetchmail.spec
# ls ../RPMS/x86_64
fetchmail-7.0.0-0.1.alpha7.el8.x86_64.rpm  fetchmail-debuginfo-7.0.0-0.1.alpha7.el8.x86_64.rpm  fetchmail-debugsource-7.0.0-0.1.alpha7.el8.x86_64.rpm

Hello Vitezslav,

Got it working with the following edit to the spec file

[root@ss-tests rpmbuild]# diff -C2 SPECS/fetchmail.spec--original SPECS/fetchmail.spec
*** SPECS/fetchmail.spec--original      2021-01-18 04:02:56.000000000 -0800
--- SPECS/fetchmail.spec        2022-08-16 09:49:01.044078379 -0700
***************
*** 1,2 ****
--- 1,3 ----
+ %define _unpackaged_files_terminate_build 0
  %global alpha alpha7

I am curious, have you ever used the --passwordfd flag?  I'd like to keep this cred encrypted and have a decryption algorithm that can pipe the decrypted oauth2 token to fetchmail but I can't for the life of me figure out exactly how to get fetchmail to read the proper input, it never works

The man page says the value should be "a file descriptor number inherited from the calling process" and implemented as "something | fetchmail --passwordfd 5  5<0" but regardless of what file descriptor number I use, it doesn't work so I am not sure I understand exactly how to configure this flag or even figure out which file descriptor number would be applicable and how to make bash stop complaining about "bash: 0: No such file or directory" which happens when you use <Integer as shown in the man example

Oh never mind, though apparently still contained in the man pages, apparently the --passwordfd has been removed from the latest fetchmail according to the most recent docs, at least it is no longer listed as an option.

So in a nutshell, oauth2 on Office 365 IMAP is working perfectly, thanks so much for the effort, when will it be included in an upcoming release?  Before MS officially ends BasicAuth option in October I believe it is?

Thank you.

Greg

Hello,

I'm very interesting by your rpm but I can't download it. I've an ssl error with wget or any web browser.

Can you fix the problem or explain me how to dowanload ?

Thank you very much.

Guillaume

Hi Greg,
I am trying to use the fetchmail-oauth2.py script to generate the token but failing.
I have set the following parameters

auth_url=https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize?
token_url=https://login.microsoftonline.com/organizations/oauth2/v2.0/token
scope=https://outlook.office.com/Mail.Read

I run the script which gives me the URL and waits for the verification code. When I then login via the URL with the required account, I see a code on the end of the redirect_url and paste this as the verification code.
After this, I get a "urllib.error.HTTPError: HTTP Error 400: Bad Request" message which seems to be from the token_url.

I am on the right track ? Any chance you could share how you got this to work.

Thanks,
Colin

Created attachment 1910277 [details]
mutt_oauth2.py.patch

I had it working for several months with Office365. In fact I have separately setup oauth2 for fetchmail (incoming mail) and separately for mutt (outgoing mail by SMTP).
But on Aug 26th 2022 the fetchmail part stopped working exactly with this "urllib.error.HTTPError: HTTP Error 400: Bad Request" error.
I have solved it by manually copying the tokens from Mutt to fetchmail as Mutt still does work: ~/.muttrc.priv.tokens -> ~/.fetchmail-refresh ~/.fetchmail-token
So far I did not have to touch it again.

Follow
        /usr/share/doc/mutt/mutt_oauth2.py.README
in my case from mutt-2.2.1-1.fc36.x86_64 and be sure to really do the part titled "-- How to create a Microsoft registration --". It looks weird but you really need to register a new application to be able to do IMAP+SMTP for your existing account. I was using the "authcode" method described there. In fact I was doing the Microsoft registration according to
fetchmail-git/README.OAUTH2 which I then just used also for Mutt so if you have some issues you can check also the fetchmail-git trunk guide.

Instead of "userid.tokens" I am using "~/.muttrc.priv.tokens" as it is a filename and so their one was dependent on current directory. It gets created by the command:
        mutt_oauth2.py ~/.muttrc.priv.tokens --verbose --authorize

~/.muttrc:
------------------------------------------------------------------------------
set folder="imap://outlook.office365.com/"
set imap_user="myemail"
set imap_authenticators="xoauth2"
set imap_oauth_refresh_command="~/bin/mutt_oauth2.py ~/.muttrc.priv.tokens"
set smtp_url="smtp://myemail@smtp.office365.com:587/"
set smtp_authenticators="${imap_authenticators}"
set smtp_oauth_refresh_command="${imap_oauth_refresh_command}"
------------------------------------------------------------------------------

I did essentially the same as expressed above, the script was originally intended for use with Mutt but simple modifications makes it work with fetchmail as well.

I also execute fetchmail via cron instead of running it as a daemon process in order to facilitate refreshing the API token with each execution.

(In reply to Greg Grasmehr from comment #18)
> I also execute fetchmail via cron instead of running it as a daemon process
> in order to facilitate refreshing the API token with each execution.

Likewise. :-)

Thanks Jan and Greg, I will have a look at the mutt solution.
Colim

Hi All,
I am back on to this (with deadline looming, although I understand there is a grace period).

I have generated refresh and access keys using the mutt_oauth2.py script (with the devicecode option : the only option that worked) 

I can use the access key with fetchmail to read mail. 

But how do I use the refresh key to generate a new access key ?

I can't see how to do this with the mutt_oauth2.py script

I thought I would be able to do this using the fetchmail-oauth2.py script
- I decoded the refresh key with the gpg key set up for the mutt script
- I have set refresh_token_file to the decoded key in the properties file and also set access_token_file
- when I run ./fetchmail-oauth2.py -c ~/oauth2Config.properties --refresh
- I get "urllib.error.HTTPError: HTTP Error 400: Bad Request"

So I am really close but not quite there

Any help appreciated.

Regards,
Colin

I do it via a cron job - here are the particulars to refresh token and obtain new mail every 10 minutes, I have not had to reauth with a second factor since the initial auth I ran weeks ago

crontab -l
*/10 * * * * /home/User/fetch-my-mail.sh 2>&1 >> /dev/null


cat /home/User/fetch-my-mail.sh
#!/bin/bash

/usr/bin/python3.8 /home/User/mutt_oauth2.py -t /home/User/encKey > /home/User/.testcreds
/usr/bin/fetchmail --user User -v -k -t 3 -r oauthTest --logfile /home/User/fetchmail/fetch-log

cat /home/User/.fetchmailrc 
poll outlook.office365.com protocol imap auth oauthbearer passwordfile "/home/User/.testcreds"

file /home/User/encKey
encKey: PGP RSA encrypted session key - keyid: F19C6143 715C4D11 RSA (Encrypt or Sign)

Thanks Greg,
This is what I get when my access token expires and I try to get a new one.
I am going to check the client secret value with my Azure admin shortly. But this works when I use "--authorize" and was copied and pasted originally, so should be no typo.

--
./mutt_oauth2.py cfg/o365-refresh-token-20220927
401 Unauthorized
invalid_client
AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '
Trace ID: 67856476-3302-49a8-a71c-c2f1a5304c00
Correlation ID: 689175cf-f48b-459b-8eaa-b4ec86484de1
Timestamp: 2022-09-27 11:50:51Z
Perhaps refresh token invalid. Try running once with "--authorize"
--
$ file cfg/o365-refresh-token-20220927
cfg/o365-refresh-token-20220927: PGP RSA encrypted session key - keyid: B7A9EBCD 4466597E RSA (Encrypt or Sign) 2048b .
--

For the refresh each 1 minute I am using:

python3 ~/azul/fetchmail-git/contrib/fetchmail-oauth2.py -c /home/azul/.fetchmail-oauth2 --auto_refresh

.fetchmailrc:
auth oauthbearer username jkratochvil passwordfile "/home/azul/.fetchmail-token"

.fetchmail-oauth2:
...
refresh_token_file=/home/azul/.fetchmail-refresh
access_token_file=/home/azul/.fetchmail-token
...

Thanks again Jan and Greg,
I have this working now.
When I reviewed with our Azure admin I was using the secret id and not the secret value in client_secret
A new secret id/value was created and when I tried to use the secret value I got a different error when getting a new access token.
BUT, when I made the secret value empty it worked.

So, just in case anyone has similar issues, here is how it has worked for me.
1) Use mutt_oauth2.py with --authorize and option devicecode to get a refresh token and initial access token (does not matter if client_secret is specified or not)
2) Use mutt_oauth2.py with no switch and client_secret set as empty to get new access token
3) To prevent the need to use the gpg passphrase, decrypt the refresh token and save 
4) Use the fetchmail-oauth2.py with the decrypted refresh token specified as refresh_token_file and client_secret empty in the parameter file to get a new access token
5) My crontab script now runs as below to get a new access token when needed before runnning fetchmail

>> ${BINDIR}/fetchmail-oauth2.py -c ${CFGDIR}/oauth2Config.properties --auto_refresh
>> 
>> fetchmail

(In reply to goger from comment #13)
> Hello,
> 
> I'm very interesting by your rpm but I can't download it. I've an ssl error
> with wget or any web browser.
> 
> Can you fix the problem or explain me how to dowanload ?
> 
> Thank you very much.
> 
> Guillaume

Hi Guillaume,

The srpm is obsolete anyway. Upstream stopped releasing alpha tarballs and
suggests using git 'next' branch to test the code.

I've prepared Copr repository with fetchmail packages for RHEL8/9 (unofficial,
unsupported) based on this branch that also contain oauth2 related contrib files
and documentation, see:
https://copr.fedorainfracloud.org/coprs/vcrhonek/fetchmail-7.x-epel/

I'm trying fetchmail v7.devel from the git repository next branch.  

If you use mutt_oauth2.py, you can set "passwordeval" in your .fetchmailrc file
to be "<path_to>/mutt_oauth2.py <path_to>/<tokenfile>" and it will automatically
be invoked on fetchmail startup w/out needing a separate execution to update the access token.

I'm now trying to get fetchmail to rerun the passwdeval() function
at the beginning of each connection attempt (by hacking into the preconnect setting)
but there are some subtleties that make simple voodoo hacking not work.
That might be something for others to try.

Okay, here's what I've been able to accomplish.

I've created a small patch against the fetchmail v7 sources
(from https://gitlab.com/fetchmail/fetchmail.git; branch: next)
which allows fetchmail to use the mutt_oauth2.py OAuth2 helper
script from the mutt mailreader project to obtain and refresh
OAuth2 access tokens in daemon mode.

It does so by adding a special sentinel value, "!PASSWDEVAL",
to the "preconnect" fetchmailrc option which causes fetchmail
to rerun and reload the command defined by "passwordeval"
prior to each connection attempt to the email server.

With this patch and the mutt_oauth2.py script, one can configure
fetchmail to retrieve mail from Google GMail in daemon mode thusly:

  set daemon 300
  poll imap.gmail.com
    proto imap
    auth oauthbearer
    user username
    preconnect "!PASSWDEVAL"
    passwordeval "~/bin/mutt_oauth2.py ~/.gmail.oauth2"
    :

I'm submitting this patch as attachment fetchmail.wrs001.patch.

It seems to work successfully for me, but it needs more testing
to make sure there aren't any "gotchas" I've overlooked,
so please try it out.

Created attachment 1918030 [details]
Patch to fetchmail7 allowing use of OAuth2 in daemon mode

The following patch to the fetchmail 7 development branch (aka "next")
allows fetchmail to use the mutt_oauth2.py OAuth2 helper script from
the mutt mailreader project to obtain and refresh OAuth2 access tokens
in daemon mode.

It does so by adding a special sentinel value, "!PASSWDEVAL", to the
"preconnect" fetchmailrc option which causes fetchmail to rerun and
reload the command defined by "passwordeval" prior to each connection
attempt to the email server.

With this patch and the mutt_oauth2.py script, one can configure
fetchmail to retrieve mail from Google GMail in daemon mode thusly:

  set daemon 300
  poll imap.gmail.com
    proto imap
    auth oauthbearer
    user username
    preconnect "!PASSWDEVAL"
    passwordeval "~/bin/mutt_oauth2.py ~/.gmail.oauth2"
    :

                                        -- WRSomsky <somsky> 2022-10-14

As this is an update to fetchmail itself, I'm also going to look
into the procedure to submitting it to the upstream project.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.