Current File : //proc/thread-self/root/usr/share/doc/imapsync/FAQ.d/FAQ.Office365.txt
$Id: FAQ.Office365.txt,v 1.38 2022/06/22 17:07:35 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Office365.txt
======================================================================
Imapsync tips for Office365.
======================================================================
Questions answered in this FAQ are:
Q. Can I use imapsync to transfer from or to Office365 accounts?
Q. Can imapsync work for users that are administrators for
an Office 365 domain? (quick answer: no)
Q. Does imapsync support OAUTH2 authentication for Office365 accounts?
Q. How to sync from Office365 to XXX?
Q. How to sync from XXX to Office365
Q. For Office365 I have double and triple-checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
Q. How can I access an Office365 shared mailbox?
Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?
Q. Office365 fails with "User is authenticated but not connected".
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
Q. The imap connection to Office365 is not working very efficiently,
is there a solution to fix that?
Q. Office365 throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
Q. What are the receive and sending limits of Office365?
Q. The sync fails with many "Trying command when NOT connected!".
What can I do?
Q. How to see or migrate public folders on Office365?
R. https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/migrate-to-exchange-online?view=exchserver-2019
https://www.exchangesavvy.com/moving-your-public-folders-to-office-365-what-you-need-to-know/
Q. Office365 refuses to create the folder named "Files" with the error
"NO Folder name is reserved". What happens?
Q. Office365 users complain that a folder named "Files" contains
messages with no sender.
Q. From XXX to Office365, read receipts are all resent again after sync.
Even for old messages. How can I fix that?
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
Q. From XXX Office365 I get this error message sometimes:
"BAD Command Argument Error 11". What does it mean?
Q. From XXX to Office365 the flag Flagged does not seem to be well synced.
What can I do?
Q. How to migrate from or to Office 365 with an admin/authuser account?
Q. Couldn't create folder [trash] "Mailbox already exists".
Now the questions again with their answers.
======================================================================
Q. Can I use imapsync to transfer from or to Office365 accounts?
R0. Yes. But IMAP access to Office365 account is not always allowed
by default so it has to be allowed in the server configuration part.
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
R1. Imapsync doesn't support OAUTH2 authentication for Office365
accounts yet. That's not totally true, if you have an access token
to access your mailbox then you can authenticate with imapsync this
way:
imapsync ... --oauthaccesstoken1 tokenfile
where "tokenfile" is a file containing the access token.
R2. Enable double-step authentication and configure it but after use
an "app password" with imapsync. Details:
a) Go to
https://account.microsoft.com/security
b) Click on "Advanced Security Options"
Turn "Two-step verification" on. Follow the steps and finish".
c) Then now "App passwords" is available.
Click on "Create a new app password".
Use this password to authenticate with imapsync.
d) Delete this app password when the job with imapsync is finished.
R3. Also, check a license is assigned to that account in Office365.
R4. From Dave Pusey
https://github.com/imapsync/imapsync/issues/317#issuecomment-1027776418
I quote Dave nearly verbatim:
"I had created an app password, and security defaults are already off.
I have now figured out the issue.
Despite the MS365 and EXO admin centers showing that IMAP and Basic
Auth were all enabled, it turns out that in Oct 2021, Microsoft began
disabling basic auth for all tenants that had have never used it by
that point. There was an item in my Message Center from that date
saying my tenant was being done.
You can re-enable it for specific protocols (IMAP in this case) using
the diagnostic process detailed at
https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-september-2021-update/bc-p/2782230
(see the section "Limited Opt Out")
R5. Microsoft introduced something called "security defaults" which is enabled
by default for new tenants. One of the rules blocks IMAP access as of imapsync.
The funny thing is that you can't disable a single rule of this
security package without buying additional licenses.
Switching the whole thing off allows the IMAP login.
Also, disable double-step authentication on the Azure/Active Directory portal.
See here:
https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults
Thanks to Stephan Buhre for this R5 answer.
R6. Are there special characters in the password?
https://imapsync.lamiral.info/FAQ.d/FAQ.Authentication_failure.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt
R7. Triple check the hostname then. Try all of these:
* outlook.office365.com
* imap-mail.outlook.com
* imap.outlook.com
======================================================================
Q. Can imapsync work for users that are administrators for
an Office 365 domain? (quick answer: no)
R. I doesn't seem possible to use imap for administrators, so imapsync
won't be able to work for an administrator's mailbox.
See:
https://github.com/imapsync/imapsync/issues/310#issuecomment-1002396218
https://exhaust.lewiscollard.com/post/146866104/office365-to-migadu-migration/
Solution: as explained in the article above, use Davmail as a proxy to
access this mailbox. http://davmail.sourceforge.net/
Thanks to Lewis Collard for this report and solution.
======================================================================
Q. Does imapsync support OAUTH2 authentication for Office365 accounts?
R. Yes but partially. Imapsync won't help you getting an access token
but if you have one then you can use it with imapsync this way:
Office365 as source:
imapsync ... --oauthaccesstoken1 tokenfile1
Office365 as destination:
imapsync ... --oauthaccesstoken2 tokenfile2
======================================================================
Q. How to sync from Office365 to XXX?
R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
If IMAP cannot be enabled or if it doesn't work well, you can try DavMail
(http://davmail.sourceforge.net/) which acts as an IMAP gateway
through Outlook Web Access.
R. On Windows, use:
imapsync.exe ... --office1
On Unix, use:
imapsync ... --office1
Option --office1 is like (release 1.970 or higher):
imapsync ... --host1 outlook.office365.com \
--ssl1 \
--exclude "^Files$"
======================================================================
Q. How to sync from XXX to Office365
R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
If IMAP cannot be enabled or if it doesn't work well, you can try DavMail
(http://davmail.sourceforge.net/) which acts as an IMAP gateway
through Outlook Web Access.
R. Here is a command-line resume that solves most encountered issues when
migrating to Office365.
imapsync ... --office2
which is equivalent to (in imapsync release 1.870 or higher):
imapsync ... \
--host2 outlook.office365.com \
--ssl2 \
--maxsize 45000000 \
--maxmessagespersecond 4 \
--disarmreadreceipts \
--regexmess "s,(.{10239}),$1\r\n,g" \
--f1f2 "Files=Files_renamed_by_imapsync"
On Linux, you can also try the "reformime" command
that can be used like:
imapsync ... --maxlinelengthcmd "reformime -r7"
To get reformime, install the "maildrop" package.
On Linux again, there is a good Python script in the tarball that can
fix several things that Exchange or O365 have issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
======================================================================
Q. For Office365 I have double and triple-checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
R. Go to "Q. Can I use imapsync to transfer from or to Office365 accounts?"
======================================================================
Q. How can I access an Office365 shared mailbox?
R. First, create a shared mailbox, for example shared@example.com.
Then give full permissions to a licensed account user@example.com.
Now with imapsync, use the licensed user login with the syntax:
user@example.com\shared@example.com
and the password for user@example.com
Caveat: Character \ is a special character, so
use double-quotes around the user login, like:
imapsync ... --user1 "user@example.com\shared@example.com"
Sources:
https://adam-hand.com/2017/07/25/connect-a-shared-mailbox-from-o365-to-outlook-via-imap/
https://www.arclab.com/en/kb/email/imap-settings-shared-mailbox-office-exchange-online.html
https://social.technet.microsoft.com/Forums/en-US/336e02ee-6767-4810-90a0-1352bd7cc9e9/office-365-how-to-access-a-shared-mailbox-using-imap-client?forum=onlineservicesexchange
======================================================================
Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?
R. Use the option --subfolder2 backup_foo
imapsync ... --user2 sharedloginsyntax --subfolder2 backup_foo
To restore, do the reverse with:
imapsync ... --user1 sharedloginsyntax --subfolder1 backup_foo
======================================================================
Q. Office365 fails with "User is authenticated but not connected".
R1. "The message User is authenticated but not connected is due to a
bug in the Office365 server's IMAP implementation. If the client
presents a valid user name but an invalid password, the server
accepts the login, but subsequent commands fail with the
aforementioned error message." Source:
https://unix.stackexchange.com/questions/164823/user-is-authenticated-but-not-connected-after-changing-my-exchange-password
Thanks to James Abbottsmith for this link and explanation at
https://github.com/imapsync/imapsync/issues/32#issuecomment-153561647
R2. Miguel Alameda reported understanding and solving this issue
like this, the context was admin/authuser:
"The admin user had not permission in the target mailbox."
======================================================================
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
R. Office365 supports send/receive max message sizes of up to 150MB
but you need to make changes in your tenant(s) to support it.
The following PowerShell command will increase the message sizes that
can be sent/received. The trick in getting IMAPSync to work is to
apply these settings to the accounts performing the migration,
NOT the accounts associated with the target mailbox (assuming you're
using service accounts to perform transfers on behalf of users).
Set-mailbox -Identity $UPN -MaxReceiveSize 150mb -MaxSendSize 150mb
e.g.
Set-mailbox -Identity "migrationaccount@testtenant.onmicrosoft.com" -MaxReceiveSize 150mb -MaxSendSize 150mb
We're transferring data between Office 365 tenants so we set these
values on the migration accounts in the source and target tenants.
Thanks to Sean McDougall, Ian Thomas & Matt Wilks from Toronto
for this FAQ item.
======================================================================
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
R. To remove the flag $MDNSent from all messages, use:
imapsync ... --regexflag "s/\$MDNSent//g"
See also the document
https://imapsync.lamiral.info/FAQ.d/FAQ.Flags.txt
======================================================================
Q. The imap connection to Office365 is not working very efficiently,
is there a solution to fix that?
R. Yes. Try DavMail
http://davmail.sourceforge.net/
I don't use it myself but a user, Yannick Palanque, reported
great results using it.
======================================================================
Q. The sync fails with many "Trying command when NOT connected!".
What can I do?
R. The --debugimap option can show you more details, especially
messages like this one:
"BAD Request is throttled. Suggested Backoff Time: 178755 milliseconds"
In that case, see the next faq item and its fixes.
======================================================================
Q. Office365 throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
R. Office365 has throttle mechanisms to limit any huge usage.
Sometimes imapsync transfers are too stressful for servers.
The following message
"Request is throttled. Suggested Backoff Time: 299961 milliseconds"
comes from the imap Office365 server, imapsync just reports it
before being disconnected from it.
To solve the throttles issues from 0365, there are two solutions
at least:
R1. Call Microsoft Office365 and ask them to remove the limits on your
mailboxes. That's not a joke, they do it for 90 days usually,
sometimes only after you reach the second technician you call,
the first one usually be not enough competent to understand
what you're talking about (I would be glad to remove this bad
fact).
R2. Play with options --maxbytespersecond or --maxmessagespersecond
or --exitwhenover
imapsync ... --maxbytespersecond 100_000
imapsync ... --maxmessagespersecond 2
imapsync ... --exitwhenover 1_000_000_000
I don't know the upper value that avoids the default throttling from 0365
and I guess it changes over time.
======================================================================
Q. What are the receive and sending limits of Office365?
R1. 3600 messages per hour (22 July 2021). Maybe 1200 as imapsync can
be viewed as a single sender. Tell me what you experienced,
I'll update this point.
https://docs.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#receiving-and-sending-limits
In that case, imapsync can adapt to this with (1 message/second = 3600 messages per hour)
imapsync ... --maxmessagespersecond 1
or
imapsync ... --maxmessagespersecond 0.33
R2. I also found "Microsoft theoretically allows for about 300MB of
throughput per user per hour." at
https://www.systools.in/blog/microsoft-office-365-throttling-policy/
In that case, imapsync can adapt to this with (83333 bytes/second = 300 MBytes/hour)
imapsync ... --maxbytespersecond 83333
======================================================================
Q. Office365 refuses to create the folder named "Files" with the error
"NO Folder name is reserved". What happens?
R. The folder Files is a standard folder in Office365.
It should not be synced in IMAP. See the next question.
======================================================================
Q. Office365 users complain that a folder named "Files" contains
messages with no sender.
R0. To fix this, add --exclude Files
imapsync ... --exclude Files
If you use --office1 then imapsync will add this exclusion
automatically like using the option:
imapsync ... --exclude "^Files$"
If you use --office2 then imapsync will add a renaming of any
"Files" folder on host1, like using the option:
imapsync ... --f1f2 "Files=Files_renamed_by_imapsync"
The host2 account ends up with a folder named
"Files_renamed_by_imapsync", but no complaining.
R1. This folder "Files" seems to be a standard folder in Exchange Online,
but it is not. This folder contains all attachments in every email
that is in the mailbox but without any headers.
This causes some confusion for users as these appear in their search
results as duplicate lines but without the sender details or even
the message body.
This folder seems to be usually hidden so IMAP clients can’t see it,
but for some reason sometimes it becomes visible.
R2. It looks to be a common problem with Exchange Online.
I’m not sure what causes the folder to appear.
More info here:
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_exchon-mso_o365b/exclude-the-exchange-online-system-folder-called/2adbdf84-db4a-4c7f-ac29-738757980a0d
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_dep365-mso_o365b/no-sender-emails-in-files-folder/534bae8f-a7d7-4f5f-8ed7-5bad0d5fa23f
(This question/answer is taken quasi verbatim from Perttu Aaltonen)
======================================================================
Q. From XXX to Office365, read receipts are all resent again after sync.
Even for old messages. How can I fix that?
R. Imapsync can remove the header containing this read-receipt request.
On Unix or Windows use:
imapsync ... --disarmreadreceipts
Since read receipts should be sent for unseen messages that will go
to a seen state after the migration, you could be strict and apply
the regex only to seen messages.
Selecting seen message can be done with:
imapsync ... --search1 "SEEN" --disarmreadreceipts
A second run has to be run without the --disarmreadreceipts
for unseen messages:
imapsync ... --search1 "UNSEEN"
If fact --disarmreadreceipts is just an option equivalent to:
--regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms'
That regex changes the header Disposition-Notification-To. It prefixes
it with an X- so that it becomes inactive.
Disposition-Notification-To: blabla
becomes
X-Disposition-Notification-To: blabla
Thanks to David Karnowski for pointing and solving this issue.
======================================================================
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
R1. "SSL read or write error" happens sometimes, it isn't related to
imapsync directly but to the ssl underlying library when communicating
with Exchange in TLS/SSL encrypted mode.
Next runs should put the sync further, so rerun the syncs
until it is well completed.
R2. Another solution is to remove --tls or --ssl options for Exchange
and accept clear text syncs.
R3. See also the FAQ FAQ.SSL_errors.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.SSL_errors.txt
======================================================================
Q. From XXX Office365 I get this error message sometimes:
"BAD Command Argument Error 11". What does it mean?
R. This error message comes from the Office365 IMAP server when it
encounters any problem. Most of the time it is one of the following:
* Some messages are bigger than the size limit.
45 MB by default on Office365.
I don't know if it can be upped by configuration for Office365.
If you can't fix this limit on Office365 then use the option
--maxsize 45000000 for 45 MB to tell imapsync to skip those messages.
imapsync ... --maxsize 45000000 # 45 MB for Office365
* Quota reached. The whole account is full.
It can be upped by configuration.
* You use --synclabels --resynclabels from a previous command line
related to syncing from Gmail to Gmail. Remove them.
* Some messages have some lines too long. Use option --maxlinelength
to skip messages whose max line length is over some bytes.
--maxlinelength 1000 is an RFC2822 must but most servers support
higher values. Office365 supports 10500 characters line length:
imapsync ... --maxlinelength 10500
In case you prefer fixing messages with long lines the hard way,
instead of skipping them with --maxlinelength 10500, just use:
On Windows
imapsync ... --regexmess "s,(.{10500}),$1\r\n,g"
On Unix
imapsync ... --regexmess 's,(.{10500}),$1\r\n,g'
Have also in mind that Office365 closes the connection after 10 errors
encountered so you might also see "BYE Connection closed" errors from
Office365, which means Office365 leaves the session and says goodbye,
come back later. Redo some sync then.
On Linux, there is a good Python script in the tarball that can
fix several things that Office365 has issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
It's called fix_email_for_exchange because Office365 is Exchange,
or at least started to be an Exchange server with the same issues.
======================================================================
Q. From XXX to Office365 the flag Flagged does not seem to be well synced.
What can I do?
R. Use the following trick. Run imapsync twice, one with --regexflag
and one without, like this:
1) imapsync ... --regexflag "s/\\Flagged//g"
2) imapsync ...
You can add --debugflags if you want to see what imapsync gets and
does in detail with flags.
The magic of this trick is on ignoring the \Flagged flag on the first
sync and setting it on the second sync, with STORE instead of APPEND.
This Office365 bug seems that Office365 gets and sets well the Flagged
flag with APPEND in IMAP but then it forgets it with other protocols;
With STORE it sets and gets the "\Flagged" flag everywhere.
Thanks to Dave Murray and Simon Savva for reporting and solving
this issue.
======================================================================
Q. How to migrate from or to Office 365 with an admin/authuser account?
Note from Yago Torres Fernandez:
(a working command using admin/authuser on host2 Office 365)
imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated@domain.com ^
--password2 XXXX --ssl2
but previously in Office365, you must do something like the following, using Powershell:
Add-MailboxPermission -identity user_to_be_migrated@domain.com -user user_admin@domain.com -accessrights fullaccess -inheritancetype all
Note from Betsy Lawlor: You can use global modern authentication with two factor
on Exchange Online (M365) but you must have "AllowBasicAuthImap" on
the admin account you are using to migrate the mail.
Note from Guido (5 April 2022):The way I fixed it was by turning off security defaults
https://docs.microsoft.com/nl-nl/azure/active-directory/fundamentals/concept-fundamentals-security-defaults.
You still need to check IMAP access on an account-basis though.
Remark: PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
Note from Rafael Alvarez Ballesteros:
When you get an Office365 license you will receive an admin user to handle
your licenses and products like admin@yourcompanyname.onmicrosoft.com.
This account is the administrator account; some weeks or months ago Microsoft
has decided you need to use two-factor authentication by default,
so if two factors authentication is enabled you will no be able to sync
the mail (it will not connect to host2).
OWA (I think this is the two-factor authentication) needs to be disabled
globally and enabled individually on the users you want to.
One account can have the right to access other mailboxes no matter if admin or
any other account but needs to have OWA disabled to be able to connect
to the office365 server.
Note from Martin Paulucci:
I had to remove the domain part for the user
but not for the admin. Example:
imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated
See also:
http://linux-france.tk/prj/imapsync_list/msg02203.html
Subject: RE: [imapsync] Office 365 - 'Master User'?
Date: Mon, 1 Jun 2015 17:53:54 +0000
======================================================================
Q. Couldn't create folder [trash] "Mailbox already exists".
R. Some servers take care of character cases in folder names,
some servers do not, like Exchange. Since not respecting the case
can merge two different folders into one then imapsync respects the case.
For example, if a host1 server has a folder name called "trash"
and the host2 server already has a folder "Trash" or "TRASH"
then imapsync will try to create the folder "trash" on host2
because trash and Trash are different strings. But if host2
does not respect character case it will consider folder "trash"
already exists and will say it. That's the error message reported
by imapsync: "Mailbox already exists". This message comes from the server.
The folder creation fails but messages are well transferred in
so take a look at this warning, understand why it happens
and it should be fine most of the time.
To avoid this warning use --regextrans2 to map the folder names
imapsync ... --regextrans2 "s/^trash$/Trash/"
If there are two folders Trash and trash on host1 then both
will be merged into only one Trash folder on host2.
In case of the option --delete2 is used the regextans2 above becomes
mandatory, otherwise imapsync will sync messages from the
first Trash and then delete them when syncing trash.
If you want to avoid merging folders that are considered different
on host1 but are considered the same on destination host2 because of
case sensitivities and insensitivities, use --nomixfolders
======================================================================
======================================================================
Mini Shell