There are two basic ways to migrate a mail server from one machine to another: with and without service downtime.
Migrating a mail server with scheduled downtime
Use this method when you can allow for a brief downtime of mail services.
Step 1. Prepare the migration
Complete the setup of your new mail server. Lower the TTL in your DNS config.
Step 2. rsync mailboxes on the fly
Rsync mailboxes from the old server to the new mail server on the fly. This allows you to limit scheduled downtime.
Copying mailboxes on the fly has one big upside: You can silently test a specific user to see whether everything works or not.
Read the rsync manual very carefully. Test it on one mailbox first. Select appropriate flags.
Step 3. Set up DNS routing
Set up your new mail server as a secondary mail exchanger.
Step 4. Stop both MTA and MDA services on your old mail server
Your SMTP server must not accept any new mail. This also means that you must prevent your mail users from accessing their mail directories to prevent them from making changes (saving drafts of new email, for example). In practice, this requires you to stop the MDA (e.g. Dovecot) as well as the MTA (e.g. Postfix).
Your new mail server is not yet running. This is your scheduled downtime.
Step 5. Rsync final changes
During your scheduled service downtime, rsync only those files that have changed. This will limit your downtime.
Step 6. Clean up
Mail queue will keep email that it cannot send. Look at your mail queue on the old server. It should be empty.
Deactivate the MX record for the old mail server and stop mail services on the old mail server.
Migrating a mail server without downtime
The migration is a lot more difficult when you cannot tolerate any downtime of the mail service; in this case it is wise to migrate mailboxes user by user.
Configure aliases on both servers. Both servers should accept emails for both names, but internally redirect email to the appropriate server using aliases.
After migrating the last user, point the MX entry to the new server only and wait until mailqueue on the old server empties out, then shut down that server.