Migrate with mongomirror
¶
On this page
mongomirror
is a tool for manually migrating data from an existing
MongoDB replica set to a MongoDB Atlas replica set.
mongomirror
does not require you to shut down your existing
replica set or applications, and does not import user or role data
or copy the config
database.
Use mongomirror
for:
- Running a one-time migration of a dataset into a MongoDB Atlas cluster from a MongoDB deployment hosted outside of MongoDB Atlas.
- Running a one-time migration of a dataset from one Atlas cluster into another Atlas cluster.
See also Choosing a Data Import and Migration Tool.
Upgrade Path¶
mongomirror
supports the following migration paths:
Source Replica Set MongoDB Version | Target Atlas Replica Set MongoDB Version |
---|---|
5.0 | 4.4, 5.0 |
4.4 | 4.2, 4.4, 5.0 |
4.2 | 4.0, 4.2, 4.4, 5.0 |
4.0 | 3.6, 4.0, 4.2, 4.4, 5.0 |
3.6 | 3.6, 4.0, 4.2, 4.4, 5.0 |
3.4 | 3.6, 4.0, 4.2, 4.4, 5.0 |
3.2 | 3.6, 4.0, 4.2, 4.4, 5.0 |
3.0 | 3.6, 4.0, 4.2, 4.4, 5.0 |
2.6 | 3.6, 4.0, 4.2, 4.4, 5.0 |
To downgrade from 4.4 to 4.2, 4.2 to 4.0, or 4.0 to 3.6, the
source FCV
must match the target.
Download mongomirror
¶
Operating System | Download |
---|---|
Amazon Linux 1 | |
Amazon Linux 2 | |
Debian 8.1 | |
Debian 9.2 | |
Debian 10 | |
macOS 64-bit | |
RHEL 6.2 | |
RHEL 7.0 | |
RHEL 7.1 PPC64LE | |
RHEL 8.0 | |
RHEL 8.0 s390x | |
RHEL 8.1 PPC64LE | |
SLES 12 | |
SLES 15 | |
Ubuntu 14.04 | |
Ubuntu 16.04 | |
Ubuntu 16.04 ARM | |
Ubuntu 16.04 PPC64LE | |
Ubuntu 18.04 | |
Ubuntu 18.04 ARM | |
Ubuntu 18.04 PPC64LE | |
Ubuntu 20.04 | |
Ubuntu 20.04 ARM | |
Windows |
Prerequisites¶
Source MongoDB Deployment¶
- The source MongoDB deployment must be a replica set. If the
source is a standalone MongoDB deployment, before running
mongomirror
, convert the standalone to a replica set. - The source replica set must run MongoDB version 2.6 or higher.
- The source MongoDB replica set cannot be an
M0
free cluster orM2/M5
shared cluster. - Don't change the
featureCompatibilityVersion
flag at any time during the procedure. - When you migrate from MongoDB 4.4 or earlier to an Atlas cluster running MongoDB 5.0 or later, drop geoHaystack indexes from your collections.
- Don't perform a resharding operation on the source cluster during a migration.
mongomirror
is not compatible with TTL indexes. Drop any existing TTL indexes and rebuild them when the migration process is complete. If you do not want to drop an existing index because it is important for query performance, contact support for alternative options.- Don't make any namespace changes during the migration
process, such as using the
renameCollection
command or executing an aggregation pipeline that includes the$out
aggregation stage. - If you must rename or drop a collection during the migration process, contact Atlas support.
Required Access on Source Replica Set¶
mongomirror
must have network access to the source
replica set. If the source replica set requires authentication,
you must include user credentials when running mongomirror
.
You must specify a database user with, at a minimum, the following privileges:
- Read all databases and collections. The
readAnyDatabase
role on theadmin
database covers this requirement. - Read the oplog. See Oplog Access.
- Run the
getParameter
command.
If no such user exists, create the user in your source MongoDB
replica set. Different MongoDB server versions have different
built-in roles. Select a built-in role based on your MongoDB
server version and run the appropriate commands in the
mongosh
:
For source clusters running MongoDB version 3.4 or later a user must have, at a minimum, both
clusterMonitor
andreadAnyDatabase
roles. For example:use admin db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "clusterMonitor", "readAnyDatabase" ] } ) For source clusters running MongoDB version 3.2 a user must have, at a minimum, both
clusterManager
andreadAnyDatabase
roles, as well as read access on thelocal
database. This requires a custom role, which you can create with the following commands:use admin db.createRole( { role: "migrate", privileges: [ { resource: { db: "local", collection: "" }, actions: [ "find" ] } ], roles: ["readAnyDatabase", "clusterManager"] } ) db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "migrate" ] } ) For source clusters running MongoDB version 2.6 or 3.0 a user must have, at a minimum, the
readAnyDatabase
role. For example:use admin db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "readAnyDatabase" ] } )
Target Atlas Deployment¶
The target Atlas deployment:
- Can't be an
M0
free cluster,M2
, orM5
shared cluster. - Can't be a serverless instance.
- Must be a replica set.
- Must have the version that is the same as or later than the source cluster MongoDB version. See Upgrade Path.
- Must not contain any namespaces that overlap with the source cluster.
If
mongomirror
detects overlapping namespaces on the target cluster it fails before starting the process. If your target cluster contains overlapping namespaces, delete all data from the target cluster with--drop
, or list the namespaces to import from the source cluster with--includeNamespace
. Must include in its IP access list either:
- The public IP address of the server on which
mongomirror
is running, or - If set up for VPC peering, either the peer's VPC CIDR block (or a subset) or the Security Group of the peer VPC.
- The public IP address of the server on which
Required Access on Target Cluster¶
mongomirror
must have network access to the target cluster.
You must specify a database user with the Atlas admin
role to run mongomirror
. To learn more, see Add Database Users.
mongomirror
Process¶
When you start mongomirror
, it:
- Connects to Atlas over TLS/SSL.
- Performs an initial sync, copying collections from the existing MongoDB replica set to the target cluster in Atlas.
After the initial sync, continuously tails the replica set's oplog for incoming changes and replays them on the target cluster in Atlas.
mongomirror
does not copy theconfig
database. As of version 0.9.0,mongomirror
uses wire compression if you enable it on either the source or the target cluster and use--compressors
to specify which compression libraries to allow.NoteStarting in version 0.5.0,
mongomirror
builds all indexes on the target cluster in the foreground, regardless of how the indexes were built on the source cluster. Foreground index builds block all other operations on the database. To learn more, see Index Build Operations on a Populated Collection.
Once started, mongomirror
runs continuously until you shut it down:
- If you shut down
mongomirror
during the initial sync stage, before you restartmongomirror
, check that the target cluster is empty, or runmongomirror
with--drop
. - If you shut down
mongomirror
during the oplog tailing stage, the process stops tailing the oplog. You can restartmongomirror
to continue tailing from the last processed oplog record using--bookmarkFile
.
Run mongomirror
¶
Set up database user in the source replica set.¶
If the source replica set requires authentication, you must include
user credentials when running mongomirror
. For requirements, see
Required Access on Source Replica Set.
Make note of the username and password for this user, as you must
specify these credentials when running mongomirror
.
Set up a database user in the target Atlas cluster.¶
You must specify a database user with the
Atlas admin
role to run mongomirror
.
See Add Database Users for documentation on creating a database
user.
If no such user exists, create the user:
- In the Security section of the left navigation, click Database Access. The Database Users tab displays.
- Click Add New Database User.
- Add an Atlas admin user.
Make note of the username and password selected for the new
user, as you must specify these credentials when running
mongomirror
.
Update IP Access List.¶
If the host where you will run mongomirror
is not in your cluster's
IP Access List, update the list.
You can specify either:
- The public IP address of the server on which
mongomirror
will run, or - If set up for VPC peering, either the peer's VPC CIDR block (or a subnet) or the Security Group of the peer VPC.
Open the connect dialog.¶
- Click Databases in the top-left corner of Atlas.
- From the Database Deployments view, click Connect for the Atlas cluster into which you want to migrate data.
Copy the target cluster host information.¶
You can get your Atlas cluster's hostname information from the Atlas user interface.
You don't need to use a driver to migrate data with mongomirror
.
The connection string should look similar to the following example. This example has been broken into multiple lines for readability:
mongodb://<username>:<PASSWORD>@ 00.foo.mongodb.net:27017, 01.foo.mongodb.net:27017, 02.foo.mongodb.net:27017/test? ssl=true&replicaSet=myAtlasRS&authSource=admin
Copy the value of replicaSet
and append the host list as comma
separated values, similar to the following:
myAtlasRS/00.foo.mongodb.net:27017,01.foo.mongodb.net:27017,02.foo.mongodb.net:27017
Use this value for --destination
in the next step.
Start mongomirror
.¶
Start mongomirror
with the following options:
--host
set to the host string of the source cluster.--ssl
if the source cluster has TLS/SSL enabled.--username
to the source cluster username created for this procedure.--password
to the password for the source cluster username.--authenticationDatabase
to the database in the source cluster where the username specified to--username
was created.--destination
to the Atlas cluster host string.--destinationUsername
to the target Atlas cluster username.--destinationPassword
to the password for the target Atlas cluster username.
Your final command should resemble the following example:
mongomirror --host "MySourceRS/host1.example.net:27017,host2.example.net:27017,host3.example.net:27017" \ --ssl \ --username "mySourceUser" \ --password "mySourceP@$$word" \ --authenticationDatabase "admin" \ --destination "myAtlasRS/00.foo.mongodb.net:27017,01.foo.mongodb.net:27017,02.foo.mongodb.net:27017" \ --destinationUsername "myAtlasAdminUser" \ --destinationPassword "atlasPassword"
See Options for more mongomirror
options.
To complete the migration process, switch to Atlas.
Switch to Atlas¶
After mongomirror
completes the initial sync and tails
the replica set's oplog, you can switch over to the target
Atlas cluster.
Stop your application.¶
This ensures that no additional writes occur on the source cluster.
Wait for mongomirror
to report 0s of lag between the source and destination clusters.¶
This signifies that the source and destination clusters are in a consistent state.
Stop mongomirror
.¶
Once the Atlas cluster is up to date, stop mongomirror
.
Update client applications to use the Atlas cluster.¶
Update your client application(s) with the Atlas connection string provided via the Connect button on the cluster panel.
For details on connections to Atlas, see Connect via Driver.
Monitoring¶
mongomirror
logs its progress to the standard output in the terminal.
During the initial sync, mongomirror
logs a progress bar for each
collection it copies. For example:
2021-08-09T16:35:50.227-0000 [#....................] park.events 2179/34184 (6.4%) 2021-08-09T16:35:50.227-0000 [#############........] zoo.animals 29000/49778 (58.3%)
When tailing the oplog, mongomirror
logs the lag time, in seconds,
between the most recent oplog entry on the source cluster and the last
processed oplog entry on the target cluster. For example:
2016-12-12T16:22:17.027-0800 Current lag from source: 6s
A lag time of 6 seconds means that the last oplog entry
mongomirror
processed occurred 6 seconds before the most
recent one available on the source cluster.
The amount of time it takes mongomirror
to catch up may be
greater or less than 6 seconds, depending on the number of entries
that arrive per second.
A lag time of 0 seconds indicates that mongomirror
is processing
entries that arrived less than one second before the latest oplog entry.
If the source cluster has indexes, mongomirror
builds the same indexes
on the target cluster. The progress log shows the start and end times of
the index building process. To view the progress of the index builds,
you can either:
Use the currentOp command on the primary node of the target cluster. The
command
field shows information about the running operation. Index building entries look similar to the following:"command" : { "createIndexes" : "perfs", "indexes" : [ { "key" : { "animal" : "text" }, "name" : "animal_text" } ]... Download the mongodb logs for your target cluster and search for recent entries for index-related lines. Log messages related to index building look similar to the following:
{"t":{"$date":"2021-08-09T16:35:50.227+00:00"},"s":"I", "c":"INDEX", "id":20447, "ctx":"conn1080","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"c4a62739-bf19-456d-bbd6-7baa29f1063b"}}}}
Performance¶
To avoid contention for network and CPU resources, run mongomirror
on
hosts other than your replica set's mongod
instance hosts.
mongomirror
affects your source replica set's performance similar to having a secondary:- For the initial sync stage, the load scales with the size of your data set.
- Once an initial sync completes, the load scales with oplog gigabytes used per hour.
Command Syntax, Options, and Examples¶
For mongomirror
syntax, options, and examples, see the
mongomirror reference page.