Live Migrate (Pull) a Sharded Cluster into Atlas

Migration Path

Atlas live migration (pull) supports the following migration paths:

Pre-Migration Validation

Atlas performs a number of validation checks on the source and target clusters before starting the live migration procedure.

• The source cluster must be a sharded cluster.

  If the source is replica set cluster, use pull-type live migration to migrate the cluster to a Atlas replica set first, then scale your cluster to a sharded cluster.

  If the source is a standalone, convert the standalone to a replica set first before using pull-type live migration. Then scale your cluster to a sharded cluster.

• The source cluster must use CSRS (Config Server Replica Sets). See Replica Set Config Servers.
• Atlas must have connectivity to the hostname and port of each node in the source cluster.
• Atlas must be able to stop and start the Sharded Cluster Balancer on the source cluster.

• The source cluster has the same feature compatibility version and major MongoDB version as the target cluster. The major MongoDB verison is the first two digits of the full version, such as 4.0.x, 4.2.x, or 4.4.x.

  To check the feature compatibility version of a host in the source cluster, run the following command from mongosh:

  db.runCommand( { getParameter : 1, "featureCompatibilityVersion" : 1 } )

  Use the setFeatureCompatibilityVersion database command to set the featureCompatibilityVersion flag as needed.

• The target Atlas sharded cluster must have the same number of shards as the source sharded cluster.

Source Cluster Security

Various built-in roles provide sufficient privileges. For example:

• For source clusters running MongoDB 3.6 or later a user must have the readAnyDatabase, the clusterMonitor, and the backup roles.

  To verify that the database user that will run the live migration process has these roles, run the db.getUser() command on the admin database.

  use admin
  db.getUser("admin")
  {
    "_id" : "admin.admin",
    "user" : "admin",
    "db" : "admin",
    "roles" : [
      {
        "role" : "backup",
        "db" : "admin"
      },
      {
        "role" : "clusterMonitor",
        "db" : "admin"
      }
      {
        "role" : "readAnyDatabase",
        "db" : "admin"
      }
    ]
  } ...

  In addition, the database user from your source cluster must have the role to read the oplog on your admin database. To learn more, see Oplog Access.

• For source clusters running MongoDB 3.4 a user must have, at a minimum, both clusterMonitor and readAnyDatabase roles. For example:

   use admin
   db.createUser(
     {
       user: "mySourceUser",
       pwd: "mySourceP@$$word",
       roles: [ "clusterMonitor", "readAnyDatabase" ]
     }
   )

• For source clusters running MongoDB 3.2 a user must have, at a minimum, both clusterManager and readAnyDatabase roles, as well as read access on the local database. This requires a custom role, which you can create with the following commands:

    use admin
    db.createRole(
         {
           role: "migrate",
           privileges: [
             { resource: { db: "local", collection: "" }, actions: [ "find" ] }
           ],
           roles: ["readAnyDatabase", "clusterManager"]
         }
       )
     db.createUser(
         {
           user: "mySourceUser",
           pwd: "mySourceP@$$word",
           roles: [ "migrate" ]
         }
       )

• For source clusters running MongoDB 2.6 or 3.0 a user must have, at a minimum, the readAnyDatabase role. For example:

    use admin
       db.createUser(
         {
           user: "mySourceUser",
           pwd: "mySourceP@$$word",
           roles: [ "readAnyDatabase" ]
         }
       )

Specify the user name and password to Atlas when prompted by the live migration procedure.

Atlas only supports SCRAM for connecting to source clusters that enforce authentication.

Network Encryption

During pull live migrations, if the source cluster does not use TLS encryption for its data, the traffic from the source cluster to Atlas is not encrypted. Determine if this is acceptable before you start a pull live migration procedure.

Source Cluster Balancer

Atlas live migration stops the sharded cluster balancer on the source cluster at the start of the procedure, and starts the balancer at the end of the procedure.

If you cancel live migration, Atlas restarts the balancer on the source cluster.

Target Cluster Configuration

When configuring the target Atlas cluster, consider the following:

• The live migration process streams data through a MongoDB-managed application server. Each server runs on infrastructure hosted in the nearest region to the source cluster. The following regions are available:

  Europe

  • Frankfurt
  • Ireland
  • London

  Americas

  • Eastern US
  • Western US

  APAC

  • Mumbai
  • Singapore
  • Sydney
  • Tokyo

• Due to network latency, the live migration process may not be able to keep up with a source cluster that has an extremely heavy write load. In this situation, you can still migrate directly from the source cluster by pointing the mongomirror tool to the destination Atlas cluster.
• The live migration process may not be able to keep up with a source cluster whose write workload is greater than what can be transferred and applied to the destination cluster. You may need to scale the destination cluster up to a tier with more processing power, bandwidth or disk IO.

• You cannot target a Global Cluster as the target for live migration.

  Important

  Once you start the live migration process, you cannot modify the target Atlas cluster. If you need to scale up the target cluster, cancel the live migration process, then scale up the cluster and restart the live migration process.

Database Users and Roles

Atlas does not migrate any user or role data to the target cluster.

If the source cluster enforced authentication, you must re-create the credentials used by your applications on the target Atlas cluster. Atlas uses SCRAM for user authentication. See Configure Database Users for a tutorial on creating database users in Atlas.

Canceling Live Migration

You can cancel the process at any time by clicking Cancel. Atlas displays the Sharded Cluster Live Import in Progress message for the target cluster until the cluster is ready for normal access.

If you cancel the live migration process before completion, Atlas does not remove any data migrated up to that point. If you restart the live migration process using the same Atlas cluster as the target cluster, Atlas wipes all data from the cluster.

Testing the Target Cluster

You may want to migrate data to your target cluster, then stop the migration process and test your target cluster while leaving the source cluster running and serving data to your applications.

To test your target cluster with production data, follow the migration procedure as far as step 2. When you're ready to perform the complete migration process, skip step 2 and proceed to step 3.

Avoid Namespace Changes

Don't make any namespace changes during the migration process, such as using the renameCollection command or executing an aggregation pipeline that includes the $out aggregation stage.

Avoid Elections

The live migration process makes a best attempt to continue a migration during temporary network interruptions and elections on the source or target clusters. However, these events might cause the live migration process to fail. If the live migration process can't recover automatically, restart it from the beginning.

Target Cluster Network Access

During live migration, the mongos processes on the target cluster are shut down and cluster connectivity via the mongos servers is suspended. The mongos processes restart automatically once migration is complete.

Rolling Restarts

After the migration process completes, your cluster restarts each of its members one at a time. This is called a rolling restart, and as a consequence, a failover will occur on the primary. To ensure a smooth migration, consider running a Test Failover procedure before migrating your data to the target cluster.

Pre-Migration Checklist

Before starting the import process:

• If you don't already have a destination cluster, create a new Atlas deployment and configure it as needed. For complete documentation on creating an Atlas cluster, see Create a Cluster.

• After your Atlas cluster is deployed, ensure that you can connect to it from all client hardware where your applications run. Testing your connection string helps ensure that your data migration process can complete with minimal downtime.

  1. Download and install mongosh on a representative client machine, if you don't already have it.
  2. Connect to your destination cluster using the connection string from the Atlas UI. For more information, see Connect via mongosh.

  Once you have verified your connectivity to your target cluster, start the import procedure.

Procedure