Manage Online Archives
On this page
- View Online Archives
- View Online Archives in the UI
- Retrieve an Online Archive Using the API
- Retrieve All Online Archives for a Cluster Using the API
- Edit an Archiving Rule
- Edit an Archiving Rule Through the UI
- Edit the Partition on the Cloud Object Store
- Edit an Archiving Rule Through the API
- Query Online Archive
- Connection String
- Performance Considerations
- Query Price
- Delete an Online Archive
- Delete an Online Archive from the UI
- Delete an Online Archive Through the API
Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.
After you configure an online archive, you can do the following:
View Online Archives
Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.
You can view the list of online archives for a cluster through the Atlas UI and API.
View Online Archives in the UI
To view the list of Online Archives:
Navigate to the Database Deployments page for your project.
- If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
- If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
- If the Database Deployments page is not already displayed, click Databases in the sidebar.
Navigate to the Online Archive tab for your cluster.
- Click the name of the cluster.
- Click the Online Archive tab to view the list of online archives, if any, for the cluster.
The page displays the online archives for the cluster. For each online archive, you can see the following information:
Column Name | Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Archive Field | The date field based on which documents are archived. | ||||||||||||||||
Archive Last Updated | The date when the archive was last modified. | ||||||||||||||||
Archival Age Limit | The number of days used to qualify documents for archiving. | ||||||||||||||||
Deletion Age Limit | The number of days after which to delete the data in the archive. | ||||||||||||||||
Partition Fields | The other commonly used query fields used for partitioning data
on the cloud object storage. | ||||||||||||||||
Status | The status of the Online Archive. Value can be one of the following:
| ||||||||||||||||
Actions | Operations that you can perform on the Online
Archive. |
Retrieve an Online Archive Using the API
To retrieve an online archive through the API, send a GET request
to the onlineArchives endpoint with the
unique ID of the online archive to retrieve. To learn more about the
API syntax and options, see Get One Online Archive.
Retrieve All Online Archives for a Cluster Using the API
To retrieve all the online archives configured for a cluster using
the API, send a GET request to the onlineArchives endpoint for the cluster. To learn more about
the syntax and options, see Get all Online Archives for Cluster.
Edit an Archiving Rule
Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.
You can modify the number of days to keep data on the Atlas cluster (the Age Limit) or the custom JSON query used to select documents for archiving from the Atlas UI and API. You can't change the archiving criteria from Date Match to Custom Filter, or vice versa.
Edit an Archiving Rule Through the UI
To edit an online archive, in your Atlas UI:
Navigate to the Database Deployments page for your project.
- If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
- If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
- If the Database Deployments page is not already displayed, click Databases in the sidebar.
Navigate to the Online Archive tab for your cluster.
- Click the name of the cluster.
- Click the Online Archive tab to view the list of online archives, if any, for the cluster.
Click the ellipsis (...) in the Actions column for the online archive to display the list of allowed actions.
You can:
- Pause Archiving (only if state is Active)
- Edit Archive
- Delete Archive
- Resume Archiving (only if state is Paused)
Select Edit Archive from the dropdown to make changes to your archiving rule, the number of days to keep archived data, and the time window for running data archiving jobs.
- To edit Date Match criteria, modify the number of days Atlas stores data on the active Atlas cluster in the Archival Age Limit section.
To edit Custom Criteria, enter a valid JSON filter to select the documents for archiving.
NoteAtlas uses the specified query with the db.collection.find(query) command. Custom queries do not support JavaScript expressions. Also, you can't pass an empty document
{}to return all documents.To modify the number of days after which Atlas deletes archived data, enter or modify the number of days in the Deletion Age Limit section. You can specify between
7and9125days, or leave the field empty to disable the data expiration rule. It takes 24 hours for this change to take effect.NoteAtlas Online Archive data expiration is available as a Preview feature. The feature and the corresponding documentation may change at any time during the Preview stage.
To modify the scheduled time for data archiving jobs, make changes to any of the following:
- Frequency. You can choose to run the job every day, on a specific day of the week, or on a specific date every month. If you wish to archive on the 29th, 30th, or 31st of every month, Atlas doesn't run the archiving job for those months that don't include those dates.
- Time window. Select the period of time during which you want Atlas to run the data archiving job. You must specify a minimum of two hours.
NoteOnline Archive scheduled archiving is available as a Preview. The feature and corresponding documentation may change at any time during the Preview stage.
You can also disable the schedule by toggling Schedule Archive Window. If you disable the schedule, Atlas reverts to the default schedule and runs the archiving job periodically.
Atlas starts using the new data archiving schedule immediately after you change it. However, if an archiving job is currently running, Atlas doesn't interrupt the running job and the setting takes effect after the job completes.
Click Save for the changes to take effect.
It takes 24 hours for changes to Deletion Age Limit to take effect.
Edit the Partition on the Cloud Object Store
You can't modify the partition fields or structure from the
Atlas UI or API. However, you can manually migrate the data from
the cloud object storage using mongodump, delete the online archive, use mongorestore to
restore the data on the Atlas cluster, and then create a new online
archive for the collection with the desired partition fields and
structure.
Edit an Archiving Rule Through the API
To edit an online archive through the API, send a PATCH request
to the onlineArchives endpoint with the
unique ID of the online archive to update. To learn more about the
API syntax and options, see Update an Online Archive.
Query Online Archive
You can run queries against your archived data.
Connection String
To run queries, you must first connect to your Online Archive. Your cluster connection string allows you to only query data in your Atlas cluster. To query your Online Archive, you must use one of the following:
- Connect to Online Archive and Cluster - this read-only connection string allows you to read data directly from the live cluster. This impacts available resources for IOPS, and from your Online Archive.
- Connect to Online Archive - this read-only connection string allows you to read data from the Online Archive only and doesn't affect cluster resources.
Performance Considerations
In general, your queries against archived data are much slower than your queries against data on the Atlas cluster. When you query data in your cluster and Online Archive through the federated connection string:
- Blocking queries, such as sorts that consume and process all input documents to the sort operation before returning results, have performance characteristics associated with the slowest storage, the archive, being queried. The sort operations require all data from the sources being queried before returning the results.
- Streaming queries, such as finds, have performance characteristics associated with the highest performing storage, the Atlas cluster, being queried. Atlas returns the results as soon as they are available, which means returning results from the archive takes longer than returning results from the Atlas cluster.
Query Price
For your federated and archive-only queries, you incur costs for the following items.
Data Scan
During data scan, Atlas processes data from both the cluster and
the archive. Atlas runs as much of the query on the cluster as it
can to minimize the amount of data it needs to scan. For example, for a
match query that specifies a specific value, Atlas only
retrieves documents with the specified value from the cluster.
Atlas then combines the retrieved documents with the archived data
and returns.
For blocking queries that need to access all data stored in the
underlying cluster, Atlas retrieves all data. For example, for a
sort (with no match), Atlas retrieves all data from the
cluster and archive to be sorted.
Data Access
MongoDB charges a fee for each partition that you query in the archive. If your query requires querying specific partitions, MongoDB downloads the partitions and each downloaded partition corresponds to a single access.
Data Seek
To find partitions based on the query and query fields, Atlas runs operations on the archive. Each such operation that Atlas runs finds up to 1000 partitions. Atlas runs the minimum number of required operations to find the partitions required to satisfy the query. For example, if your query requires 100 partitions that are covered in your query fields, Atlas runs only one operation to satisfy the query.
Data Transfer
Data that is transferred to the federated infrastructure incurs data transfer costs.
Delete an Online Archive
Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.
You can delete an online archive through the Atlas UI and API. When you remove an online archive, you also delete data on the cloud object storage for the online archive. After you delete this data, you can't restore it.
If you drop a database or collection configured for online archive, the data from the collection, if archived, continues to be available on the cloud object storage. You incur costs for storage on the cloud object storage. Alternatively, if you delete the cluster, Atlas deletes all the online archives configured for the cluster. This also deletes any archived data from the cloud object storage.
If you delete all the online archives, you also delete the Data Lake and you create a new Data Lake when you create an online archive again.
After you delete an online archive, its state moves to Deleted.
You can create another online archive for the same database,
collection, and fields as the deleted online archive if there is no
other online archive for the same database, collection, and fields in
the Active state.
Delete an Online Archive from the UI
To delete an online archive, in your Atlas UI:
Navigate to the Database Deployments page for your project.
- If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
- If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
- If the Database Deployments page is not already displayed, click Databases in the sidebar.
Navigate to the Online Archive tab for your cluster.
- Click the name of the cluster.
- Click the Online Archive tab to view the list of online archives, if any, for the cluster.
Click the ellipsis (...) in the Actions column for the online archive to display the list of allowed actions.
You can:
- Pause Archiving (only if state is Active)
- Edit Archive
- Delete Archive
- Resume Archiving (only if state is Paused)
Select Delete Archive from the ellipsis menu to display the Delete Online Archive confirmation window.
Enter the name of the online archive to delete and click Delete.
Delete an Online Archive Through the API
To delete an online archive through the API, send a DELETE
request to the onlineArchives endpoint with
the unique ID of the online archive to delete. To learn more about the
API syntax and options, see Delete an Online Archive.