The Import Upload collection includes resources that enable you to upload and import large files (greater than 5GB in size) in multiple parts.
The collection consists of four resources:
- The Initiate multipart upload resource initiates upload of database artifacts with the specified file extension.
- The Get pre-signed URL for multipart upload resource retrieves a pre-signed URL for uploading the database artifact to an S3 bucket.
- The Upload data part by number resource uploads the specified data as part of the upload job.
- The Complete multiple upload resource completes the upload of the database artifact to Data Bridge.
These resources may be used as part of a single workfow for uploading large files to Data Bridge.
Data Migration Recipe
Moody's RMS recommends that you use Import/Upload collection resources to upload large databases in all data integration projects. Although this workflow is more complicated than a single-stream upload workflow, multipart upload jobs provide for greater throughput and better recovery from network issues. For step-by-step instructions and sample code, see Import Large Databases.