Upload Mapping File

Map relations between uploaded account, location, and reinsurance data and EDM database fields

Suggest Edits

Overview

The MRI import process relies on an import mapping file to match columns of data in the uploaded source files to EDM database columns. The import mapping file enables the operation to ensure that structured data in the source file is imported into the correct destination field in the EDM.

The import mapping file also enables you to specify which properties will be imported into the EDM from the uploaded account, location, and reinsurance source files.

Step 5.1: Create import mapping file

An import mapping file defines the relationships between account, location and reinsurance data and between columns in the uploaded account, location, and reinsurance resource files and the corresponding columns in the EDM.

The MRI validator does not distinguish between the different language conventions, and may produce erroneous warning messages as a result. Warning messages do not impact the accuracy of the import job, which can still complete successfully.

The mapping file must be in the MIDI file format (MFF).

Step 5.2: Get S3 path and credentials

The Get S3 path and credentials enables you to fetch the path to an S3 bucket and temporary security credentials that will enable you to upload an flat file of exposure data.

The operation requires that you specify the bucketId of an Amazon S3 bucket as a path parameter. 

curl --location --request POST 'https://{host}/riskmodeler/v1/storage/{{bucketId}}/path' \
--header 'Content-Type: application/json' \
--header 'Authorization: {api_key}'

The request package identifies the fileInputType, fileName, fileSize, and fileType. 

{
	"fileInputType": "MRI",
	"fileType": "mapping",
	"fileSize": 9105,
	"fileName": "mapping.txt"
}

The fileType and fileName attributes are required.

  • The fileInputType attribute identifies the job type (ALM or MRI).
  • The fileName attribute specifies the name of the flat file to be uploaded.
  • The fileSize attribute specifies the size of the flat file in kilobytes.
  • The fileType attribute specifies the type of data contained in the flat file. One of account (account data), risk (location data), reins (reinsurance data), or mapping (mapping data).

If successful, the response returns a 201 status code and base64 encoded temporary security credentials from the AWS Security Token Service.

Step 5.3: Upload mapping file to S3

The RMS Risk Modeler API does not provide a custom operation for uploading flat file data to S3 buckets. You must use the AWS APIs to manage this process.

In this procedure, you will use the S3 bucket path and temporary user credentials to upload account data to the S3 bucket.

First, you must decode to the accessKeyId, secretAccessKey, sessionToken, and s3Path values and pass the decoded values to a S3 client. The sample code is in Java 8.

private static String base64Decode(String text) {
    return new String(Base64.getDecoder().decode(text));
}

Pass the decoded accessKeyId, secretAccessKey, and sessionToken to the Amazon getS3Client() method to create an Amazon S3 client.

private static AmazonS3 getS3Client(String accessKey, String secretKey, String sessionToken){
    BasicSessionCredentials sessionCredentials = new BasicSessionCredentials(
            accessKey,
            secretKey,
            sessionToken);
 
    return AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
            .withRegion(Regions.EU_WEST_1)
            .build();
}

Amazon TransferManager is a high-level utility for managing transfers to Amazon S3 that makes extensive use of Amazon S3 multipart uploads.

Once you have the Amazon S3 client, you can pass the the s3Client, bucketName, key, and the filePath to the TransferManager.

private static void upload(AmazonS3 s3Client, String bucketName, String key, String filePath) {
    try {
        TransferManager tm = TransferManagerBuilder.standard()
                .withS3Client(s3Client)
                .build();
 
        // TransferManager processes all transfers asynchronously,
        // so this call returns immediately.
        Upload upload = tm.upload(bucketName, key, new File(filePath));
        System.out.println("Object upload started");
 
        // Optionally, wait for the upload to finish before continuing.
        upload.waitForCompletion();
        System.out.println("Object upload complete");
    }catch( Exception ex){
        System.out.println(ex.getMessage());
    }
}

The parameters are derived from previous steps:

  • The bucketName can be extracted from the the initial section of the decoded s3Path. If the s3Path is rms-mi/preview/tenant/50000/import/mri/3929, the bucketName is rms-mi.
  • The key combines the remaining portion of the s3Path with the fileId, fileName in the following pattern: s3Path/fileId-fileName. For example, preview/tenant/50000/import/mri/3929/12373-fileName.
  • The filePath specifies the absolute path to flat file you want to upload.

Updated 6 months ago