Upload Account Data File

Securely import account data to AWS S3 buckets

12331233

Overview

Risk Modeler MRI import APIs enable you to securely connect to Amazon S3 so that you may upload flat files of account data using Amazon S3 API services.

The account flat file contains account exposure data in a structured format. Use the Get credentials and file URI by S3 bucket service get temporary credentials to an S3 bucket and the AWS SDK to upload a flat file of account data to an S3 storage bucket.

Once uploaded to AWS, account data may be imported into the EDM using an MRI job.

Step 1: Define account MRI file

The account file is a flat file that contains account data in a plain text format.

Column headers are defined in the first line of the data. Subsequent lines contain account exposure data. The flat file may use commas, semicolons, or tabs to delimit data values. The file may be saved in the CSV or TXT formats.

The MRI process requires that certain account exposure data fields are included in the flat file. Risk Intelligence™ validates the data specified in the flat file prior to importing that data into the EDM.If required fields are missing, a validation error is thrown. For a complete list of required fields, see "Import Validations" in the Risk Modeler 2.0 User Guide.

The account file is a flat file that contains account data in a plain text format. The account file specifies column headers in the first line of the data. Subsequent lines contain account exposure data. The flat file may use commas, semicolons, or tabs to delimit data values. The file may be saved in the CSV or TXT formats.

ACCNTNUM    ACCNTSTAT   ACCNTNAME   ACCNTSTATE  USERDEF1    USERDEF2    USERDEF3    USERDEF4    USERTXT1    USERTXT2    UWRITRNAME  BRANCHNAME  PRODID  PRODNAME    CEDANTID    CEDANTNAME  POLICYNUM   LOBNAME INCEPTDATE  EXPIREDATE  UNDCOVAMT   UNDCOVCUR   PARTOF  PARTOFCUR   POLICYTYPE  POLICYSTAT  POLICYSTRUCTURE MINDEDAMT   MINDEDCUR   MAXDEDAMT   MAXDEDCUR   BLANDEDAMT  BLANDEDCUR  BLANLIMAMT  BLANLIMCUR  DEDTYPE COVBASE LIMITGU BLANPREAMT  BLANPRECUR  BILIFELINE  COV1LIMIT   COV1LCUR    COV1DED COV1DCUR    COV1PREM    COV1PCUR    COV2LIMIT   COV2LCUR    COV2DED COV2DCUR    COV2PREM    COV2PCUR    COV3LIMIT   COV3LCUR    COV3DED COV3DCUR    COV3PREM    COV3PCUR    COMBINEDLIM COMBINEDLCUR    COMBINEDDED COMBINEDDCUR    COMBINEDPREM    COMBINEDPCUR    COND1NAME   COND1TYPE   COND1LIMIT  COND1DEDUCTIBLE COND1DEDUCTIBLETYPE PARENTCOND1NAME COND2NAME   COND2TYPE   COND2LIMIT  COND2DEDUCTIBLE COND2DEDUCTIBLETYPE PARENTCOND2NAME COND3NAME   COND3TYPE   COND3LIMIT  COND3DEDUCTIBLE COND3DEDUCTIBLETYPE PARENTCOND3NAME COND4NAME   COND4TYPE   COND4LIMIT  COND4DEDUCTIBLE COND4DEDUCTIBLETYPE PARENTCOND4NAME COND5NAME   COND5TYPE   COND5LIMIT  COND5DEDUCTIBLE COND5DEDUCTIBLETYPE PARENTCOND5NAME POLICYUSERTXT1  POLICYUSERTXT2  POLICYUSERTXT3  POLICYUSERTXT4
AD     Andorra     AD    Andorra     0                  Property
AR     Argentina   AR    Argentina   0                  Property
AT     Austria     AT    Austria     0                  Property
AU     Australia   AU    Australia   0                  Property
BE     Belgium     BE    Belgium     0                  Property
BG     Bulgaria    BG    Bulgaria    0                  Property
BO     Bolivia     BO    Bolivia     0                  Property
...

The MRI process requires that certain account exposure data fields are included in the flat file. Risk Intelligence™ validates the data specified in the flat file prior to importing that data into the EDM.If required fields are missing, a validation error is thrown. For a complete list of required fields, see "Import Validations" in the Risk Modeler 2.0 User Guide.

Step 2: Get security credentials and S3 bucket path

The Get S3 path and credentials service enables you to fetch the path to an S3 bucket and temporary security credentials that will enable you to upload an flat file of exposure data.

The service takes two required parameters: a bucketId that specified in the endpoint path and a fileName, the name of an flat file that contains exposure data. All other parameters are specified in the request body.

In this example, you will use bucketId values returned by the POST storage service to upload account data. The bucketId is specified in the endpoint path.

curl --location --request POST 'https://{host}/riskmodeler/v1/storage/{{bucketId}}/path' \
--header 'Content-Type: application/json' \
--header Authorization: {api_key} \
--data-raw '{
    "fileName": "accexp.txt",
    "fileSize": 9105,
    "fileType": "account"
}'

The request package identifies the fileName , fileSize , and fileType. Only the fileName attribute is required.

  • The fileName attribute specifies the name of the flat file to be uploaded.
  • The fileSize attribute specifies the size of the flat file in kilobytes.
  • The fileType attribute specifies the type of data contained in the flat file. One of account (account data), risk (location data), reins (reinsurance data), or mapping (mapping data).

If successful, the response returns a 201 status code and base64 encoded temporary security credentials from the AWS Security Token Service.

"accessKeyId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"secretAccessKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"sessionToken": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"s3Path": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

The temporary security credentials enable you to programmatically sign AWS requests. Signing helps to secure requests by verifying the identity of the requester and protecting the the data in transit.

  • The accessKeyId attribute specifies a base64 encoded S3 access key ID, a unique identifier for your S3 access key.
  • The secretAccessKey attribute specifies a base64 encoded S3 secret access key. The access key ID and secret access key enable you to sign AWS requests.
  • The s3Path attribute specifies a base64 encoded path to the S3 bucket. For example, https://{{bucketname}}/tenant/50000/import/mri/3929
  • The sessionToken attribute specifies a base64 encoded S3 session token.

The Location HTTP response header specifies the fileId at the end of the header value. For example, location: http://<host>/mi/api/v1/storage/3929/12373. In this example, the fileId is "12373". You will use this fileId when you upload the flat file to the S3 bucket.

Step 3: Upload account file to S3

The RMS Risk Modeler API does not provide a custom service for uploading flat file data to S3 buckets. You must use the AWS APIs to manage this process.

In this procedure, you will use the S3 bucket path and temporary user credentials to upload account data to the S3 bucket.

First, you must decode to the accessKeyId, secretAccessKey, sessionToken, and s3Path values and pass the decoded values to a S3 client. The sample code is in Java 8.

private static String base64Decode(String text) {
    return new String(Base64.getDecoder().decode(text));
}

Pass the decoded accessKeyId, secretAccessKey, and sessionToken to the Amazon getS3Client() method to create an Amazon S3 client.

private static AmazonS3 getS3Client(String accessKey, String secretKey, String sessionToken){
    BasicSessionCredentials sessionCredentials = new BasicSessionCredentials(
            accessKey,
            secretKey,
            sessionToken);
 
    return AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
            .withRegion(Regions.EU_WEST_1)
            .build();
}

Amazon TransferManager is a high-level utility for managing transfers to Amazon S3 that makes extensive use of Amazon S3 multipart uploads.

Once you have the Amazon S3 client, you can pass the the s3Client, bucketName, key, and the filePath to the TransferManager.

private static void upload(AmazonS3 s3Client, String bucketName, String key, String filePath) {
    try {
        TransferManager tm = TransferManagerBuilder.standard()
                .withS3Client(s3Client)
                .build();
 
        // TransferManager processes all transfers asynchronously,
        // so this call returns immediately.
        Upload upload = tm.upload(bucketName, key, new File(filePath));
        System.out.println("Object upload started");
 
        // Optionally, wait for the upload to finish before continuing.
        upload.waitForCompletion();
        System.out.println("Object upload complete");
    }catch( Exception ex){
        System.out.println(ex.getMessage());
    }
}

The parameters are derived from previous steps:

  • The bucketName can be extracted from the the initial section of the decoded s3Path. If the s3Path is rms-mi/preview/tenant/50000/import/mri/3929, the bucketName is "rms-mi".
  • The key combines the remaining portion of the s3Path with the fileId, fileName in the following pattern: s3Path/fileId-fileName. For example, "preview/tenant/50000/import/mri/3929/12373-fileName".
  • The filePath specifies the absolute path to flat file you want to upload.

Did this page help you?