Creating an AWS S3 Presigned URL

AWS allows for the creating of pre-signed URLs for their S3 object storage. These pre-signed URLs can be given to someone and that person is allowed to access to file for a specific time period. This article covers two scripts that can create the pre-signed URL. The first used only modifies the data as needed and the second uses AWS S3 Java libraries.

The first script will be more general and will allow for you to run it on any runtime architecture (atom, molecule, and cloud). This script uses methods that are allowed to be executed within the Boomi Public Cloud without triggering any errors. The latter script is based off of AWS’s libraries, require custom libraries to be loaded into Boomi, and can only run on runtimes with Java Security set to low (e.g. atom and molecules).

AWS has Java libraries that support the creation of a pre-signed URL. The AWS Java libraries assumes that the security key is located in a few possible places and it checks each of those locations. The first location is the environment variables for the operating system. The AWS library will be unable to run on the Boomi Public Cloud because of the Java Security Policy, which is set to HIGH. The script can run on an atom or molecule, which default the Java Security Policy to low since it’s not a multi-tenant.

To set up either script, you will need to add a Set Properties shape with the following parameters.

Dynamic Document Properties: 
    DDP_region: Region that the bucket is in (e.g. us-east-1) 
    DDP_bucket: Name of the bucket 
    DDP_filekey: Name of the file and any folders (e.g. folder1/subfolder1/file.txt) 
Dynamic Process Properties: 
    DPP_s3accesskey: User access key 
    DPP_s3secretkey: User secret key

Dynamic Document Properties: DDP_region: Region that the bucket is in (e.g. us-east-1) DDP_bucket: Name of the bucket DDP_filekey: Name of the file and any folders (e.g. folder1/subfolder1/file.txt) Dynamic Process Properties: DPP_s3accesskey: User access key DPP_s3secretkey: User secret key

Figure 1. This example shows how to setup the Set Property shape and the Data Process shape. The message shape is used to extract DDP_presignedurl, which is the output of the script.

Figure 2. Set Properties shape with all parameters set.

AWS Presigned URL Script For All Runtimes

The first version of the AWS Presigned URL script is without the AWS S3 libraries and is able to be executed on any Boomi runtime (Atom, Molecule, Private or Public Cloud). To use the script, add a Set Properties shape with the properties above. Then add a Data Process shape and select Custom Script and Groovy 2.4. Everything within the text editor can be replaced with the code below. The link will be outputted to a dynamic document property called “DDP_presignedurl” and the incoming data left unchanged.

The expiration variable is used to set how long the URL is valid. The number is in seconds and the maximum value is 7 days. The script has it set to default to 1 days, but updated as required.

// Groovy 2.4
// Doc: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

import java.util.Properties;
import java.io.InputStream;
import com.boomi.execution.ExecutionUtil;
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.nio.charset.StandardCharsets
import java.security.MessageDigest
import java.text.DateFormat
import java.text.SimpleDateFormat

// Set global variables
String accessKey = ExecutionUtil.getDynamicProcessProperty("DPP_s3accesskey");
String secretKey = ExecutionUtil.getDynamicProcessProperty("DPP_s3secretkey");
String sep = "%2F"; // Used to encode the '/'
String serviceName = "s3";
long expiration = 86400;  // Set expiration time in seconds. Max is 7 days.


for (int i = 0; i < dataContext.getDataCount(); i++) {
    InputStream is = dataContext.getStream(i);
    Properties props = dataContext.getProperties(i);

    String regionName = props.getProperty("document.dynamic.userdefined.DDP_region");
    String bucketName = props.getProperty("document.dynamic.userdefined.DDP_bucket");
    String keyName = props.getProperty("document.dynamic.userdefined.DDP_filekey");

    // Get Current Date
    DateFormat shortDateFormat = new SimpleDateFormat("yyyyMMdd");
    DateFormat longDateFormat = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'")
    Date date = new Date()
    String shortDate = shortDateFormat.format(date);
    String longDate = longDateFormat.format(date);

    String baseUrl = "https://"" + bucketName + ".s3.amazonaws.com/"";

    // Set Query Parameters
    String queryParameters = "X-Amz-Algorithm=AWS4-HMAC-SHA256" +
            "&X-Amz-Credential=" + accessKey + sep + shortDate + sep + regionName + sep +
            serviceName + sep + "aws4_request" +
            "&X-Amz-Date=" + longDate +
            "&X-Amz-Expires=" + expiration +
            "&X-Amz-SignedHeaders=host";

    // Set Canonical Request - keyName and queryParameters might need to be URL encoded
    String canonicalRequest = "GET" + "\n" +
            "/" + keyName + "\n" +
            queryParameters + "\n" +
            "host:" + bucketName + ".s3.amazonaws.com" + "\n\n" +
            "host" + "\n" +
            "UNSIGNED-PAYLOAD";

    // Set String to sign
    MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
    String stringToSign = "AWS4-HMAC-SHA256" + "\n" +
            longDate + "\n" +
            shortDate + "/" + regionName + "/" + serviceName + "/aws4_request" + "\n" +
            base16encode(messageDigest.digest(canonicalRequest.getBytes("UTF-8")))

    // Set Signature
    byte[] dateKey = getHMAC256(("AWS4" + secretKey).getBytes(StandardCharsets.UTF_8), shortDate);
    byte[] dateRegionKey = getHMAC256(dateKey, regionName);
    byte[] dateRegionServiceKey = getHMAC256(dateRegionKey, serviceName);
    byte[] signingKey = getHMAC256(dateRegionServiceKey, "aws4_request")
    String signature = base16encode(getHMAC256(signingKey, stringToSign));

    // Create Authorized URL
    String url = baseUrl + keyName + "?" + queryParameters + "&X-Amz-Signature=" + signature;

    props.setProperty("document.dynamic.userdefined.DDP_presignedurl", url.toString());
    dataContext.storeStream(is, props);
}


private static byte[] getHMAC256(byte[] key, String input) {
    byte[] hash;
    Mac sha256HMAC = Mac.getInstance("HmacSHA256");
    SecretKeySpec secretKey = new SecretKeySpec(key, "HmacSHA256");
    sha256HMAC.init(secretKey);
    hash = sha256HMAC.doFinal(input.getBytes(StandardCharsets.UTF_8));
    return hash;
}

private static String base16encode(byte[] byteArray) {
    char[] HEX = [
            '0', '1', '2', '3', '4', '5', '6', '7',
            '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'] as char[];
    StringBuffer hexBuffer = new StringBuffer(byteArray.length * 2);
    for (int i = 0; i < byteArray.length; i++)
        for (int j = 1; j >= 0; j--)
            hexBuffer.append(HEX[(byteArray[i] >> (j * 4)) & 0xF]);
    return hexBuffer.toString();
}

AWS Presigned URL Script With AWS S3 Libraries

The second script will create the same presigned URL except it uses the AWS S3 libraries. This script must be ran on a runtime that does not have Java Security Policy set to HIGH, which means it can not be executed on the Boomi Public Cloud. The AWS S3 libraries tries to read the OS environment variables, which when attempted results in an error within environments where Java Security Policy is set to HIGH.

There are 4 libraries that need to be downloaded and loaded into your account. Here are the general steps on how to add a jar file to your account. Download the files below from maven. Click on the links provided and then click on “jar” to download.

Figure 3. Download jar file from AWS SDK Repo.

Download the Following Jar Files:

After you have download and loaded the jar files into your account, you are able to being to setup the integration process to run the Groovy script. The setup is similar to the script above in which you need to add a set property shape with the desired dynamic document properties and dynamic process properties. Then add the script below to a Data Process shape and set the language to Groovy 2.4.

The expiration variable is used to set how long the URL is valid. The number is in seconds and the maximum value is 7 days. The script has it set to default to 1 days, but updated as required.

// Groovy 2.4

/*
Required jars:
aws-java-sdk-core-1.12.153.jar
https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.12.153
aws-java-sdk-s3-1.12.153.jar
https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3/1.12.153
httpclient-4.5.11.jar
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient/4.5.1
httpcore-4.4.13.jar
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore/4.4.13
joda-time-2.10.5.jar
https://mvnrepository.com/artifact/joda-time/joda-time/2.10.5
 */


import java.util.Properties;
import java.io.InputStream;
import com.boomi.execution.ExecutionUtil;
import com.amazonaws.HttpMethod;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;


for (int i = 0; i < dataContext.getDataCount(); i++) {
    InputStream is = dataContext.getStream(i);
    Properties props = dataContext.getProperties(i);

    Regions clientRegion = Regions.(props.getProperty("document.dynamic.userdefined.DDP_region"));
    String bucketName = props.getProperty("document.dynamic.userdefined.DDP_bucket");
    String objectKey = props.getProperty("document.dynamic.userdefined.DDP_filekey");
    String accessKey = ExecutionUtil.getDynamicProcessProperty("DPP_s3accesskey");
    String secretKey = ExecutionUtil.getDynamicProcessProperty("DPP_s3secretkey");

    BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKey, secretKey);

    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withRegion(clientRegion)
            .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
            .build();

    Date expiration = new Date();
    long expTimeMillis = expiration.getTime();
    expTimeMillis += 1000 * 60 * 60; // 1 hour
    expiration.setTime(expTimeMillis);

    // Generate the presigned URL.

    GeneratePresignedUrlRequest generatePresignedUrlRequest =
            new GeneratePresignedUrlRequest(bucketName, objectKey)
                    .withMethod(HttpMethod.GET)
                    .withExpiration(expiration);

    URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

    props.setProperty("document.dynamic.userdefined.DDP_presignedurl", url.toString());
    dataContext.storeStream(is, props);
}

Article originally posted at Boomi Community.

Published Feb 24, 2022