Using S3 with Authentication

When using Amazon S3 for remote storage, it might be required to secure access to the S3 buckets instead of using 'public' access. This means that the requests to the content in the bucket must be authenticated. See Using S3 storage regarding configuring Amazon S3 storage.

There are two ways to do this:

use the Authorization header

send a signature as a URL-encoded query-string parameter.

The mod_unified_s3_auth module supports both.

More information can be found in Amazon's Authenticating Requests documentation.

Using Webserver Directives for S3 authentication 

It is possible to use webserver directives and let USP sign the request automatically with the following webserver directives.

This covers one specific use case: both content (mp4/ismv) and server manifest (ism) are placed in the S3 bucket and the manifest references the content locally (no paths or URLs, just the filename).

The directives for Apache are the following:

Option	Description
S3SecretKey	The AWS secret key.
S3AccessKey	The AWS access key.
S3Region	Region of bucket where data is stored, added in 1.7.29. Required for AWS Signature v4.
S3SecurityToken	The AWS security token, if required; see Temporary security credentials in IAM. (Added in 1.11.13)
S3UseHeaders	Set to 'on' to use HTTP headers for AWS authentication, instead of query parameters.

The keys can be created in the AWS IAM portal and managed there as well (active/inactive, delete).

AWS Signature v4 

As of January 30, 2014, newer AWS regions such as eu-central-1 require the use of the Signature v4 for authentication. To enable the use of v4 signatures the name of the S3 region must be set when configuring access to the bucket. If the region is set, v4 signatures will be used whether or not the region requires it. Not setting the S3 region will result in USP using the older v2 signature method that may not be supported by all regions.

AWS Security Tokens 

AWS's Identity and Access Management (IAM) allows creating temporary, limited-privilege Security Tokens, which can be used to access AWS services such as S3. (See Temporary security credentials in IAM for more information.)

In addition to the existing S3-related directives, we have added a new S3SecurityToken directive in 1.11.13, where you can specify such a Security Token, if it is required to access the S3 resource.

Note

Please be aware S3 Security Tokens have a limited life or "Expiration" value, therefore a new token will need to be obtained before the active token expires.

Tokens can be obtained in several different ways. A session token can be generated by the aws-cli sts (security token service) api by assuming a role which has been configured with least privilege policy.

The following example demonstrate how this can be achieved.

$ aws sts assume-role \
--role-arn arn:aws:iam::12345678910:role/my-test-role \
--role-session-name session1 \
--duration-seconds 3600

{
    "Credentials": {
        "AccessKeyId": "12345678901",
        "SecretAccessKey": "v/12345678901",
        "SessionToken": "TEST92test48TEST+y6RpoTEST92test48TEST/8oWVAiBqTEsT5Ky7ty2tEStxC1T==",
        "Expiration": "2022-02-25T13:00:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "12345678901:my-test-role",
        "Arn": "arn:aws:sts::12345678901:assumed-role/my-role/my-test-role"
    }
}

An alternative method of obtaining credentials is by using the AWS Instance Metadata Service Version 2 (IMDSv2).This is a session-oriented method of obtaining credentials via HTTP request from an AWS EC2 instance.

The following example demonstrate how this can be achieved.

$ TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") && \
  curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/my-test-role

{
        "Code": "Success",
        "LastUpdated": "2022-02-25T12:00:00Z",
        "Type": "AWS-HMAC",
        "AccessKeyId": "12345678901",
        "SecretAccessKey": "v/12345678901",
        "Token": "TEST92test48TEST+y6RpoTEST92test48TEST/8oWVAiBqTEsT5Ky7ty2tEStxC1T==",
        "Expiration": "2022-02-25T13:00:00Z"
}

Note

Due to the way in which Unified Origin is configured via apache2/httpd virtual host configurations files any new credentials need to be added to these files and apache2/httpd restarted for these to take affect.

One way to ensure Unified Origin is always using valid credentials could be through the form of a script triggered using through scheduled automation such as a cronjob.

Below is an example shell script demonstrating how the the imdsv2 endpoint used to obtain credentials, update and restart apache.

#!/bin/bash

# Define current config location
APACHE_CUR_CONFIG=/etc/apache2/sites-enabled/unified-streaming.conf

# Define current credential variables
APACHE_S3_ACCESS_KEY=$(grep S3AccessKey "${APACHE_CUR_CONFIG}" | awk '{print $2}' )
APACHE_S3_SECRET_KEY=$(grep S3SecretKey "${APACHE_CUR_CONFIG}" | awk '{print $2}' )
APACHE_S3_SECURITY_TOKEN=$(grep S3SecurityToken "${APACHE_CUR_CONFIG}" | awk '{print $2}' )

# Define imdsv2 endpoint
REQUEST=$(curl -s -X GET -o /dev/null -w "%{http_code}" http://169.254.169.254/latest/meta-data/iam/security-credentials)

# Define if statement to check imdsv2 endpoint is available
# endpoint will only be available if a instance-profile has been assignd to # the ec2 via either the console or launch-template.
if [[ $REQUEST == "200" ]]; then

    # Define imdsv2 variable to obtain credentials
    TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
    ROLE=$(curl -s -X GET -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials)
    CREDS=$(curl -s -X GET -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/"$ROLE")

    # Define temp variables to populate with newly obtains credentials
    S3_ACCESS_KEY=$( jq -r  '.AccessKeyId' <<< "${CREDS}" )
    S3_SECRET_KEY=$( jq -r  '.SecretAccessKey' <<< "${CREDS}" )
    S3_SECURITY_TOKEN=$( jq -r '.Token' <<< "${CREDS}" )

    # Define if statement to compare active with new credentials
    # if these credentials differ (due to aws automatically rotating these)
    # then backup config, update & gracefully restart apache to avoid impacting active requests/connections.
    if [[ $APACHE_S3_ACCESS_KEY != $S3_ACCESS_KEY ]] || [[ $APACHE_S3_SECRET_KEY != $S3_SECRET_KEY ]] || [[ $APACHE_S3_SECURITY_TOKEN != $S3_SECURITY_TOKEN ]]; then

        ## Define message with date/time stamp which can be used for logging
        #echo "$(date) - New S3 Credentials available.. updating apache #config and restarting"

        cp ${APACHE_CUR_CONFIG} /tmp/unified-streaming.conf.old
        sed -i "s@${APACHE_S3_ACCESS_KEY}@${S3_ACCESS_KEY}@g; s@${APACHE_S3_SECRET_KEY}@${S3_SECRET_KEY}@g; s@${APACHE_S3_SECURITY_TOKEN}@${S3_SECURITY_TOKEN}@g" ${APACHE_CUR_CONFIG}
        /usr/sbin/apachectl -k graceful

    else
        echo "$(date) - Current credentials OK... exiting"
        exit 0
    fi

else
    exit 0
fi

Note

For more information on how to use S3 Security tokens with unified streaming cli-tools, please see Authenticate requests to AWS S3.

Warning

Please be reminded the majority of S3 buckets require requests to be authenticated using AWS Signature v4.

Apache Configuration 

To secure access to Amazon S3 buckets, <Proxy> sections are the appropriate locations to specify the S3 authentication directives.

The directives can appear in two places:

In a <Proxy> section. This is the recommended location.
In a <Location> section, which is used in combination with the IsmProxyPass directive in a <Directory> or <Location> section. This is the traditional way of specifying the S3 parameters, and it is still supported for backwards compatibility.

Each Amazon S3 bucket must have its own <Proxy> section. These are then used automatically for each request sent to the Amazon S3 bucket.

An additional benefit for using <Proxy> sections is that performance is improved by keeping the connections to Amazon S3 bucket alive.

For example, if the bucket is called mybucket, and it is hosted in Amazon's eu-central-1 region, the section becomes:

<Proxy "http://yourbucket.s3-eu-central-1.amazonaws.com/">
  S3SecretKey  YOUR_SECRET_KEY
  S3AccessKey  YOUR_ACCESS_KEY
  S3Region     YOUR_REGION
  S3UseHeaders on
  ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
</Proxy>

This can be used both when your manifests and/or media files are directly referring to Amazon S3 buckets, and when IsmProxyPass directives redirect requests to Amazon S3 buckets, see Cloud Storage for an outline of the different use cases.

In this case, there is no need to add any S3 parameters to the sections containing IsmProxyPass.

Schematically, it works like the following:

player --> cdn --> shield-cache --> origin (mod-smooth-streaming) --> signing (mod_unified_s3_auth) --> s3

The S3UseHeaders setting determines how the information is shared, either by adding a number of additional request headers, or otherwise by adding a number of additional query arguments.

Following is a complete example <VirtualHost>:

<VirtualHost *:80>
  ServerName www.example.com
  DocumentRoot /var/www/yoursite

  # Enable just in time packaging for VOD and using subrequests for I/O backend requests.
  <Location "/">
    UspHandleIsm on
    UspEnableSubreq on
    IsmProxyPass https://yourbucket.s3-eu-central-1.amazonaws.com/
  </Location>

  # If proxying to SSL hosts is desired, you must turn on SSLProxyEngine.
  SSLProxyEngine on

  <Proxy "https://yourbucket.s3-eu-central-1.amazonaws.com/">
    S3SecretKey YOUR_SECRET_KEY
    S3AccessKey YOUR_ACCESS_KEY
    S3Region YOUR_REGION
    S3UseHeaders on
    ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
  </Proxy>
</VirtualHost>

Legacy IsmProxyPass Example 

In previous versions of Unified Origin, when subrequests were not available, S3 authentication parameters were typically placed in a <Location> section, with a corresponding <Directory> section containing an IsmProxyPass directive. E.g.:

<Location /s3/>
  S3SecretKey YOUR_SECRET_KEY
  S3AccessKey YOUR_ACCESS_KEY
  S3Region YOUR_REGION
</Location>

<Directory /var/www/yoursite/s3/>
  IsmProxyPass https://yourbucket.s3-eu-west-1.amazonaws.com/
</Directory>

Assuming the DocumentRoot of the site is /var/www/mysite, the URI path /s3/example.ism would be mapped by IsmProxyPass to https://mybucket.s3-eu-west-1.amazonaws.com/example.ism, and the S3SecretKey, S3AccessKey and S3Region parameters from the <Location> section would be used for authentication.

Using S3 with Authentication

Using Webserver Directives for S3 authentication

AWS Signature v4

AWS Security Tokens

Apache Configuration

Legacy IsmProxyPass Example

Using Webserver Directives for S3 authentication 

AWS Signature v4 

AWS Security Tokens 

Apache Configuration 

Legacy IsmProxyPass Example 