RefreshingAWSCredentials with .NET

Where I am currently working we have Single Sign On for AWS API calls and need to use task accounts to connect and get temporary credentials. To that end, its not very easy to have long running processes making calls to AWS API’s such as S3 and SQS.

I am working a proof of concept which has a 3rd party .NET component which listens to SQS messages, calls into a proprietary API then dumps the results on S3. This code wasn’t written by people who knew about the hoops we need to authenticate.

To complicate things, the context the application runs under isn’t the context of the service account which has been granted rights to the appropriate role and the instance profile doesn’t have the correct rights. (For reasons I won’t go into, the Ops team won’t correct this).

So, having written that, I realise we’re looking at a very niche use case, but if it looks familiar, read on.

Solution

Behind the scenes, there is a PowerShell in the background running as a scheduled task to go through Single Sign On and get new tokens to write in the credential file. I won’t go into any more detail than that as its very company specific.

As the 3rd party application creates the client on startup it uses the latest credentials but they don’t get refreshed from the credential file. I found an abstract class in the .NET SDK called RefreshingAWSCredentials which looked promising.

With this class, you can set an expiration for the Credential object such that any AWS SDK client that is using it for the API calls - for example;

var s3Client = new AmazonS3Client(ExternalRefreshingAWSCredentilas.Credentials);

will create an S3Client that is given the refreshing credentials specified below.

using Amazon.Runtime;
using Amazon.Runtime.CredentialManagement;
using log4net;
using System;
using System.Configuration;
 
namespace AwsCredentialsExample.Credentials
{
    public class ExternalRefreshingAWSCredentials : RefreshingAWSCredentials
    {
        private static readonly object lockObj = new Object();
        private static readonly ILog Logger = LogManager.GetLogger(typeof(ExternalRefreshingAWSCredentials));
        private readonly string credentialFileProfile;
        private readonly string credentialFileLocation;
        private static ExternalRefreshingAWSCredentials refreshingCredentials;
        private CredentialsRefreshState credentialRefreshState;
        private int refreshMinutes = 45;

        public static ExternalRefreshingAWSCredentials Credentials {
            get {
                if (refreshingCredentials == null) {
                    lock (lockObj) {
                        if (refreshingCredentials == null) {
                            refreshingCredentials = new ExternalRefreshingAWSCredentials();
                        }
                    }
                }
                return refreshingCredentials;
            }
        }

        private ExternalRefreshingAWSCredentials()
        {
             credentialFileProfile = ConfigurationManager.AppSettings["CredentialFileProfile"];
             credentialFileLocation = ConfigurationManager.AppSettings["CredentialFileLocation"];
             if (ConfigurationManager.AppSettings.HasKey("ClientRefreshIntervalMinutes"))
             {
                 refreshMinutes = int.Parse(ConfigurationManager.AppSettings.Get("ClientRefreshIntervalMinutes"));
             }
             Logger.Info(string.Format("Credential file location is {0}", credentialFileLocation));
              Logger.Info(string.Format("Credential file profile is {0}", credentialFileProfile));
            credentialRefreshState = GenerateNewCredentials();
        }
 
        public override void ClearCredentials()
        {
            Logger.Info("Clearing the credentials");
            credentialRefreshState = null;
        }
 
        protected override CredentialsRefreshState GenerateNewCredentials()
        {
            Logger.Info(string.Format("Generating credentials, valid for {0} minutes", refreshMinutes));
            var credFile = new StoredProfileAWSCredentials(credentialFileProfile, credentialFileLocation);
            return new CredentialsRefreshState(credFile.GetCredentials(), DateTime.Now.AddMinutes(refreshMinutes));
        }
 
        public override ImmutableCredentials GetCredentials()
        {
            if (credentialRefreshState == null || credentialRefreshState.Expiration < DateTime.Now)
            {
                credentialRefreshState = GenerateNewCredentials();
            }
            return credentialRefreshState.Credentials;
        }
    }
}

Usage

There are three configurations to be used with this. These should be added as AppSettings in the app.config

  1. CredentialFileLocation - The location of the credential file that is being updated externally (in this case by PowerShell)
  2. CredentialFileProfile - The profile from the credential file to use
  3. ClientRefreshIntervalMinutes - How long to keep the credentials before expiring them (defaults to 45 minutes)

As suggested before, you can now create your AWS clients passing in the Credentials property in place of any of the normally used Credentials objects.

var s3Client = new AmazonS3Client(ExternalRefreshingAWSCredentials.Credentials);

Generating test data with Faker

Python is one of those languages where if you can concieve it, there is probably already a library for it.

One great library is Faker - this makes the generation of sensible test data much easier and removes a lot of the issues around having to using unrealistic junk values when creating it on your own.

There is lots to see, and your probably best off reading the docs, but this is to give you an overview.

Installation

Installation is simple, just use pip to install;

pip install faker

Usage

Now that you have it installed, you can use python REPL or ptpython to have a play.

### Some basics

from faker import Factory

fake = Factory.create()
fake.first_name(), fake.last_name(), fake.email()

This will give you a tuple with a random name and email;

('Mary', 'Bennett', 'jamesrodriguez@hotmail.com')

Localisation

If you want to get UK post codes, you can tell the factory a localisation to use when generating the data;

from faker import Factory

fake = Factory.create('en_GB')

fake.street_address(), fake.postcode()

which will yield;

('Studio 54\nCollins fork', 'L2 7XP')

Synchronising Multiple Fakes

Everytime you call a method on the fake object you get a new value. If you wanted to synchronise two fake objects you can use the seed. This will mean that the each consecutive call from each fake will return the same value.

This is probably more easily demonstrated in code;

from faker import Factory

fake1 = Factory.create()
fake2 = Factory.create()

fake1.seed(12345)
fake2.seed_instance(12345)

fake1.name(), fake2.name()
fake1.name(), fake2.name()

This will result in a tuple containing the same name across synchronised fake objects.

('Adam Bryan', 'Adam Bryan')
('Jacob Lee', 'Jacob Lee')

Making it a bit more interesting

In a previous pose I fiddled with credit card data where I created test data. Faker can be used to help out here. The code below isn’t an example of amazing Python, its just simple code to show it working.

First, we bring in the imports that are going to be used;

import csv
import random
from faker import Factory
from faker.providers import credit_card

Some helper methods, these are just to keep things clean

def get_transaction_amount():
    return round(random.randint(1, 1000) * random.random(), 2)

def get_transaction_date(fake):
    return fake.date_time_between(start_date='-30y', end_date='now').isoformat()

Some more helpers for the creation of records for our customer and transaction

def create_customer_record(customer_id):
    fake = Factory.create('en_GB')
    return [customer_id, fake.first_name(), fake.last_name(), fake.street_address().replace('\n', ', '), fake.city(), fake.postcode(), fake.email()]

def create_financials_record(customer_id):
    fake = Factory.create('en_GB')
    return [customer_id, fake.credit_card_number(), get_transaction_amount(), get_transaction_date(fake)]

A helper function to save the records to file

def flush_records(records, filename):
    with open(filename, 'a') as file:
        csv_writer=csv.writer(file, delimiter = ',', quotechar = '"', quoting = csv.QUOTE_MINIMAL)
        for record in records:
            csv_writer.writerow(record)
    records.clear()

Finally the main calling block to create the records

def create_customer_files(customer_count=100):
    customer_records = []
    financial_records = []
    for id in range(1, customer_count):
        customer_id = str(id).zfill(10)
        customer_records.append(create_customer_record(customer_id))
        financial_records.append(create_financials_record(customer_id))
        if len(customer_records) == 100:
            flush_records(customer_records, 'customer.csv')
            flush_records(financial_records, 'financials.csv')
    flush_records(customer_records, 'customer.csv')
    flush_records(financial_records, 'financials.csv')

create_customer_files()

Once we run this, we’ll have 2 files with customer details and a credit card transaction.

Customer records

0000000001,Clifford,Turner,"Flat 17, Smith crescent",Johnsonborough,DN5 7JJ,ucooper@gmail.com
0000000002,Amy,Clements,"Studio 96s, Anne harbor",Maureenfurt,LA53 8HZ,marshalllee@williams-hart.info
0000000003,Robin,Sinclair,5 Lesley motorway,Bryanbury,E2 9EU,sheilawhitehead@miles.com

Financial records

0000000001,4851450166907,179.28,2009-06-01T07:03:43
0000000002,370196022599953,229.46,2017-12-11T10:14:59
0000000003,4285121047016,10.61,1995-04-23T23:54:19

By sharing the customer ID across both files we have some semblence of referential integrity.

This code only creates a single transaction per customer, it can be easily modified to create multiple transactions by adjusting the create_financial_records to take an optional argument of transaction_count=1 and updating the append to handle an array of arrays ```


AWS EC2 Comparison

When working with EC2 instances, you really want to be right sizing the instance from the outset. With Amazon regularly bringing out new classes of instance it is hard to keep track of what is available now and what the characteristics are.

Some time ago I came across EC2Instances.info. This site regularly scrapes the instance information pages in the AWS documentation to gather the latest data about available instances and aggregate it all together.

EC2Instances screen shot

The functionality I find most useful is when I know I want for example 4GB or RAM, I can filter to get the list of all suitable, then sort them in ascending order and apply additional search criteria.

Next time you’re looking for an instance type, its worth a look.


Bing of the Day revisited

Back in 2014 I wrote an article about borrowing the bing wallpaper to use on my Macbook. Not long after that, I found that I was missing the odd “Bing of the Day” so I decided to run it in the cloud.

This post cover the steps of how that is achieved so you can “borrow” the Bing of the Day into S3.

Prerequisite

You will need an Amazon Web Services account. The Lambda function that runs this will fit within your free tier and it will take some time before the images dent your S3 free tier.

I’m not going to cover creating an account, but if you don’t have one then you can go to the AWS account creation page to get started.

Steps

Creating the S3 Bucket for your images

First thing you’re going to need is somewhere to store the images. I’m going to create a new bucket in S3 called owensbingoftheday. As per S3 requirements, you’re going to need to come up with your own unique bucket name.

In the AWS management console, go to the S3 service and create the new bucket, the create page should look like this;

Create the S3 Bucket for storing bing images

Creating the Lambda Role

The next thing you’re going to need is an IAM Role which the Lambda function can execute as. First we will create the locked down policy the role is going to use to access the S3 bucket.

Creating the policy

In the IAM service of the AWS Management console, create a new policy and switch to the JSON tab then add the following JSON. You need to change the bucket name in the Resource section to match the name of the bucket you specified in the previous step.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WriteBingImages",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::owensbingoftheday/*"
        }
    ]
}

The policy should look something like this; (you can ignore the error at the top of the screen shot)

Create a new policy to write to S3

Creating the role

Now that the policy has been created, we need the role that Lambda will execute as. This role will be created to trust Lambda. There is no need to explicity specify the Trust Policy for the role, we just select Lambda when creating the role.

Create a new role in the IAM service in AWS Management console. Ensure you select the service as Lambda to allow Lambda to assume the role.

Create the new role

Next, attach the policy we created earlier, you can filter to find it

Create the new role

Finally, save the policy with a unique name

Create the new role

At this point, all of the plumbing is done to create the new Lambda function. Go to the Lambda service in AWS Management console.

Creating Lambda Function

For this function we’re going to be starting a function from scratch. As the code to download the Bing of the day wallpaper is written in Python, we’ll be using that for the runtime.

Create a new function and the first page should look something like this;

Create the new role

Specify the new role that you’ve just created, it will allow the function to PUT our images into the specified bucket.

On the next screen, paste the code below into the code box and Save.

import boto3
import json
import os
import urllib3

from datetime import datetime

IMAGE_ARCHIVE_URL = 'http://www.bing.com/HPImageArchive.aspx?format=js&idx=0&n=1&mkt=en-US'
http = urllib3.PoolManager()

def get_image_details(payload):
    image = payload.get('images')[0]
    urlbase = image.get('urlbase')
    name = image.get('fullstartdate')
    return name, 'http://www.bing.com'+ urlbase + '_1920x1080.jpg'


def save_to_s3(name,  path):
    s3 = boto3.client('s3')
    year = datetime.now().year
    bucket_name = os.environ['BUCKET_NAME']
    object_name = '%s/%s.jpg' % (year, name)
    print ("uploading {}".format(object_name))
    s3.upload_file(path, bucket_name, object_name)


def get_image(payload):
    name, image_url = get_image_details(payload)
    path = '/tmp/%s.jpg' % name
    with open(path, 'wb') as f:
        pic = http.request('GET', image_url)
        f.write(pic.data)
    return name, path


def lambda_handler(event, context):
    req = http.request('GET', IMAGE_ARCHIVE_URL)
    if req.status != 200:
        print("Could not get the image archive data")
    else:
        payload = json.loads(req.data.decode('utf-8'))
        name, path = get_image(image_url, path)
        save_to_s3(name, path)
        

The lambda function can now be tested. We’ll need an event to send so using the drop down next to the Test button, create a new test event.

Fill out the details for an empty event so it looks like this;

Create the test event

Now, when you press the Test button, it will send the empty event to trigger the lambda function, all being well, you’ll find today’s image in the specified S3 bucket.

Scheduling the function

The intention was to move this from the laptop to run automatically within AWS. We now need to trigger this function to download once a day. To achieve this, we’re going to use a CloudWatch trigger with a schedule of 24 hours.

Creating the CloudWatch rule

Give the rule and name and save it. Our lambda function will now be triggered every 24hours to download the Bing of the day.

Downloading to your machine

The last thing to do is the routine download of the latest files. To do this I use the aws cli and the sync function. this will download anything not already on my machine;

aws s3 sync s3://owensbingoftheday/2019 ~/Pictures/bing-wallpapers/2019

New Year, New Intentions

Last year I managed 3 blog posts so this year my aim is to beat that by at the very least doubling it.

At the end of 2018 I completed my AWS DevOps Engineer - Professional certification, to that end I’m planning to blog a lot more about AWS and abstractions of what I’m working on day to day.