Previous Posts

  • 30 Apr 2019 : CIDR Cheatsheet for available IP Address

    This is less of a post and more of a CIDR to mask cheatsheet to remind me how many IPs I get for a given block. CIDR Subnet Total IPs Usable IPs /32 255.255.255.255 1 1 /31 255.255.255.254 2 0 /30 255.255.255.252 4 2 /29 255.255.255.248 8 6 /28 255.255.255.240...
  • 03 Apr 2019 : git alias for concise history

    This is pretty short post. H/T to @prokopp for telling me know about this. Git allows you to add aliases in your global config - this is the first one I’ve actually added and all it does is a concise, clearly formatted git log. To add it to you git...
  • 27 Feb 2019 : How it works - www.aws-glossary.info

    I recently posted an entry announcing www.aws-glossary.info. In this post I want to cover how it actually works. Basic Architecture The logical diagram below details the basic structure of the application. The core components are; Component Purpose Github Static content stored in github CodePipeline On commit to the master branch...
  • 21 Feb 2019 : Dynamic AWS Glossary

    Yesterday I created a scraper for the AWS documentation landing page to give an aws glossary of all of the services that have available documentation. Originally I posted it as a page in this site, I’ve now moved it to https://www.aws-glossary.info
  • 16 Feb 2019 : Delving in the depths of my github

    This morning I was trying to solve an issue where http://owenrumney.co.uk has an SSL error when trying to access my blog but https://www.owenrumney.co.uk is fine! (its still not resolved). As I was poking around in my github account I looked at what I had done over the years… taking me...
  • 28 Jan 2019 : Using Amazon SES with HP Scan-to-Email Printer

    Overview We have just replaced our aging Canon MX885 Multi Function printer with a new colour Laser printer, an HP M281fdw multi function. One of the scanning options is to “Scan to Email”. This is something I’ve found really useful in my current clients office. What’s the issue then? The...
  • 23 Jan 2019 : Creating a Kerberos Keytab file with ktutil

    NOTE: Creating a keytab file is easy enough but I have to refresh myself each time so I thought I would document it in a blog post. Assumptions I’m assuming for anyone who is doing this that you have your /etc/krb5.conf in order and that isn’t going to get in...
  • 19 Jan 2019 : Creating a simple Dockerised Flask App

    This post covers the steps to create a simple dockerised flask app to cover some of the basic steps required when creating a REST(ish) service that can be run as a Docker container. The App Rather than go with the obvious “Hello, World!” type example, I decided I’d try and...
  • 16 Jan 2019 : Adding retry logic to urllib3 Python code

    In this post I’m going to cover the basics of implementing retry logic using urllib3. There is probably a solid argument saying “why aren’t you just using requests?”, as it happens, requests uses urllib3 and it’s Retry functionality. For the purposes of this post, lets imagine that we have a...
  • 15 Jan 2019 : Refreshing AWS credentials in Python

    In a recent post I covered an using RefreshingAWSCredentials within .NET AWS SDK to solve an issue with the way my current organisation has configured SingleSignOn (SSO) and temporary credentials. Essentially, the solution involves a background process updating a credenial file then using a time limited AWSCredential object to refresh...
  • 09 Jan 2019 : RefreshingAWSCredentials with .NET

    Where I am currently working we have Single Sign On for AWS API calls and need to use task accounts to connect and get temporary credentials. To that end, its not very easy to have long running processes making calls to AWS API’s such as S3 and SQS. I am...
  • 07 Jan 2019 : Generating test data with Faker

    Python is one of those languages where if you can concieve it, there is probably already a library for it. One great library is Faker - this makes the generation of sensible test data much easier and removes a lot of the issues around having to using unrealistic junk values...
  • 06 Jan 2019 : AWS EC2 Comparison

    When working with EC2 instances, you really want to be right sizing the instance from the outset. With Amazon regularly bringing out new classes of instance it is hard to keep track of what is available now and what the characteristics are. Some time ago I came across EC2Instances.info. This...
  • 04 Jan 2019 : Bing of the Day revisited

    Back in 2014 I wrote an article about borrowing the bing wallpaper to use on my Macbook. Not long after that, I found that I was missing the odd “Bing of the Day” so I decided to run it in the cloud. This post cover the steps of how that...
  • 02 Jan 2019 : New Year, New Intentions

    Last year I managed 3 blog posts so this year my aim is to beat that by at the very least doubling it. At the end of 2018 I completed my AWS DevOps Engineer - Professional certification, to that end I’m planning to blog a lot more about AWS and...
  • 02 Aug 2018 : Introducing gitsearch-cli

    The first version of gitsearch-cli is now available. This command line interface allows you to search github repositories and users using keywords and (currently) a handful of additional criteria. Installation To install git search you can use pip3 with the following command; pip3 install gitsearch-cli Usage By default the search...
  • 25 May 2018 : Allow connection to dockerised elasticsearch other than localhost

    We need to access ElasticSearch in a namespace within minikube and the other Pods can’t connect to 9200. It turns out that from the box its limited to localhost and the network.host property needs updating. Setting network.host in the elasticsearch.yml configuration file on a docker container will put the instance...
  • 27 Jan 2018 : Argument defaults in shell scripts

    Regularly when writing a shell script I find that I want to be able to pass an argument into the script but only sometimes. For example if I want the script to output to /tmp folder for the most part but I’d like the opportunity to select the output myself....
  • 10 Nov 2017 : Running Spark against HBase

    Its reasonably easy to run a Spark job against HBase using the newAPIHadoopRDD available on the SparkContext. The general steps are, create an HBaseConfiguration create a SparkContext create a newAPIHadoopRDD perform job action To get this working, you’re going to need the HBase libraries in your build.sbt file. I’m using...
  • 29 Aug 2017 : Replacing an incorrect git commit message

    If you have committed some code to git (or in the current case, BitBucket) and you have made an error in the commit message (in the current case, referenced the wrong Jira ticket), all is not lost. To replace the commit message perform the following actions. git commit -amend Change...
  • 17 Jul 2017 : TFL Cycling DataSet - Part 2

    Following on from part 1 of this mini series. I’ve got my local environment ready to go and I have pulled down some test data to work with. The next step is to start having a look at some of the data. Loading in the data We know that the...
  • 16 Jul 2017 : TFL Cycling DataSet - Part 1

    I’m hoping this will be a reasonably accurate account of my play with the TfL Cycling DataSets. I’m still forming my plan, however loosely I think I want to end up with a visualisation where the bike points are highlighted in over a time series as bikes are taken and...
  • 11 Apr 2017 : CSV to Markdown table - Sublime Package

    Now I’m writing almost all documentation in markdown then using Pandoc to convert it to Mediawiki or docx as required, I needed to finds an easier way to quickly create my tables. It doesn’t do anything fancy, but I created a sublime package to do the conversion of a csv...
  • 05 Apr 2017 : Go routines and channels

    I’m having a bit of a dabble with Go, as a by product of working with Elastic Search beats. One thing I’ve been looking at today is the channels to allow two go routines to communicate with each other and I came up with a fairly cheesy way to play...
  • 17 Mar 2017 : Monit

    There are lots of monitoring and alerting tools out there and I’m sure everyone has there own preference on which they’re going to use. We have selected monit for simple monitoring of disk space, tunnels and processes because its simple to setup and does exactly what we’re asking it to...
  • 10 Mar 2017 : Using AWS Simple Email Service with Oozie in Cloudera

    I’ve moved to a new role where I will be doing a lot more “devops” type work which hopefully comes with more interesting subjects to start writing about. Oozie and AWS Simple Email Services The Cloudera cluster I have started working on is hosted in AWS and has all the...
  • 14 Feb 2017 : Wordle Interpretation of CV

    Today someone showed me a handle word cloud tool called wordle. I put the skills from my CV into it and had a play with the output… I think it looks quite cool.
  • 03 Jan 2016 : First go at creating a Garmin watch face

    For a while Garmin have been adding “wearable technology” functionality to their watches - notifications from the phone etc. I got a Garmin Fenix 3 for Christmas and had a play with the vast number of watch faces and widgets that were available. The recent updates have really added some...
  • 10 Dec 2015 : Scala eXchange 2015 - Embracing the community

    Today I’m at my first Scala eXchange conference - the 5th annual one to be precise. The day started with the keynote session by @jessitron which was both inspiring and enlightening. The general tone being that code should be clear and useful so that others can learn. Drawing on examples...
  • 02 Jul 2015 : Unit testing HDFS code

    I need to write a couple of unit tests for some code to add a log entry into HDFS but I don’t want to have to rely on having access to full blown HDFS cluster or a local install to achieve this. The MiniDFSCluster in org.apache.hadoop:hadoop-hdfs can be used to...
  • 17 Jun 2015 : Writing a Flume Interceptor

    He we are in June, some five months since the last post and I finally have some time and content to sit and write a post. In April 2013 I started working with Hadoop, the plan was to suck in server application logs to determine who was using what data...
  • 13 Jan 2015 : Quick introduction to pyspark

    All the work I have been doing with AWS has been using Python, specifically boto3 the rework of boto. One of the intentions is to limit bandwidth when transferring data to S3 the idea is to send periodic snapshots then daily deltas to merge and form a latest folder so...
  • 06 Jan 2015 : Client side encryption using Boto3 and AWS KMS

    Towards the end of 2014 Amazon released the KMS service to provide a cheaper cut down offering for Key Management Services than those provided with the CloudHSM solutions (although it still uses hardware HSM underneath). KMS service can be accessed through IAM service at the bottom option on the left...
  • 03 Dec 2014 : Adventures with Spark, part two

    Some time ago, back in September, I wrote a post on starting my adventures with Spark but didn’t progress things very far. On thing that was holding me back was a reasonably real world problem to use as a learning case. I recently came across a question which seemed like...
  • 01 Dec 2014 : AWS HTTPSConnectionPool max retries exceeded

    I’m working with a new AWS account and I am moving to testing Boto3 to use the KMS service. I needed to make sure that the AWS account and secret keys were updated so ran aws configure to quickly update them. I added the new keys and saw that default...
  • 14 Oct 2014 : Git hangs while unpacking objects (Windows)

    I’m not sure if this is because we’re behind a proxy, the network has issues or my work laptop isn’t great, but for some reason the git clones very often hang during the unpacking of objects. remote: Counting objects: 21, done. remote: Total 21 (delta 0), reused 0 (delta 0)...
  • 14 Sep 2014 : Adventures with Spark, part one

    For 18 months I’ve been working with Hadoop. Initially it was Hortonworks HDP on Windows then Hortonworks HDP on CentOS and for production we settled on Cloudera CDH5 on Red Hat. Recently we’ve been introduced to Spark and subsequently Scala which I am now in the process of skilling up...
  • 13 Sep 2014 : Update Wallpaper from Bing (OSX)

    I’m not a huge fan of Bing search engine, I’ve tried to use it but I don’t like the format of the search results and I don’t think it’s particularly good at finding relevant results either. I do like Bing wallpapers, and I use Bing Desktop on my Windows laptop...
  • 13 Aug 2014 : Creating environment variables from the command line (Windows)

    I know that it is incredibly lazy and a non problem but I find it quite tedious in Windows 8 to go digging for the system environment variable GUI whenever I need to add or update something. Generally I’m already in the command prompt so I was keen to find...
  • 11 Jul 2014 : SFTP Connection Closed - password expired?

    I’ve just had an interesting problem with an SFTP account that suddenly stopped working from a cron job. When the account was used directly from the bash prompt the response was simply Connection closed immediately. Every thing was set up correctly as far as authorized_keys and /etc/ssh/sshd_config looked fine but...
  • 03 Jul 2014 : Cross Join Columns in Excel

    As part of a project I’m working on I need to create a cross join of users against symbols; the idea being that any of the given users might look at any of the given symbols. I have a list of users and a list of symbols which I essentially...
  • 25 Jun 2014 : Using yum history

    I have been setting up a couple of Red Hat boxes and I need to have a cron task to mount a network share from an AD domain, copy the files then dismount. In the past I have successfully accessed the network shares using a given set of credentials but...
  • 06 Apr 2014 : Office365 and SharePoint iPad Friendly

    I have been doing some work for a company who have a user base who primarily use iPads to access the SharePoint site. There was a requirement to allow those users to easily access the team site without getting their screens filled with the quick launch and the bloated title....
  • 18 Mar 2014 : TMA470 first assignment returned

    I got the result and feedback for the first assignment of the final Open University project today. Its been a long week waiting for it, not least because a significant number of others on the course have been reporting back very low marks. The difference with TM470 assignments is that...
  • 04 Mar 2014 : OU Final project finally started

    At the eleventh hour I finally settled on the project that I am going to be working on for the next 8 months. I had previously intended to work on a project that would allow me to play with some javascript graph layout and visualisation libraries. Unfortunately this intention didn’t...
  • 14 Feb 2014 : First attempt at a Force Directed Graph

    For the final project of my open university degree I am looking at creating an application to bring together a proprietary dataset of information about available workers and cross it with social media data. The project is in the early stages (very early) but I started playing with the graphing...
  • 06 Feb 2014 : DIY Plyo box

    Since November 2013 I’ve been working out in my garage with a cross trainer and timed interval training. I did have a membership to Virgin Active, but between my commute, Open University degree and family I just don’t have time to get there. Having the garage I can go straight...
  • 31 Jan 2014 : rhn_register crashes while starting

    I’m setting up a tactical farm of Linux servers on some desktops till we can get the permanent kit installed in the data center. The new servers are running on trader desktops so they’re reasonably good kit. Too satisfy security requirements we need to use a customised build of Red...
  • 16 Jan 2014 : SharePoint 2010/2013 Developer Dashboard

    The SharePoint developer dashboard, when switched on, will render at the bottom of the page in SharePoint 2010 and appear as an icon in the top right hand corner of SharePoint 2013. If the DisplayLevel is set to OnDemand for SharePoint 2010 then the behaviour will be as SharePoint 2013,...
  • 13 Jan 2014 : Partitioning a large disk in RedHat

    For the project I am currently working on, we have an order for a Hadoop Appliance from our supplier to be placed in the Mega Data Centre for processing all of our log files. The order is in, but with all the red tape its going to be a while...
  • 12 Jan 2014 : Deleting SQL database from the command line

    For one reason or another, my SharePoint 2013 development environment became corrupt and the quickest way to get it up and running again was to start a fresh with SharePoint. The problem I faced was when running the Products and Technologies tool to configure the instance with the existing sites...
  • 07 Jan 2014 : Moving to git pages

    Not too long ago I moved my website to a static page structure using Sandra.Snow. At the time, I was aware that it was based on the idea of Jekyll with a Microsoft lilt using Nancy to generate the static files. As a long time Microsoft user it made sense...
  • 02 Jan 2014 : Windows to Linux transition

    I sat with my kids while they watched Justin Time (they love it) on New Years day and I started to clear up the disk in my Dell XPS 13. Its a 256GB SSD and while its not tiny it does need a certain about of housekeeping to keep on...
  • 20 Dec 2013 : Refresh Linux mounts after changes made to /etc/fstab

    When an update has been made to /etc/fstab the changes won’t immediately and automatically take effect so there is a need to refresh them manually. This can be done either by rebooting, which is a pain or just use the mount command mount -a
  • 12 Nov 2013 : WPF Commands and Events with MVVM

    I have been doing some revision on my WPF, particularly around ModelView-View-Model pattern. One thing that I’ve always found a bit difficult to get right is the handling of events on the window and getting them back to the view model. I personally believe that the goal of a completely...
  • 15 Oct 2013 : Focus on content on page load

    One of the requirements for a piece of SharePoint/Office 365 work I’m working on at the moment is for easy finger usage when accessing with an iPad. The user base coming in on iPad are performing a specific task so a lot of the navigation and quick launch is not...
  • 06 Oct 2013 : SharePoint and Office 365

    Since starting to use Azure more and more I’ve found myself revisiting suddenly enthused by the web and cloud. I’ve had the obligatory Amazon free tier but done little with it. (I’ve recently talked with Amazon about a project at work where I learned quite a lot about glacier and...
  • 29 Sep 2013 : Creating pet projects and the Constant Reminder

    Now I’ve got into the Azure platform I’ve decided to start making the most of the MSDN voucher I get for free credits. , I am a serial “new project” creator, some of these projects make it as far as being interesting to look at and some don’t. Rather than...
  • 27 Sep 2013 : New blog, new platform

    An interesting chain of events happened last week; @thecodejunkie relaunched his blog, I happen to follow his twitter feed and he happened to mention it. The chain continued, I followed the link to his blog for a look and read the post there, I am back and this time I...