AWS HTTPSConnectionPool max retries exceeded

I’m working with a new AWS account and I am moving to testing Boto3 to use the KMS service. I needed to make sure that the AWS account and secret keys were updated so ran aws configure to quickly update them.

I added the new keys and saw that default region was set to [Ireland] so accepted default and ran the following code

import boto3 s3 = boto3.resource('s3') for bucket in s3.buckets.all(): print(bucket.name)

I was puzzled to get the following error;

>botocore.vendored.requests.exceptions.ConnectionError: HTTPSConnectionPool(host='s3.ireland.amazonaws.com', port=443): Max retries exceeded with url: /... read more

Git hangs while unpacking objects (Windows)

I’m not sure if this is because we’re behind a proxy, the network has issues or my work laptop isn’t great, but for some reason the git clones very often hang during the unpacking of objects.

remote: Counting objects: 21, done. remote: Total 21 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (21/21), done.

There is a way to recover this, if you Ctrl+C to exit the git command then cd into the folder cloned into.

git fsck notice: HEAD points to the unborn branch (master) Checking object directories: 100% (256/256), done. notice: No default references dangling commit: 0a343894574c872348974a89347c387324324

The bit we’re interested in is the dangling commit, if we merge this commit manually all will be fine

git merge 0a343894574c872348974a89347c387324324

Job done,... read more


Adventures with Spark, part one

For 18 months I’ve been working with Hadoop. Initially it was Hortonworks HDP on Windows then Hortonworks HDP on CentOS and for production we settled on Cloudera CDH5 on Red Hat. Recently we’ve been introduced to Spark and subsequently Scala which I am now in the process of skilling up on, the plan is to blog as I learn.

For the first entries I imagine it won’t be much more than the basic tutorial you could read elsewhere, however the plan is to get more detailed as I learn more.

I can’t introduce Scala better than Scala School so its worth taking a look at that.

I am going to use JetBrains IntelliJ IDEA for developing fuller applications, however for playing and learning you can download Spark for Hadoop in TAR format from the Spark Download Page and use the Spark shell.

For now I just extracted it... read more


Update Wallpaper from Bing (OSX)

I’m not a huge fan of Bing search engine, I’ve tried to use it but I don’t like the format of the search results and I don’t think it’s particularly good at finding relevant results either.

I do like Bing wallpapers, and I use Bing Desktop on my Windows laptop to update my desktop to Bings daily wallpaper.

Now that I’ve moved to a Mac I still want to get the picture, but the application is Windows only - so the script below will do the job for you. I’ve set it to download to the users picture folder ~/Pictures/bing-wallpapers just using the current date for the filename.

>import urllib2 import json from os.path import expanduser response = urllib2.urlopen("http://www.bing.com/HPImageArchive.aspx?format=js&idx=0&n=1&mkt=en-US") obj = json.load(response)... read more

Creating environment variables from the command line (Windows)

I know that it is incredibly lazy and a non problem but I find it quite tedious in Windows 8 to go digging for the system environment variable GUI whenever I need to add or update something.

Generally I’m already in the command prompt so I was keen to find a way to create them from there without having to go into search for it each time.

Since Windows XP, setx has been available as an extra download, and more recently it’s included in Windows out of the box - this is the command that I wanted.

To create a persistent STORM_HOME environment variable, use the following command. The /M sets it as a system variable rather than the default user variable.

setx /M STORM_HOME d:\storm-latest

There are a number of other options, do setx /? to see them.

read more