/home/jeevanullas

Up in the Cloud!

AWS SDK for Go with Eucalyptus

AWS SDK for Go is currently under development but already there are devops project using the SDK. This prompted us to figure out how to get going with the SDK against Eucalyptus endpoints.

This post shows how to install and configure AWS SDK for Go. It then shows some sample code written in Go using the AWS SDK for Go. It finally highlights current known limitations of Eucalyptus (AWS compatibility).

Installing Go
  • Download the Go archive from here: https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
  • Extract the archive in /usr/local/go/
    tar zxvf go1.4.2.linux-amd64.tar.gz
  • Set the PATH environment variable:
    export PATH=$PATH:/usr/local/go/bin
  • Configure Go Workspace
    mkdir $HOME/go
  • Set the GOPATH environment variable
    export GOPATH=$HOME/go
Install and Configure AWS SDK for Go
  • Download AWS SDK for Go
    go get github.com/awslabs/aws-sdk-go
  • Download dependency for AWS SDK for Go
    go get github.com/vaughan0/go-ini
  • Add local fork as remote
cd $GOPATH/src/github.com/awslabs/aws-sdk-go
git remote add fork https://github.com/jeevanullas/aws-sdk-go.git
  • Switch branch to fork
    git branch fork
  • Configure the URL for the remote fork
    git remote set-url fork git@github.com:jeevanullas/aws-sdk-go.git
  • Modify the appropriate source code to add Eucalyptus cloud endpoints as mentioned here, you should change this according to your endpoints.

Note We disable SSL as the certificate used for Eucalyptus cloud is self signed and currently I have not found a way to bypass the mandatory x.509 checks in AWS SDK for Go

Note We enable more verbose logging of the API calls for debug purpose only

Note We currently only plan to test EC2 and IAM endpoints

  • Commit and Push changes to local fork (assuming current working directory is $GOPATH/src/github.com/awslabs/aws-sdk-go/ and working branch is fork)
git add .
git commit
git push
  • Configure cloud credentials to be used by Go programs
mkdir $HOME/.aws/
touch $HOME/.aws/credentials
vi $HOME/.aws/credentials

[default]
aws_access_key_id = <access key>
aws_secret_access_key = <secret key>
EC2 operations

This section list down some sample code for EC2 API calls using AWS SDK for Go.

  • DescribeInstances

In order to run the sample code shown above save the text into a file named ec2-describe.go and issue following command

go run ec2-describe.go

Output of the run should look something similar to this:

=> [2015-04-28 04:33:53.491970997 +0000 UTC] ec2.DescribeInstances(&{DryRun:<nil> Filters:[] InstanceIDs:[] MaxResults:<nil> NextToken:<nil> metadataDescribeInstancesInput:{SDKShapeTraits:false}})
---[ REQUEST PRE-SIGN ]------------------------------
POST / HTTP/1.1
Host: compute.emea.eucalyptus.com:8773
Content-Type: application/x-www-form-urlencoded; charset=utf-8
User-Agent: aws-sdk-go/0.5.0
Action=DescribeInstances&Version=2015-03-01
-----------------------------------------------------
---[ CANONICAL STRING  ]-----------------------------
POST
/
host:compute.emea.eucalyptus.com:8773
x-amz-date:20150428T043353Z
host;x-amz-date
c040d85fe255c0deb44974d26dca43bef55c2ae2b5e8bca7e2c14f8d72c7c186
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20150428T043353Z
20150428/eucalyptus/ec2/aws4_request
f6ea60f14dd53ee04c97349a8db5354d30c9afec179f82325cf0aa4afa752837
---[ SIGNED URL ]--------------------------------
http://compute.emea.eucalyptus.com:8773/
-----------------------------------------------------
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: compute.emea.eucalyptus.com:8773
User-Agent: aws-sdk-go/0.5.0
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=/20150428/eucalyptus/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=411844e6e46df3d5dd0e3a3dce28d808bef8ccd6b24acdbad82c41dd361d322d
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: c040d85fe255c0deb44974d26dca43bef55c2ae2b5e8bca7e2c14f8d72c7c186
X-Amz-Date: 20150428T043353Z
Accept-Encoding: gzip
Action=DescribeInstances&Version=2015-03-01
-----------------------------------------------------
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 3385
Content-Type: application/xml; charset=UTF-8
-----------------------------------------------------
> Number of reservation sets:  2
  > Number of instances:  1
    - Instance ID:  i-03534581
  > Number of instances:  1


    - Instance ID:  i-effbd63e

The below code is written to run an instance from the CentOS 6.6 EMI on Eucalyptus cloud

The output of go run ec2-run-instance.go is shown below

=> [2015-04-28 06:47:07.980124067 +0000 UTC] ec2.RunInstances(&{AdditionalInfo:<nil> BlockDeviceMappings:[] ClientToken:<nil> DisableAPITermination:<nil> DryRun:<nil> EBSOptimized:<nil> IAMInstanceProfile:<nil> ImageID:0xc20800ad70 InstanceInitiatedShutdownBehavior:<nil> InstanceType:0xc20800ad90 KernelID:<nil> KeyName:0xc20800ada0 MaxCount:0xc20800ad80 MinCount:0xc20800ad88 Monitoring:0xc20800add0 NetworkInterfaces:[] Placement:<nil> PrivateIPAddress:<nil> RAMDiskID:<nil> SecurityGroupIDs:[] SecurityGroups:[0xc20800adc0] SubnetID:<nil> UserData:<nil> metadataRunInstancesInput:{SDKShapeTraits:false}})
---[ REQUEST PRE-SIGN ]------------------------------
POST / HTTP/1.1
Host: compute.emea.eucalyptus.com:8773
Content-Type: application/x-www-form-urlencoded; charset=utf-8
User-Agent: aws-sdk-go/0.5.0
Action=RunInstances&ImageId=emi-d1a30feb&InstanceType=m1.large&KeyName=sshlogin&MaxCount=1&MinCount=1&Monitoring.Enabled=true&SecurityGroup.1=default&Version=2015-03-01
-----------------------------------------------------
---[ CANONICAL STRING  ]-----------------------------
POST
/
host:compute.emea.eucalyptus.com:8773
x-amz-date:20150428T064707Z
host;x-amz-date
1bc51820e6090f2226ae5a56d83eea8f7f298ba0ab6d5116497ba657a5c18117
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20150428T064707Z
20150428/eucalyptus/ec2/aws4_request
01541eed8a31e8db39edd141417c8fc4c483e4a5e1ebac97dc17dd05e3c7e3a7
---[ SIGNED URL ]--------------------------------
http://compute.emea.eucalyptus.com:8773/
-----------------------------------------------------
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: compute.emea.eucalyptus.com:8773
User-Agent: aws-sdk-go/0.5.0
Content-Length: 168
Authorization: AWS4-HMAC-SHA256 Credential=/20150428/eucalyptus/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=e7b46d35c3afcb973d65e6eb9512801cda4657750db453502f63503d5c90660f
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: 1bc51820e6090f2226ae5a56d83eea8f7f298ba0ab6d5116497ba657a5c18117
X-Amz-Date: 20150428T064707Z
Accept-Encoding: gzip
Action=RunInstances&ImageId=emi-d1a30feb&InstanceType=m1.large&KeyName=sshlogin&MaxCount=1&MinCount=1&Monitoring.Enabled=true&SecurityGroup.1=default&Version=2015-03-01
-----------------------------------------------------
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 1273
Content-Type: application/xml; charset=UTF-8
<RunInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2015-03-01/"><requestId>920f29fe-649e-4bab-bfb0-8e30241935d4</requestId><reservationId>r-19f5d012</reservationId><ownerId>849254641345</ownerId><groupSet><item><groupId>sg-156cf40d</groupId><groupName>default</groupName></item></groupSet><instancesSet><item><instanceId>i-42ae2954</instanceId><imageId>emi-d1a30feb</imageId><instanceState>{gfm-js-extract-pre-1}<name>pending</name></instanceState><privateDnsName>euca-10-30-3-98.eucalyptus.internal</privateDnsName><dnsName/><reason>NORMAL:  -- []</reason><keyName>sshlogin</keyName><amiLaunchIndex>0</amiLaunchIndex><productCodes/><instanceType>m1.large</instanceType><launchTime>2015-04-28T06:47:10.697Z</launchTime><placement><availabilityZone>emea-az-01</availabilityZone></placement><monitoring><state>enabled</state></monitoring><privateIpAddress>10.30.3.98</privateIpAddress><ipAddress>10.30.3.98</ipAddress><groupSet><item><groupId>sg-156cf40d</groupId><groupName>default</groupName></item></groupSet><architecture>x86_64</architecture><rootDeviceType>instance-store</rootDeviceType><rootDeviceName>/dev/sda1</rootDeviceName><blockDeviceMapping/><virtualizationType>hvm</virtualizationType><tagSet/><iamInstanceProfile/></item></instancesSet></RunInstancesResponse>
-----------------------------------------------------
{
  Groups: [{
      GroupID: "sg-156cf40d",
      GroupName: "default"
    }],
  Instances: [{
      AMILaunchIndex: 0,
      Architecture: "x86_64",
      IAMInstanceProfile: {
      },
      ImageID: "emi-d1a30feb",
      InstanceID: "i-42ae2954",
      InstanceType: "m1.large",
      KeyName: "sshlogin",
      LaunchTime: 2015-04-28 06:47:10.697 +0000 UTC,
      Monitoring: {
        State: "enabled"
      },
      Placement: {
        AvailabilityZone: "emea-az-01"
      },
      PrivateDNSName: "euca-10-30-3-98.eucalyptus.internal",
      PrivateIPAddress: "10.30.3.98",
      PublicDNSName: "",
      PublicIPAddress: "10.30.3.98",
      RootDeviceName: "/dev/sda1",
      RootDeviceType: "instance-store",
      SecurityGroups: [{
          GroupID: "sg-156cf40d",
          GroupName: "default"
        }],
      State: {
        Code: 0,
        Name: "pending"
      },
      StateTransitionReason: "NORMAL:  -- []",
      VirtualizationType: "hvm"
    }],
  OwnerID: "849254641345",
  ReservationID: "r-19f5d012"
}
IAM Operations

This section contains the API calls from AWS IAM that works against Eucalyptus.

  • ListGroups
    Sample code shown below lists down the IAM groups and works against Eucalyptus.
Documentation

Detailed documentation along with sample code is available at following page http://godoc.org/github.com/awslabs/aws-sdk-go

Limitations

Eucalyptus object storage does not support AWS Signature version 4. AWS SDK for Go only have Signature version 4 support. So we cannot use AWS SDK for Go with Eucalyptus object storage.

This is just the beginning. This now opens up endless possibilities of things to try out. In the next blog post we will show how to use Terraform AWS provider out of the box against Eucalyptus. Stay tuned.