Most developers don’t like to think that their software leaks memory, resources, etc, but in fact it most likely does. Even if the software that you wrote does not leak, keep in mind the system your software runs on top of may leak.

I think there are about 200K lines of code (LOC) that make up Xen, the virtualization platform that Amazon uses, 500K LOC or so for the Java VM (if you are using Java) and 2.5M LOC for a typical distribution of Linux. That is a lot of software - surely there will be memory and resource leaks somewhere in there.

If there are leaks, the system will degrade over time. The longer you keep the system running, the slower it will get. EC2 makes this really easy to solve - just kill your instance every so often and launch a new fresh instance. This means you should be using custom AMIs, EBS and or S3 to store your data of course.

To ensure your application appears to the outside world as a stable system, you’ll need to use Elastic Load Balancing, or at least Elastic IP to create a stable fascade on top of your ever-changing EC2 instances.

The usual assumption is that EC2 should be cheaper than buying a real server. But, lately, I’ve been wondering why we make that assumption?

Below are 3 reasons why I think EC2 should actually be more expensive than a real server.

1. Convenience - even the fastest hardware vendor will take days if not weeks to deliver a new server. With EC2, you can have as many servers as you would like, and they will be ready in a few minutes. We are accustomed to paying more for convenience like this (i.e. express shipping, home delivery, etc), so why should computing be any different?
2. No buyer remorse - how often have you purchased the wrong server? Maybe it is more powerful than you need? Maybe it is less powerful than you need? Usually we just buy as much as we can afford and live with the consequences. With EC2, there is no buyer remorse. You can choose amongst a set of hardware archetypes to meet your needs - you are billed hourly, so you can literally ‘trade in’ your server whenever you need to without any kind of penalty. That seems like a benefit that we often overlook.
3. Portability - a real server might be close in cost or even cheaper in the long term than EC2. But, what happens if you need to move it to another data center? Or another continent? With EC2, you can move your server instance between zones (i.e. data centers) and regions (i.e. continents) with a few clicks of your mouse.

Thanks!

Eric.

Amazon’s Mechanical Turk is named after an 18th century machine named ‘Turk’ that could beat all comers at chess. You can read a nice write up about it here http://www.guardian.co.uk/technology/2009/may/21/answering-services

A friendly reader made an interesting comment about how I sound somewhat secretive when I sign-off on my posts. That was really never my intention, I was actually just trying to sound casual. So, in light of that, I thought I would share a little about myself.

A little history about myself …

I was born in the Moscow, ID, USA, but I was raised mostly in a small, Canadian city named Windsor. I attended the University of Western Ontario, where I studied Computer Science.

I started my professional career in a small company in Toronto called KL Group working in technical support. KL Group was one of the first companies to produce commercial user interface components for Motif. I’m sure I’m dating myself, but Motif is a toolkit for X-Windows, a graphical user interface package used in my UNIX distributions. KL Group eventually introduced components for the then-new Java JDK; this was an area that I worked in as well.

I learned a tremendous amount working for KL Group. Technical support is not the most glamorous position, but boy do you ever learn a lot! Since KL Group built user interface components, the customers I supported were mostly developers who were integrating these components into their applications. Looking back, I think the reason I learned so much from this job is that I was forced to solve problems that I had absolutely on influence in designing. IMHO, when you’re writing code yourself, you can influence the sorts of problems you have to solve because you get to choose (to some degree) - the programming language, the execution environment, the platform, etc. In some cases, as a developer, if you hit a really tough problem, you can find a way around it.

But, in technical support, I didn’t have that option - I had to work with whatever environment, design, etc the customer already had in place. Not to mention there is always a severe time crunch because usually calling technical support is the last resort our customers took.

Anyways, KL Group was a great learning experience for me.  After finishing my CS degree, I joined Microsoft.

At the time, Windows and Microsoft software was actually quite unfamiliar for me. My university and KL Group were both primarily UNIX and Java. It’s funny to say this now, but at the time I didn’t even own a PC my only experience with a PC and Windows was from using a PC X windows server at KL Group.

My first job with Microsoft was in their Developer Tools Division - I really love developer tools and would spend most of my 8-year career here. I worked initially in quality assurance for Visual Studio. Quality assurance is another position that is not very glamorous, but is a great learning experience. When you’re charged with finding bugs in software, you get a really good idea of how software should be built.

I eventually left my beloved Developer Division to work as a developer in Windows 2003 - I was part of the team that worked on UDDI. If I remember correctly, we had the distinction of being the first .NET feature in Windows 2003, as well as one of the first web service related products.

UDDI was great fun, but after a while I realized that Developer Tools was where my heart was. I returned as a program manager for the profiler feature in Visual Studio Team System.  At Microsoft, feature teams are essentially made up of developers, testers and program managers. A lot has been said and written about program managers at Microsoft, so there isn’t much for me to add. This was an interesting position, but I’m not sure it was the best fit for me; but being a PM helped me get to my next position, which I think was the best fit for me.

My next and final stop at Microsoft was to be the product manager for Team Foundation Server - the server-side component to Visual Studio Team System. I really think this is where I did my best work at Microsoft. Being a product manager is a combination of many skills - coding, speaking, evangelizing, planning, etc. I spent about 2 years traveling the world telling people about this new product.

While I loved my job, after 2 years or so I decided it was time to move on. I’ve always had the dream of starting my own business, so in late 2006 I decided to do that.

I started a small consulting company named CounterPunch Software. My business was basically training, custom development and building product demonstrations. My legal company name is still CounterPunch Software, but in 2008 I decided to rename my company as WonderAffect. I thought this name better reflected the product demonstration related work that I was doing.

Building product demonstrations led me to Amazon Web Services. I loved using AWS for quickly spinning up computers to host demos on. Even when I was building Windows-based demonstrations, I often used SQS, SimpleDB and S3 for storage and communication.

I’ve made use of AWS since 2006 or so and decided to concentrate on it as part of my business. Recently, I started this web site to blog about subjects in AWS that are of interest to me. I also do training, development and consultation for Amazon Web Services.

If you want to see what I’m up to, check out: www.learnaws.com and www.wonderaffect.com I used to blog at blogs.counterpunchsoftware.com, but I haven’t posted there in quite some time.

Finally, here is what I look like (granted it is a glammed-up profile shot :)):

Well that’s all I can think of to share right now; if anyone has any questions, please feel free to ask.

Thanks!

Eric Lee.

And as an added bonus, they are running a Tax Day sale today!

EC2 Reserved Instance Pricing is now available for the EU region

Similar to a futures market for computing power, EC2 Reserved Instances are a great way to lock in a price; now it is available in Europe.

We are excited to announce that Amazon Elastic Compute Cloud (Amazon EC2) Reserved Instances are now available in Europe. Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly usage charge for that instance. After the one-time payment for an instance, that instance is reserved for you, and you have no further obligation; you may choose to run that instance for the discounted usage rate for the duration of your term, or if and when you do not use the instance, you will not pay usage charges on it.

Please visit http://aws.amazon.com/ec2 for more information on Reserved Instances, including pricing for the Europe Region.

VAT and tax help

Helpful tax info (including some information related to VAT) can be found here: http://aws.amazon.com/tax-help/

SQS is now available in the EU

This is really new, Simple Queue Service is now available in the EU region - http://aws.amazon.com/about-aws/whats-new/2009/04/09/announcing-amazon-sqs-wsdl-version-2009-02-01-and-amazon-sqs-in-europe/.

Handling Amazon Web Services Disputes

This is not the most pleasant thing to talk about, and I don’t think there have been any major disputes so far, but European AWS users should note section #14 in the user agreement (http://aws.amazon.com/agreement/) about how disputes are handled.

14. Disputes

14.1. Notwithstanding anything to the contrary, we may seek injunctive or other relief in any state, federal, or national court of competent jurisdiction for any actual or alleged infringement of Amazon’s or any third party’s intellectual property and/or proprietary rights. Any dispute relating in any way to your visit to the AWS Website or to products or services sold or distributed by AWS or its affiliates in which the aggregate total claim for relief sought on behalf of one or more parties exceeds $7,500 shall be adjudicated in any state or federal court in King County, Washington, and you consent to exclusive jurisdiction and venue in such courts. You further acknowledge that our rights in the Amazon Properties are of a special, unique, extraordinary character, giving them peculiar value, the loss of which cannot be readily estimated and may not be adequately compensated for in monetary damages.

14.2. Governing Law. By using the Services, you agree that the laws of the State of Washington, without regard to principles of conflicts of laws, will govern this Agreement and any dispute of any sort that might arise between you and us. The parties expressly exclude application of the United Nations Convention for the International Sale of Goods to this Agreement.

Service Level Agreements

This is not specific to Europe of course, but it came up during the training so I thought I would try and gather the various SLA information.

The SLA for EC2 can be found here http://aws.amazon.com/ec2-sla/, here are some highlights:

AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit…

The SLA for S3 can be found here http://aws.amazon.com/s3-sla/, again here are some highlights:

AWS will use commercially reasonable efforts to make Amazon S3 available with a Monthly Uptime Percentage (defined below) of at least 99.9% during any monthly billing cycle (the “Service Commitment”). In the event Amazon S3 does not meet the Service Commitment, you will be eligible to receive a Service Credit…

There is no SLA for SQS right now, but here are a few bits of information about security, usage, etc.

From the Amazon User Agreement:

5.4.1. Provided that you comply with the terms of this Agreement and our policies and procedures for the use of Amazon SQS, you may use Amazon SQS in connection with data owned or lawfully obtained by you (such data, to the extent to which actually used in connection with Amazon SQS, “Your Amazon SQS Content”). You acknowledge that neither we nor our licensors are responsible in any manner, and you are solely responsible, for your Amazon SQS, we will not sell or license your Amazon SQS Content and will not disclose Your Amazon SQS Content except as we may determine to be necessary or desirable to comply with the Agreement, the request of a governmental or regulatory body, subpoenas or court orders, or for other legal purposes.

5.4.2. Your use of Amazon SQS is subject to the limits specified in the most recent user documentation, including limits on the available number of queues or messages, message size, and the number of days during which a message or inactive queue can be maintained. You may not knowingly create and maintain inactive queues. We may delete, without liability of any kind, any of your Amazon SQS Content that sits in a queue or any queue that remains inactive for more than the number of days specified in the user documentation.

Some relevant SQS FAQs (http://aws.amazon.com/sqs/faqs/):

Q: How reliably is my data stored in Amazon SQS? Amazon SQS stores all queue and message information in Amazon’s network of highly reliable, highly available data centers. All messages are stored redundantly on multiple servers and in multiple data centers, which means that no single computer or network failure renders SQS messages inaccessible.

Q: How can I secure the messages in my queues? Authentication mechanisms are provided to ensure that messages stored in Amazon SQS queues are secured against unauthorized access. Only the AWS account owners can access the queues they create.

Amazon SQS uses proven cryptographic methods to authenticate your identity, either through the use of your Access Key ID and request signature, or through the use of an X.509 certificate. For the details of how to use either of these authentication mechanisms with Amazon SQS, please see the Amazon SQS Developer Guide.

Q: What happens if there is no activity against a queue for an extended period of time? We reserve the right to delete a queue if none of the following requests have been issued against the queue for more than 30 consecutive days: SendMessage, ReceiveMessage, DeleteMessage, GetQueueAttributes and SetQueueAttributes. You should design your application with this in mind.

Since CloudFront is based on S3, there isn’t too much more to say in terms of SLA. One interesting thing to note, if you expect to serve a lot of content:

By default, a customer’s Amazon CloudFront distributions support peak data transfer speeds of 1,000 megabits per second and peak request rates of 1,000 requests per second. If you expect more than this amount of traffic, please complete the form below. We will add more capacity to your distributions within 2 business days.

The form mentioned in the quote above can be found here http://aws.amazon.com/cloudfront-request/

SimpleDB is still in beta so there is no SLA associated with it. Not too much to note about it - one blurb in the Amazon usage rights about old data being deleted after 6 months of inactivity.

5.9.2. You must comply with the terms of the technical documentation applicable to (including the Amazon SimpleDB Developer Guide) as posted by us and updated by us from time to time on the AWS Website, including, without limitation, any limitations on the number and total size of domains, items and attributes that may be stored on the Amazon SimpleDB servers. We may delete, without liability of any kind, any of your Amazon SimpleDB Content that has not been accessed in the previous 6 months.

That’s the major small-print stuff I wanted to point out. If I missed anything please let me know.

Thanks!

Eric.

Amazon recently announced a new service - Amazon Elastic MapReduce. This new service is basically an environment where Hadoop - the popular grid computing framework - is installed and ready to run jobs.

All you really need to do is write your job and choose how many instances you want in your grid.

I wrote an involved sample and write-up for this service based on the Freebase data set, but this service is so easy to use I wanted to get another sample written as well.

A common use for grid computing frameworks like Hadoop is to use them to index web sites. It is commonly believed that Google uses Hadoop (or a framework like Hadoop) to create their massive search indexes.

I decided to do something similar on a much smaller scale. Like many people in technology, I like to read TechCrunch. Part of the fun in reading that site is going through the comments; it is a very lively site, so there are literally dozens of comments per article.

I wanted to get an idea of who is posting the most comments; this kind of question is perfect for Amazon Elastic MapReduce.

The first thing I did was to use HTTrack to download TechCrunch.com to my local file system. I hope they didn’t mind the extra traffic - as far as I can tell HTTrack is pretty good about following robot guidance, so hopefully it wasn’t too much of a nuisance.

I actually started up a Windows EC2 instance and ran HTTrack from there; EC2 instances get to use Amazon’s network insfrastructure so the download was pretty fast. TechCrunch is a huge site, so it took about 8 hours to get the entire site down.

HTTrack downloads everything and replicates directory structures so that each link will work. This is great, but Amazon Elastic MapReduce (actually Hadoop) cannot traverse directories, so I needed a flat layout.

I wrote a small program to traverse into each directory created by HTTrack and look for HTML files. Each file that was found was given a unique name (a GUID) and copied into a flat directory.

Amazon Elastic MapReduce uses S3 for its input and output. So, my next step was to take my flat list of uniquely named HTML files and copy them into an S3 bucket. I used S3 Organizer to do that. I ran this from my Windows EC2 instance as well since there were so many files - it took a few hours for the upload to finish.

With my files in S3, the fun with Amazon Elastic MapReduce starts. There are 2 ways to write jobs in Amazon Elastic MapReduce (and in Hadoop) - you can use streaming, or you can write a Java JAR.

I find streaming to be much easier. All you do is read from standard input and write to standard output; as a friendly reader pointed out, Amazon Elastic MapReduce supports a number of different languages.

Develop your data processing application authored in your choice of Java, Ruby, Perl, Python, PHP, R, or C++. There are several code samples available in the Getting Started Guide that will help you get up and running quickly.

In my experience, I’ve only used Ruby and Python for streaming jobs and they both worked well. I had some trouble with PHP during the private beta and have not had a chance to try it again.

I’m going to use Ruby for this example. The basic structure of a streaming job looks as shown below:

Amazon ElasticMap Reduce is going to traverse the S3 bucket location we give it and give our code the contents of each file. Amazon ElasticMap Reduce will decide how many instances of our code it needs. We don’t have to worry about that in our code.

The code we are writing is called the mapper - essentially, we are taking raw input and outputting a structured format. Another chunk of code - called a reducer - will handle this structured output.

So what will our input look like? Since we are just handling the HTML files, our input will literally be the HTML pages that make up TechCrunch.com. As an example, let’s have a look at the page that announced Amazon Elastic MapReduce (http://www.techcrunch.com/2009/04/02/with-hadoop-amazon-adds-a-web-scale-file-system-to-its-cloud-computer/)

We are interested in comment authors, so scroll down to the Comments section. I am using FireBug - a really handy tool for analyzing web pages - and it lets me right-click on an element and view its source.

You can of course do a complete view source for the page, but that does not take into account dynamically generated (a la AJAX) source. FireBug works from the document object model, so it is able to show you AJAX-generated code.

Anyways, in our particular case I don’t think there is any dynamically generated HTML so it does not matter; but FireBug is a good tool to keep in mind.

The result of our inspection is

We can see that our comment author value is contained in span tag that has a CSS class of comment_author vcard. In this particular case, there is a link to the author’s blog.

There are actually 3 variations in how the comment author is shown:

1. There can be a link as shown above
2. There can be a link to their Facebook profile
3. There name can just be shown with no link

Our mapper code will handle each of these cases. Ruby has pretty convenient Regular Expressions, so we can define the patterns we are looking for ahead of time.

The first pattern finds a comment author; the second pattern helps us narrow down whether we have a link in the name.

I’m not a regular expressions expert, so there may be a more efficient way to describe the pattern; with regular expressions there always seems to be a be

The first part of our code is quite simple. We will take each line of input that we are given and try to match it with our regular expression.

If we get a match, we have a comment author.

If we have a comment author, then we want to check to see if there is a link to the name (case #1 and #2) or if it is just the name (case #3).

We are going to test for case #3, that is the easiest to handle

If this if statement evaluates to true, then we are just going to print out some structured output

We will talk more about why the output looks that way. For now, our mapper code looks as shown below.

Now let’s handle case #1 and #2. All we are doing is using our second regular expression to pull out the value behind the link. You’ll notice a little bit of a hack where I am removing the ending fb:name element. I couldn’t find an easy way to express that with a regular expression, so it seemed easy enough to just remove it afterwards.

So, in both cases, we end up printing out a value like LongValueSum:Eric Lee 1. This is the output of the mapper.

In Amazon Elastic MapReduce (and in Hadoop in general), we process our data by using mappers and reducers. A simple explanation of how these work together is that the mapper makes some sense of the raw data, but it does not do any correlation. The reducer takes the output of the mapper and does the correlation.

In our case, our mapper is ‘making some sense of the data’ by looking for the comment author values. There are lots of pages in TechCrunch that either do not have comments, or are simply not blog posts. This input gets filtered out by our mapper since nothing in it matches our search terms.

We can write a reducer, but Amazon Elastic MapReduce has a few standard reducers - one of which is called aggregate. The aggregate reducer takes output like LongValueSum:Eric Lee 1 and correlates it for us.

For example, if we were to parse through all of TechCrunch.com, we will have authors who have made several comments. So, our mapper will produce output like:

LongValueSum:Eric Lee 1

LongValueSum:Eric Lee 1

LongValueSum:Eric Lee 1

LongValueSum:Eric Lee 1

Remember, it is not the job of the mapper to do any correlation; just output structured data. If we use the default aggregate reducer, it will take this output and correlate it for us to produce:

LongValueSum:Eric Lee 4

I’m pretty lazy so I try to design my map reduce usage to use the default aggregate reducer as much as possible. It seems like most of the time, that type of correlation is really what you want anyways.

At this point we have our raw data from HTTrack, we have our mapper and we know we are going to use the default reducer. Let’s do a quick test before we send all of this to Amazon Elastic MapReduce.

Of course, we can test our code by installing Hadoop on our local system and running a job. But, one of the reasons that I like streaming jobs so much is that they are easy to test; to do a simple sanity test, we don’t need a local installation of Hadoop.

For testing purposes, I downloaded a few of the HTTrack files to my local system.

Amazon Elastic MapReduce is going to take each of these files and send them to my mapper. I can mimic this by using cat (on Mac/Linux) or type (Windows).

If I run this command, all of my files will be outputted to standard output; now we just have to point this to our mapper. We can do that by using the pipe (|) command.

This simulation is good enough for a quick sanity test of our mapper - if we run it, we should see our structured output.

In the little snippet above, we can see a few cases where our reducer will do some correlation for us. Your results will vary based on your test data.

At this point we are ready to use Amazon Elastic MapReduce.

The first step is to upload our code into S3 - I use S3 Organizer to create some sub-directories to help me organize my code.

This is where I ran into a little confusion. As you might know, in S3, there is no such thing as hierarchies. All buckets are essentially peers and in each bucket there are objects. There is no such thing as buckets within buckets.

But, S3 Organizer allows you to use a / in your object names to mimic a directory structure in the UI. This is purely a UI trick - in reality, your S3 data is all flat.

What can be a little confusing is that all of the samples in Amazon Elastic MapReduce use this nomenclature. It’s actually pretty handy once you get used to it, but it is a little confusing at first.

Creating a job in Amazon Elastic MapReduce is really easy. First, you go to the console https://console.aws.amazon.com/elasticmapreduce/home

Sign-up for the Amazon Elastic MapReduce service if you haven’t already done so.

Once you’ve signed in, create a new job by clicking the Create New Job Flow button.

This brings up a dialog box that will define your new job.

The name of the job does not really matter, pick anything you want and press the Continue button.

The next page is where you configure your job; take a deep breath at this point. The data you are entering is not difficult, but you can’t make any typos here. There is limited checking in the dialog, you really just have to get all of the values right.

The first piece of data is the input location - this is the S3 bucket where your raw HTTrack flat files are. I use S3 Organizer to navigate to the location and then cut and paste the value.

Amazon Elastic MapReduce does not want the leading /, so the value I put into the input location is learnaws-mapreduce/data/techcrunch/
Make sure to enter your value exactly as shown in S3 Organizer (minus the leading slash); this includes the trailing slash.

I know I’m probably beating you over the head about not making any typos - the reason is that when you start your job, there is a fair bit of overhead that Amazon Elastic MapReduce will go through. This overhead usually takes about 5 minutes to complete. There are a lot of typo-related errors that are not caught until after this overhead is complete - so you might have to wait 5 minutes before your job fails because of a simple typing mistake.

The second location is your output location. This is just a bucket location in S3 that you have created. The trick here is that it must be a location that does not exist yet. Amazon Elastic MapReduce will not overwrite your output, so if you give it a location that already exists, it will fail.

Using S3 Organizer again, here is how I organize my output.

As you can see, I have a few output locations already (i.e. tc_comments_3, tc_comments_2, etc). The value I specify in the dialog will be a new location (i.e. tc_comments_4).

It makes sense to double-check your values at this point. The input location has to exist, the output location cannot exist yet. Both location paths should have a trailing /.

OK, enough nagging about typos now we specify the location of our mapper code. It is just a direct path to our S3 object.

Our value in the dialog box should be:

Lastly, we specify our reducer. If we wrote our own reducer, then we would just give it a S3 path to the code. But since we are using a default one, we just specify the name - aggregate

That’s all the data we need for our job - one more time, double check your values and then press the Continue button.

The next page lets you choose how big of a cluster you want to process your data; feel free to keep the default or choose your own value. I used 8 instances of the default small variety to process the TechCrunch data. The processing took about 2 hours to complete.

Press the Continue button when you have chosen your setup. This should bring you to the last page of the dialog box; press the Create Job Flow dialog to create and run your job.

If all goes well, your job should complete in a couple of hours.

Amazon Elastic MapReduce and Hadoop are hiding a fair bit of complexity from us. Our small mapper Ruby script will be instantiated a number of times across the number of machines we chose. Hadoop figures out the optimal number of mappers to run in parallel based on the input we give it. Amazon Elastic MapReduce handles the work of instantiating the number of machines (EC2 instances) behind the scenes for us. Since there are a number of mapper instances writing to standard output, it helps to have a low-latency file system to handle all this output. Hadoop can use a few different distributed file systems (including its own). But, Amazon Elastic MapReduce configures Hadoop to use a S3-based distributed file system.

Again, all of this complexity is hidden from us. We can just sit back (or walk away) and let the processing complete.

For some funny reason, I felt some odd sense of accomplishment in keeping 8 computers busy for about 2 hours

The data that will be generated is stored at your output location in S3.

I did some experimentation with running another job to process this output and store it into SimpleDB. It was an interesting exercise, but ultimately I decided it was wrong for this example.

I downloaded these results to my local file system - there really isn’t all that much data once it has been correlated. Here is a snippet from one of those files.

You can see that our reducer has done the correlation for us - there are some authors with 100+ comments!

Now the question becomes - how to visualize this data? I’m sure there are many different ways of visualizing a large data set. I chose a relatively simple one.

I decided to use a tag cloud type of interface.

I found one to use at http://www.phpclasses.org/browse/package/4158.html written by a gentleman named Er. Rochak Chauhan.

This code can be configured to read from a CSV file; the format of the CSV is tag, url, count

This is really close to the format that I already have from Amazon Elastic MapReduce; basically all we need to do is replace the TAB with a comma and put in a URL.

I used another Ruby script to do this.

I didn’t feel like doing anything too fancy - all this script does is take our Amazon Elastic MapReduce output from standard input, convert the tabs to commas, put in a blank URL and spit the result to standard output.

I ran this script as:

This produces 1 giant CSV file that we can use with the tag generator.

The tag generator is well written and easy to understand. It is PHP-based, so we just need to run it on a web server that supports PHP. I use XAMPP on the Mac.

The only changes we need to make are to copy the CSV file to the same location as example.php and change the name of the CSV file being referenced.

Then, we can request this page (i.e. http://localhost/comments/example.php) to produce our tag cloud (click on the image below to see it full size).

To summarize our application, we did the following:

1. Use HTTrack to download our web site
2. Flattened out the resulting directory structure and gave each HTML file a unique name
3. Uploaded it all to S3
4. Wrote our Ruby mapper
5. Created an Amazon Elastic MapReduce job
6. Downloaded our resulting output
7. Found a handy tag cloud generator
8. Wrote a bit more Ruby to convert our output into a format our tag cloud generator understands

That’s about it!

Amazon Elastic MapReduce makes doing this type of processing really easy - if you are using a streaming job, you are really just reading from standard input and printing to standard output. You can request as big of a cluster as you need, so no job is literally too big.

Everything we did from the web-based UI can also be done from an API. Like all of the Amazon Web Services, this new service has an XML web service based API. A Ruby client is included in the resources to get you started right away. Check out http://aws.amazon.com/elasticmapreduce for more details.

Thanks!

Eric.

1. Bundled AMIs
2. EBS volumes
3. S3

Most people use a combination of these 3; I’ll describe the basic use cases for each:

Let’s use a Windows EC2 instance as an example. Suppose you want to host a web site on your instance.

The first thing you do is install IIS on your instance. As you probably know, if you terminate this instance, the installation you did prior will be gone.

In order to keep this installation, you need to bundle it into an AMI. As you’ve seen, the console makes quick work of this, you can right-click and in a few minutes you will have your AMI.

When you start up that AMI, the installation of IIS you made will still be there (note - only the C:\ is saved, not the D:\).

If you put your web files in the default IIS directory (i.e. C:\inetpub\wwwroot), then they will be saved as well.

So, you can save everything you need by just bundling. But, there are 2 drawbacks to only doing bundling:

1. Bundles are limited to 10GB in total size. This includes the base operating system. So, you have a fixed limit to how much data you can put in your instance.
2. The only way to save changes to your instance is by bundling again. Bundling is pretty easy, but probably not something you want to often (i.e. daily) basis.

What is really common to do is to bring EBS volumes into the mix. As you’ve seen, you can mount an EBS volume like a regular hard drive and see it as E:\ or something.

What a lot of people then do is use EBS volumes to store their data - so you could use set IIS to use E:\inetpub\wwwroot instead. Any change you make to data in an EBS volume is saved pretty much immediately.

It is probably more common to see databases being hosted on EBS - so I might configure SQL Server to store its databases on E:\ instead of C:\. Any change to the database is permanently saved.

So, EBS gives you a few advantages:

1. No need to re-bundle to save instance data
2. No 10GB limit, you can store as much as you want.

But there are a few drawbacks with EBS as well.

1. You pay for an EBS volume based on allocation. When you create an EBS volume of 100 GB for example, you immediately start paying for that space, whether you actually have data stored there are not.

2. EBS volumes are conceptually like USB-powered portable hard drives. They are easy to plug-into a computer (or instance), but you can’t share them with another computer simultaneously. You can duplicate the contents of an EBS volume by creating a snapshot and then creating a new volume based on that snapshot, but the sharing is not in real-time.

The last option is S3.

With S3, you put your data into buckets that are Internet-accessible. So, what you might do is put your web site contents in a .zip file in a S3 bucket. When an instance starts up, it is configured to pull down that data from S3 and unzip it onto its local (or EBS) volume.

With S3, you get a few advantages:

1. You only pay for the data you store; there is no concept like the pre-allocation in EBS, you just pay for what you upload into S3.

2. You can share data between instances in real-time (or near real-time, look up eventual consistency as it applies to AWS).

But, with S3 there are some drawbacks too:

1. You are making HTTP requests to get data, so it will be quite a bit slower relatively speaking than data in C:\ or in EBS.

2. Since you are making HTTP requests, it is not as simple as referencing a regular volume label like C:\ or E:\. Though, there are some plug-ins that map S3 buckets to volumes.

As you can probably see, each technology has its own level of convenience, cost and capability. Most people tend to use a combination of all 3 to get the results they are looking for.

Thanks!

Eric.

My domain registration is with Dreamhost. I almost always start out with Dreamhost - they are really cheap, under $20/month and you get all kinds of good stuff (i.e. Jabber server, SVN, etc).

I actually ran learnaws.com from my Dreamhost account for a little while; I wasn’t getting much traffic and I just wanted to try the concept out.

But, when I felt I wanted something more serious, I decided to move from Dreamhost. I just changed my name servers, I didn’t bother changing my domain registration. All of my payment information is setup in Dreamshost already so it makes renewals easier.

I had to decide between running my web site on EC2 or on a higher-end, local web hosting company that I use for higher traffic sites.

What I had in mind with learnaws.com was to host a lot of video tutorials. I like creating the videos and I think it’s a format that people tend to like.

Given that, I decided to use S3/Cloudfront to serve up these videos - I figured this would give me the best performance.

IMHO, you get the biggest performance gain serving up progressive download media like this when you serve it from a geographically closer server.

Since S3/Cloudfront is doing the heavy-lifting on this site, I decided to just host the web content in my local web hosting company. It’ll just spit out the static text parts of my site, so I figured I didn’t need the added performance of a dedicated EC2 instance.

I’m still deciding what my next step will be.

My big concern is that I’m using WordPress as my content management system, and as you probably know, that uses MySQL as its backend.

I’m planning on customizing WordPress to use SimpleDB as its back-end database. I think conceptually it should be possible; we’ll see how that plays out.

I think once my experimentation with SimpleDB as a WordPress backend is done, I’ll start hosting in EC2. If all goes well, I’ll put a HAProxy in front of a set of EC2 servers running my web site and those instances will talk to a set of domains in SimpleDB.

If the SimpleDB doesn’t work out, then I’ll probably have to think about doing some MySQL replication or splurge for a big, central EC2 instance.

In any case, this is what I really like about AWS; it’s very easy to adopt it step by step.

Thanks!

Eric.

I’ve found that Amazon’s Simple Queue Service is a really great way to ‘glue’ a couple of applications together. My latest integration has been with Salesforce.com.

Just in case you aren’t familiar with it, Salesforce.com is a hosted customer relationship management (CRM) system. It was one of the first ‘cloud’ services available commercially.

I’m building an event management system where people can sign up for events. The client I’m working with wants to create a ‘Lead’ in Salesforce.com for each person that signs up for an event.

I don’t want this operation to interfere too much with my customer’s sign-up experience so I’m going to do my integration Salesforce.com offline.

This is where Simple Queue Service (SQS) is so handy. What I can do is capture the information that I need to generate a lead for my customer.

The message body that I’m going to send will just be in query string format:

My sendMessage function is defined as follows:

This function is obviously really simple, but I do find it really useful.

The great thing about SQS is that your messages persist for up to 4 days. So I’m really in no hurry to write the logic to process these messages.

To work with Salesforce.com from an integration perspective, the first thing you need (assuming you don’t have a full account already) is a developer account; you need a developer account in order to use the API. You can get one from http://developer.force.com/.

Once you login, your next step is to enable the API integration feature. You can enable this from the Setup menu, which is at the very top of your screen.

Your next step is to enable the API. This is done by downloading the enterprise WSDL file. As far as I can tell, you don’t actually need the file itself; the act of downloading it enables the API for your account.

Your next step is to open up your Personal Setup

In order to access your account from the API, you have to append a security token to your password. Choose the Reset My Security Token to get this token for the first time.

That’s it, your account is all set. Now we just need a library to work with the Salesforce.com API. There is a library available for a wide selection of languages - you can see them all at: http://wiki.apexdevnet.com/index.php/API.

I chose the PHP Toolkit library; this library works really well, the only thing is that it depends on the SOAP module for PHP. You’ll get an error about not being able to find the SoapClient class if you don’t have it. If you don’t have it, you’ll need to install it (i.e. yum install php-soap).

I’m on a Mac and installing php-soap is a bit more complicated. I actually didn’t bother because I was planning on deploying my application to an EC2 server anyways. For my local development I use XAMPP. The version of PHP that is installed with XAMPP has soap installed so I just ran that php interpreter instead (i.e. /Applications/xampp/xamppfiles/bin/php).

Once you have your Salesforce.com account setup and your wrapper configured, the actually coding is pretty straighforward.

I’m using my adaptive poll class to implement my Salesforce integration worker.

PHP makes it really easy to parse a query string with its parse_str function.

I store the name of the event in my system in SimpleDb, so I do a quick query to get that; and I instantiate my Salesforce client in the code listing below:

Lastly, it is just a matter of creating the Lead object and saving some information to it:

Now when someone signs up for an event, I have an additional line of code to submit their information into my queue. This really doesn’t take very long (I actually do it with AJAX asynchronously) so there is no affect on the users’s experience.

I usually have 1 or more workers running, so as soon as that message ends up in the queue, it is processed and a lead is created in Salesforce.com.

There are a few other places in my application where I am gathering user data - I can easily just make a call to my sendMessage function to put their data into Salesforce.com as well.

In my opinion, Salesforce.com and SQS make great partners. With Salesforce.com, you can run CRM without the usual CRM headaches and high costs. SQS gives me an easy, generic way to do offline processing. I simply just dump my information into SQS. I don’t even need to have anything reading from the queue right away; my messages will persist for 4 days. Once I have my worker up and running, I can read each message and process them accordingly. Even if I have applications written in other programming languages, I can still use SQS because all I am doing is making an HTTP request with a string formatted in some way that I have defined.

Thanks!

Eric.

One of the first things you are likely to do when using a Windows AMI in Amazon EC2 is install some Windows Components. The most obvious would be a web server - IIS.

What you’ll find is that you’ll get prompted for a Windows installation CD. The way you give your Windows instance access to this CD is by creating an EBS volume from an Amazon snapshot.

Please see the video below for steps on how to do this.

1. Amazon user name and password - not a big deal if you lose this; as long as you have access to the email account associated with your account, you can reset your account and get access again.
2. Amazon key and secret - as long as you can get into your account, you can see your key and secret. If you ever suspect that your account has been compromised, you can reset your secret. Doing this means any application that is using your secret will need to be updated.
3. X.509 Certificate associated with your account - you can always download your public certificate, but you only get one chance to get your private key. When you first create your certificate, you will be prompted to save your private key. If you don’t save the key then, or you lose the key, that’s it, you need to create a new public key and you’ll get a new secret. These certificates can be used to authenticate to AWS in an application you build, but I haven’t seen many people do that. The only real issue I’ve seen is that the EC2 command line tools use these certificates. So if you lose the private key, you’ll need to get a new private key and update your EC2 command line tool installation.
4. keypair used to launch your instance - if you’re using Linux or Solaris EC2 images, you can’t lose this. If you do, you won’t be able to get access to your instance because you don’t know the root password. Amazon support may be able to help you, but that’s about it. If you are using Windows, then there is a chance you have the administrator password written down, or maybe you changed it to something you know. Otherwise, you’re stuck, you won’t be able to get access. Your only choice will be to terminate the instance and restart it. Of course, that means you will lose your instance data.

But, for a more sophisticated approach, check out Joel’s sample at http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1941. His calls his approach ‘adaptive polling’.

The motivation for picking a different polling algorithm is efficiency and cost. SQS charges you for every operation (about 1 penny/10, 000 requests), so if you poll every second when there are no messages, you are spending a bit more money than you need to.

Also, if you poll and sleep at regular intervals, you aren’t processing messages as quickly as you could be. For example, if there are 1000 messages in your queue, you don’t necessarily want to sleep for 1 second after reading each one, you might as well try to read them as fast as you can. Strike while the iron is hot as it were

I’ve implemented a version of adaptive polling in PHP, I posted that (along with the functor sample) here: http://static.learnaws.com/sample_code/PHPFunctor.zip

All of the polling is done in the AdaptivePoll class; to act on any given message, you simply have to define your bodyFunction method as shown below:

Using this class lets me quickly create workers for any processing that I want to do. The example above is obviously pretty trivial, but using SQS-driven workers and adaptive polling, I can always be assured that I’ll be as scalable as possible.

Thanks!

Eric.

The functor pattern is usually limited to a single process (i.e. wherever the code is executing), but SQS and PHP enable you to take this pattern to a distributed and scalable level.

Let’s illustrate this with an example. Consider a simple class defined in helloworld.php

In our code, we will instantiate this class and pass in a message that our object will store.

Our $helloWorld object is going to be our functor; in theory, we should be able to pass this object to another recipient, they should be able to invoke methods on it (sayHello in our case). Before we start passing this object around, we need to be able to preserve its state. We can do that by calling the serialize function in PHP.

The serialize function takes our object and stores its state as a string; in many languages, the format used is XML. By default in PHP, it is just some string representation. Our serialized object string will look something like the following:

O:10:"HelloWorld":1:{s:7:"message";s:14:"Hello World!!!";}

This is where SQS comes into play; we will send this serialized string as the body of an SQS message. Whoever receives this message can de-serialize the string into an object again and invoke its functionality.

Before we send our serialized string, it is a good idea to urlencode it first. Some of the characters may interfere with the XML format of an SQS message.

The code to create the queue, serialize and urlencode the object and send it as an SQS message looks as shown below.

Notice that we are hiding the implementation of how we are sending the message behind our sendMessage function; all we are doing is constructing the send message request and setting the body to our serialized and urlencoded state.

Since we are using SQS, our code to receive these messages can live in another process. This really helps our scalability because we can run these worker processes on multiple machines if we need to process a large number of messages.

The entire code for our worker is listed below:

We are using a class called AdaptivePoll to help us out here. I’ll explain this class in another post. For now, just know that it polls a given SQS queue and calls bodyFunction if it finds a message.

When we get a message, notice how we are url-decoding the message and then de-serializing it. When we execute sayHello we should see a Hello World!!! message get printed out - this is the same message we set in our earlier code

SQS is an elegantly simple solution to a wide range of problems; in this case, we can use it to implement a generic, distributed functor pattern.

• Architectural Consulting
• Custom Development
• Training (on-site and online)

We will publish more details about these services shortly; in the meantime, please contact us at learn@learnaws.com for more information.

Have a look at our blog for some videos and tutorials about AWS.

We also offer customized, on-site training and development services; contact us at learn@learnaws.com for more information.

More specifically, Amazon Web Services is all about cloud computing. Cloud computing is getting to be a pretty over-used term nowadays.

The way we see it, cloud computing is pretty simple. Instead of running as many computers as you can afford, let someone else do it.

Amazon is doing exactly this with their Amazon Web Services initiative.

Instead of you buying as many computers as you can afford and running them, Amazon Web Services gives you access to its vast data center - you can use as many computers as you want, for as long as you want, and you only pay by-the-hour for what you use.

The same goes for storage and communication - you pay for each gigabyte you store and you pay for each message that you send and receive. The cost is very small; literally fractions of pennies in most cases.

Amazon is well-suited to doing this. Over the last 10 years or so, Amazon has honed its computing, storage and messaging infrastructure expertise in the unforgiving world of e-commerce. Today, Amazon.com is one of the busiest, and most reliable web properties in the world. The same infrastructure that drives this site is what is being made available through Amazon Web Services.

What makes AWS really revolutionary is that it enables individuals and companies to offload the responsibilities of computing, storage and messaging to Amazon. This enables us to concentrate on our core business.

Amazon Web Services truly offers computing as a utility - we can use as much as we want, and we only ever pay for what we use.

The video below illustrates how to do exactly that; for more details on how to setup FTP for your Windows installation, please see http://support.microsoft.com/kb/323384.

When you first start out with Amazon EC2, one of the most obvious questions to ask is ‘How do I get data onto my shiny, new Amazon-run machine?’.

The answer is pretty straightforward, but not always obvious when you are first starting out.

I’ve found that this is more of a problem with Windows EC2 users. Most Linux folks are used to wget and scp so they can get their data back and forth pretty easily. There are of course Windows ports of those tools, but I think it is more common for Windows folks to use things like FTP, WebDAV, etc.

In the next set of short videos, I’m going to explore a number of ways to copy data back and forth between a Windows EC2 instance and your local physical machine.

The first method we will examine is pretty straightforward - we can simply use Remote Desktop to share the drives of our local machine with our remote machine. In the video below - I’m able to share my local file system and DVD with my Windows EC2 instance. This is particularly handy if I need to add Windows components to my instance; I can just pop my Windows installation CD into my local machine and access it remotely.

ElasticFox is a handy tool that provides exactly that. In the video below, we will illustrate how to configure this tool for Mac OS X.

ElasticFox is a handy tool that provides exactly that. In the video below, we will illustrate how to configure this tool for Window Server 2003.

In the code above, I’m calling the QueryByAttributes method with no query expression, so I will get back all of my items with all of their attributes.

My SimpleDb class has a number of delegates that you can implement - one for each of the SimpleDb methods that it implements - as listed below.

Well those are the highlights. Again, the code that I posted is not a complete implementation for the SimpleDb API. As I work on it more, I’ll post updates.

The ‘salt’ value that you use in the hash is your AWS secret. Amazon will calculate the same type of hash with your secret on its end - if the two values match, you are in business.

My code to generate the hash as follows.:

Once this signature is generated, it is just a matter of tacking it onto the query string and parsing the XML coming back as shown below.

Parsing the Response

Parsing Amazon’s response is just a matter of parsing the incoming XML. There are a few different ways of parsing XML on the iPhone. I chose to use their NSXMLParser, which does SAX-style XML parsing. Again, the scratchpad is a handy of having a look at what that XML looks like. Below is a snippet from a QueryWithAttributes response.

Building up the rest of the SimpleDb library is just a matter of parsing all of the various responses. I’ve posted my code at http://static.learnaws.com/SimpleDb_Archive.zip with the caveat that I did not implement the full SimpleDb API. I’m implementing as much as I need for the various iPhone applications that I’m building. As I flesh out this implementation, I’ll post updates.

Please see ‘Part 3’ for an example of how to use this wrapper…

If you look closely, I actually captured this screen cast on a Mac - the Windows ‘machine’ that I used is an EC2 instance running Windows Server 2003.

Mac OS X makes our job relatively straightforward since it has a terminal and a Java runtime already installed.

Please see below for the video.

The natural scalability that you get with AWS and the Internet-connectivity that the iPhone has makes the two an ideal pair.

That being the case, I was wondering what it would take to use Amazon Web Services in an iPhone application. The first service I decided to tackle was SimpleDb.

It’s really not that important for this post, but the context of my experiment is snow reports - it’s winter time in the Northwest so I thought I would use a SimpleDb domain as sort of a clearing house for snow reports from the various ski resorts in the area. My iPhone application would pull down data from this domain and display it in a table.

The decision I had was how to access SimpleDb. The Amazon Web Services all expose themselves through a combination of REST, Query and SOAP.

Accessing SimpleDb - SOAP or REST?

SOAP is a great option when you’re using .NET, Java or any other language that has a nice SOAP stack. The added advantage of SOAP is that you can use it with WS-Security, which means you can authenticate with a X.509 certificate, rather than passing around your secret and key.

But, the iPhone doesn’t really have a mature SOAP implementation. I did a little bit of searching and I found some experimental stuff, but nothing that I felt comfortable using. I’m pretty much a beginner with Objective-C and iPhone development.

I considered writing my own minimal SOAP stack, but that would involve creating the XML requests and parsing the XML response. Regardless of what method I use, I would have to parse the XML response. However, if I use REST, I am just putting together a URL request, so I can save myself 1/2 of the XML work. So, my choice was to use the SimpleDb REST API.

REST Authentication

This is actually where I ran into the most trouble. When using REST with Amazon Web Services, you have to pass along a authentication signature. Creating this signature is in concept well described by the Amazon documentation - it is just a matter of sorting your query parameters, capturing a time stamp and using your AWS secret as the key in a HMAC-SHA hash function.

The devil is in the details however. Let’s start from the beginning. The first thing I did was use a NSMutableDictionary to store my parameter values as you can see below.

Parameter Values

The snippet of code shown is actually for any generic SimpleDb call.

The parameters being added are required for all of the SimpleDb API calls.

After I’ve added these required parameters, I just need to add the parameters that are specific to the method that I’m calling.

Once that is done, to make the signature, the first step is to condense these parameters into a canonical form. The Amazon documentation does a more complete job of explaining this, but the general idea is that Amazon will take your parameters, your key, your time stamp and basically run the same hash algorithm. If the hashes are the same, then you are authenticated.

The canonical form is pretty straightforward, it is basically your parameters (both names and values) concatenated into a string. The parameters are in ascending order by their names first.

The NSMutableDictionary class has a nice method (or I guess message in Objective-C parlance) that does the sort for me - I can even indicate that I want a case insensitive comparison.

Getting this canonical string correct is absolutely crucial. If you make a mistake here, then your hash will be wrong and you will not be authenticated. I found what really helped was using the Amazon JavaScript ScratchPad for SimpleDb. This tool has a correct implementation of the signature creation algorithm, and there is an option to conveniently show the canonical string.

Having a reference implementation really helped to make sure that I was creating the right canonical string. You’ll notice that this scratch pad also has an option to see the signed URL. This helpful, but this value is not synchronized with the time stamp that was generated for the canonical string.

HMAC-SHA1 on the iPhone

Once I was able to generate canonical string, it was time to generate the hash value for it. This was actually the hardest part for me. Most languages like .NET, PHP, Java, etc have good encryption libraries, so generating this hash is trivial. On Mac OS X, I believe most people use the Open SSL library to do this type of work. However, Open SSL was not ported to the iPhone SDK. Apparently it was there in the early betas, but it was pulled out at some point.

I ended up making do with the CommonCrypto framework that is available for the iPhone.

Please see ‘Part 2’ for more details about this implementation…