SquareCows.com

Simple Workflow
Today we are introducing the Amazon Simple Workflow service, SWF for short. This new service gives you the ability to build and run distributed, fault-tolerant applications that span multiple systems (cloud-based, on-premise, or both). Amazon Simple Workflow coordinates the flow of synchronous or asynchronous tasks (logical application steps) so that you can focus on your business and your application instead of having to worry about the infrastructure.

Why?
We want to make it easier for you to build distributed, fault-tolerant, cloud-based applications! In our own work with systems of this type, we have learned quite a bit. For example:

1. The applications often incorporate a workflow — A series of steps that must take place in a predefined order, with opportunities to adjust the workflow as needed by making decisions and by handling special cases in a structured fashion.
2. The workflow often represents a business process – Think about all of the steps involved in processing an order on your favorite e-commerce site. Charging your credit card, updating your order history, arranging for the items to be shipped, shipping the items, tracking the shipment, replenishing inventory, handling returns, and much more.
3. Processes can be complex – Years ago, I was told that a single Amazon.com order needed to make its way through at least 40 different states or steps before it was considered complete. I am sure that the process has become even more complex over time.
4. Flexibility is key – Earlier attempts to specify and codify a workflow in declarative form have proven to be rigid and inflexible. At some point, procedural code becomes a necessity.
5. Ease of use is important – It should be possible to design and implement these applications without spending a lot of time acquiring specialized skills.

You can use Simple Workflow to handle many types of multi-stage operations including traditional business processes (handling an order or adding a new employee), setting up a complex multi-tiered application, or even handling the decision-making process for a multi-player online game.

Some Definitions

Let's start by defining a couple of terms:

• A Workflow is the automation of a business process.
• A Domain is a collection of related Workflows.
• Actions are the individual tasks undertaken to carry out a Workflow.
• Activity Workers are the pieces of code that actually implement the tasks. Each kind of Worker has its own Activity Type.
• A Decider implements a Workflow's coordination logic.

Let's say that we have an image processing workflow, and that it has the following tasks:

1. Accept uploaded file.
2. Store file in Amazon S3.
3. Validate file format and size.
4. Use Amazon Mechanical Turk to classify the image.
5. If the image is unacceptable, send an error message using Amazon SES and terminate the workflow.
6. If the image is acceptable, check the user's balance in the accounting system.
7. Launch an EC2 instance.
8. Wait for the EC2 instance to be ready, and then configure it (keys, packages, and so forth).
9. Convert the image to PNG format and generate a series of image thumbnails.
10. Upload the PNG image and the thumbnails to Amazon S3.
11. Adjust the user's balance in the accounting system.
12. Create an entry in the appropriate database table.
13. Send a status message to the user, again using Amazon SES.

Tasks 1 through 12 make up the Workflow. Each of the tasks is an Action. The code to implement each action is embodied in a specific Activity Worker.

The workflow's Decider is used to control the flow of execution from task to task. In this case, the Decider would make decisions based on the results of steps 4 (Mechanical Turk) and 6 (balance check). There's a nice, clean separation between the work to be done and the steps needed to do it.

Here's a picture:

What Does Simple Workflow Do?
Simple Workflow provides you with the infrastructure that you need to implement workflows such as the one above. It does all of the following (and a lot more):

• Stores metadata about a Workflow and its component parts.
• Stores tasks for Workers and queue them until a Worker needs them.
• Assigns tasks to Workers.
• Routes information between executions of a Workflow and the associated Workers.
• Tracks the progress of Workers on Tasks, with configurable timeouts.
• Maintains workflow state in a durable fashion.

Because the Workers and Deciders are both stateless, you can respond to increased traffic by simply adding additional Workers and Deciders as needed. This could be done using the Auto Scaling service for applications that are running on Amazon EC2 instances the AWS cloud.

Your Workers and your Deciders can be written in the programming language of your choice, and they can run in the cloud (e.g. on an Amazon EC2 instance), in your data center, or even on your desktop. You need only poll for work, handle it, and return the results to Simple Workflow. In other words, your code can run anywhere, as long as it can  "see" the Amazon Simple Workflow HTTPS endpoint. This gives you the flexibility to incorporate existing on-premise systems into new, cloud-based workflows. Simple Workflow lets you do "long polling" to reduce network traffic and unnecessary processing within your code. With this model, requests from your code will be held open for up to 60 seconds if necessary.

Inside the Decider
Your Decider code simply polls Simple Workflow asking for decisions to be made, and then decides on the next step. Your code has access to all of the information it needs to make a decision including the type of the workflow and a detailed history of the prior steps taken in the workflow. The Decider can also annotate the workflow with additional data.

Inside the Workers
Your Worker code also polls Simple Workflow, in effect asking for work that needs to be done. It always polls with respect to one or more Task Lists, so that one Worker can participate in multiple types of Workflows if desired. It pulls work from Task Lists, does the work, updates the Workflow's status, and goes on to the next task. In situations that involve long-running tasks, the worker can provide a "heartbeat" update to Simple Workflow. Deciders can insert Markers into the execution history of a Workflow for checkpointing or auditing purposes.

Timeouts, Signals, and Errors
Simple Workflow's Timeouts are used to ensure that an execution of a Workflow runs correctly. They can be set as needed for each type of Workflow. You have control over the following timeouts:

• Workflow Start to Close – how long an execution can take to complete.
• Decision Task Start to Close – How long a Decider can take to complete a decision task.
• Activity Task Start to Close – How long an Activity Worker can take to process a task of a given Activity Type.
• Activity Task Heartbeat – How long an Activity Worker can run without providing its status to Simple Workflow.
• Activity Task Schedule to Start – How long Simple Workflow waits before timing out a task if no workers are available to perform the task.
• Activity Task Schedule to Close – How long Simple Workflow will wait between the time a task is scheduled to the time that it is complete.

Signals are used to provide out-of-band information to an execution of a Workflow. You could use a signal to cancel a Workflow, tell it that necessary data is ready, or to provide information about an emergent situation. Each Signal is added to the Workflow's execution history and the Workflow's Decider controls what happens next.

Simple Workflow will make sure that execution of each Workflow proceeds as planned, using the Timeouts mentioned above to keep the Workflow from getting stuck if a task takes too long or if there is no activity code running.

Getting Started
Here's what you need to do to get started with Amazon Simple Workflow:

1. Write your Worker(s) using any programming language.
2. Write your Decider, again in any programming language.
3. Register the Workflow and the Activities.
4. Run the Workers and the Decider on any host that can "see" the Simple Workflow endpoint.
5. Initiate execution of a Workflow.
6. Monitor progress of the Workflow using the AWS Management Console.

The AWS Flow Framework
In order to make it even easier for you to get started with Amazon Simple Workflow, the AWS SDK for Java now includes the new AWS Flow Framework. This new framework includes a number of programming constructs that abstract out a number of task coordination details. For example, it uses a programming model based on Futures to handle dependencies between tasks. Initiating a Worker task is as easy as making a method call, and the framework takes care of the Workers and the Decision Tasks behind the scenes.

Simple Workflow API Basics
You can build your Workflow, your Workers, and your Decider, with just a handful of Simple Workflow APIs. Here's what you need to know to get started:

• Workflow Registration – The RegisterDomain, RegisterWorkflowType, and RegisterActivityType calls are used to register the various components that make up a Workflow.
• Implementing Deciders and Workers - The PollForDecisionTask call is used to fetch decision tasks, and the RespondDecisionTaskCompleted call is used to signal that a decision task has been completed. Similarly, the PollForActivityTask call is used to fetch activity tasks and the RespondActivityTaskCompleted call is used to signal that an activity task is complete.
• Starting a Workflow – The StartWorkflowExecution call is used to get a Workflow started.

The Amazon Simple Workflow API Reference contains information about these and other APIs.

 AWS Management Console Support
The AWS Management Console includes full support for Amazon Simple Workflow. Here's a tour, starting with the main (dashboard) page:

Register a workflow domain

Register a workflow for a workflow domain

Register an activity within a workflow:

Initiate execution of a workflow:

Provide input data to an execution:

See all of the currently executions of a given workflow:

Pricing
Like all of the services in the AWS Cloud, Amazon Simple Workflow is priced on an economical, pay-as-you-go basis. First, all AWS customers can get started for free. You can initiate execution of 1,000 Workflows and 10,000 tasks per month and you can keep them running for a total of 30,000 workflow-days (one workflow active for one day is equal to one workflow-day).

Beyond that, there are three pricing dimensions:

• Executions – You pay $0.0001 for every Workflow execution, and an additional $0.000005 per day if they remain active for more than 24 hours.
• Tasks, Signals, and Markers – You pay $0.000025 for every task execution, timer, signal, and marker.
• Bandwidth - You pay $0.10 for the first Gigabyte of data transferred in. There is no charge for the first Gigabyte of data transferred out, and the usual tiered AWS charges after that.

Amazon Simple Workflow in Action
Here are some ways that people are already putting Amazon Simple Workflow to use:

• RightScale is using it to ensure fault-tolerant execution of their server scaling workflow. Read Thorsten von Eicken's post, RightScale Server Orchestration and Amazon SWF Launch, for more information.
• NASA uses Simple Workflow to coordinate the daily image processing tasks for the Mars Exploration Rovers. Read our new case study, NASA JP and Amazon SWF, to see how they do it.
• Sage Bionetworks coordinates complex, heterogeneous scientific workflows. Check out the new case study, Sage Bionetworks and Amazon SWF, for complete information.

Go With the Flow
I am very interested in hearing about the applications and Workflows that you implement with Amazon Simple Workflow. Please feel free to leave a comment or to send me some email.

– Jeff;

Esri is a leading provider of Geographic Information Systems (GIS) software and geo-database management applications. Their powerful mapping solutions have been used by Governments, industry, academics, and NGOs for nearly 30 years (read their history to learn more).

Over the years, they have ported their code from mainframes to minicomputers (anyone else remember DEC and Prime?), and then to personal computers. Now they are moving their applications to the AWS cloud to provide their customers with applications that can be launched quickly and then scaled as needed, all at a cost savings when compared to traditional on-premise hosting.

Watch this video to learn more about this potent combination:

We will be participating in the Esri Federal GIS Conference in Washington, DC this month; please visit the AWS Federal page for more information about this and other events.

On that page you will also find case studies from several AWS users in the Federal Government including the Recovery Accountability and Transparency Board, the US Department of Treasury, the DOE's National Renewable Energy Laboratory, the US Department of State, the US Department of Agriculture, the NASA Jet Propulsion Laboratory, and the European Space Agency.

On March 21, we will run a free Esri on the Cloud webinar at noon EST. Attend the webinar to learn how to use AWS to process GIS jobs faster and at a lower cost than an on-premise solution.

– Jeff;

It’s always exciting to find out that an app that has changed how I consume news and blog content on my mobile devices is using AWS to power some of their most engaging features. Such is the case with Pulse, a visual news reading app for iPhone, iPad and Android. Pulse uses Amazon Elastic MapReduce, our hosted Hadoop product, to analyze data from over 11 million users and to deliver the best news stories from a variety of different content publishers. Born out of a Stanford launchpad class and awarded for its elegant design by Apple at WWDC 2011, the Pulse app blends a strong high-tech backend with great visual appeal to conquer the eyes of mobile news readers everywhere.

Pulse backend team members from left to right: Simon, Lili, Greg, Leonard

The December 2011 update included a new feature called Smart Dock, which uses Hadoop and a tool called mrjob, developed by Yelp, to analyze users’ reading preferences and continuously recommend other articles or sources they might enjoy.

To understand the level of engineering that goes behind such rich customer features, I spoke to Greg Bayer, Backend Engineering Lead at Pulse:

How big is the “big data” that Pulse analyzes every day? 

Our application relies on accurately analyzing client event logs (as opposed to web logs) to extract trends and enable other rich features for our users. To give you a sense of the scale at which we run these analyses, we literally go through millions of events per hour, which translates to as many as 250+ Amazon Elastic MapReduce nodes on any given day. Since we are dealing with event logs, generated by our users from the various platforms on which they access our app (Android, iPhone, iPad, etc.), our logs grow in proportion to our user base. For example, the recent influx of new users from Kindle Fire (Android) means we now have a lot more logs coming in from those devices.  Also, since the logs are big, we’ve found that it is very efficient to write them to disk as fast as possible – directly from devices to Amazon EC2 (see my tandem article on the logging architecture we use and the graph below, which highlights some of our numbers).

For more Pulse numbers, checkout the full infographic.

Powering Rich Features for Our Users

Much of our backend is built on industry standard systems such as Hadoop. The innovation happens in how we leverage these systems to create value. For us, it’s all about how we can make the app more fun to use and provide rich features that our users will love. For techies, you can read about many of these features in the backend section of the Pulse engineering blog and learn about all the details.

The Right Choice for Big Data

I joined the team here pretty early on as the first backend engineer. I came to Pulse after working at Sandia National Labs, where I built and managed an in-house 70-node Hadoop cluster. This was an investment of over $100,000, operational support, and over 6 months time to get it fully fine-tuned. Needless to say, I was fully aware of the cost and resources needed to run something at the scale that Pulse would need to accommodate.

AWS was and still is the only feasible solution for us. I love the flexibility to quickly stand up a cluster of hundreds of nodes and the added flexibility of choosing the pricing scheme that’s needed for a job. If I need a job done faster, I can always spin up a very large cluster and get results in minutes, or take advantage of smaller instances and the spot marketplace for Amazon Elastic MapReduce if I’m looking to complete a job that’s not time-sensitive. Since an Amazon Elastic MapReduce cluster can simply be turned off when we are done, the cost to run big queries is usually quite reasonable. Consider a cluster of 100 m1.large machines: a set of queries that takes 45 minutes to run on this cluster could cost us approximately $11 – $34 (depending on whether we bid on spot instances or use regular on-demand instances).

Lessons Learned (the bold fomatting below is our doing )

It is important to consider the trade-offs and choose the right tool for the job. In our experience, AWS provides an exceptional capability to build systems as close to the metal as you like, while still avoiding the burden and inelasticity of owning your own hardware. It also provides some useful abstraction layers and services above the machine level.

By allowing virtual machines (Amazon EC2 instances) to be provisioned quickly and inexpensively, a small engineering team can stay more focused on the development of key product features. Since stopping and starting these instances is painless, it’s easy to quickly adapt to changing engineering or needs — perhaps scaling up to support 10x more users or shutting down a feature after pivoting a business model.

AWS also provides many other useful services that help save engineering time. Many standard systems, such as load balancers or Hadoop clusters, that normally require significant time and specialized knowledge to deploy, can be deployed automatically on Amazon EC2 for almost no setup or maintenance cost.

Simple, but powerful services like Amazon S3 and the newly released Amazon DynamoDB make building complex features on AWS even easier. Because bandwidth is fast and free between all AWS services, plugging together several of these services is a great way to bootstrap a scalable infrastructure.

Thanks for your time, Greg & best of luck to the Pulse team! 

-rodica

Related: Pulse Engineering – Scaling to 10M on AWS

Earlier today, GigaOM published a cost comparison of self-hosting vs. hosting on AWS. I wanted to bring to your attention a few quick issues that we saw with this analysis:

Lower Costs in Other AWS Regions – The comparison used the AWS costs for the US West (San Francisco) Region, ignoring the fact that EC2 pricing in the US East (Northern Virginia) and US West (Oregon) is lower ($0.76 vs. $0.68 for On-Demand Extra Large Instances).

Three Year Reserved Instances – The comparison used one year Reserved Instances, but a three year amortization schedule for the self-hosted hardware. You save over 22% by using three year Reserved Instances instead of one year Reserved Instances, and the comparison is closer to apples-to-apples.

High Utilization Reserved Instances – The comparison used a combination of Medium Utilization Reserved Instances and On-Demand Instances. Given the predictable traffic pattern in the original post, a blend of High and Low Utilization Reserved Instances would reduce your costs, and still give you the flexibility to easily scale up and scale down that you don't get with traditional hosting.

Load Balancer (and other Networking) Costs - The self-hosted column does not include the cost of a redundant set of load balancers. They also need top-of-rack switches (to handle what is probably 5 racks worth of servers) and a router.

No Administrative Costs – Although the self-hosted model specifically excludes maintenance and administrative costs, it is not reasonable to assume that none of the self-hosted hardware will fail in the course of the three year period. It is also dangerous to assume that labor costs will be the same in both cases, as labor can be a significant expense when you are self-hosting.

Data Transfer Costs – The self-hosted example assumes a commit of over 4 Gbps of bandwidth capacity. If you have ever contracted for bandwidth & connectivity at this scale, you undoubtedly know that you must actually commit to a certain amount of data transfer, and that your costs will change significantly if you are over or under your commitment.

We did our own calculations taking in to account only the first four issues listed above and came up with a monthly cost for AWS of $56,043 (vs. the $70,854 quoted in the article). Obviously each workload differs based on the nature of what resources are utilized most.

These analyses are always tricky to do and you always need to make apples-to-apples cost comparisons and the benefits associated with each approach. We're always happy to work with those wanting to get in to the details of these analyses; we continue to focus on lowering infrastructure costs and we're far from being done.

– Jeff;

We launched Amazon DynamoDB with a live video feed that was very well-attended. If you didn't get a chance to watch at the time, you can do so now:

You may also enjoy this short video featuring Werner Vogels, James Hamilton, Dave Lang, and Swami Sivasubramanian:

The Introducing Amazon DynamoDB Webinar will take place on February 15th at 10:00 AM PST. Stefano Stefani (Principal Software Engineer) and Dave Lang (Product Manager) will be presenting and fielding questions from the audience.The webinar is free but you do need to sign up.

– Jeff;

As you can tell from my recent post on Amazon S3 Growth for 2011, our customers are uploading new objects to Amazon S3 at an incredible rate. We continue to innovate on your behalf to drive down storage costs and pass along the resultant savings to you at every possible opportunity. We are now happy (some would even say excited) to announce another in a series of price reductions.

With this price change, all Amazon S3 standard storage customers will see a significant reduction in their storage costs. For instance, if you store 50 TB of data on average you will see a 12% reduction in your storage costs, and if you store 500 TB of data on average you will see a 13.5% reduction in your storage costs.

Effective February 1, 2012, the following prices are in effect for Amazon S3 standard storage in the US Standard region:

The prices for all of the other regions are listed on the Amazon S3 Pricing page. For the AWS GovCloud region, the new lower prices can be found on the AWS GovCloud Pricing page.

There’s been a lot written lately about storage availability and prices. We’ve often talked about the benefits that AWS’s scale and focus creates for our customers. Our ability to lower prices again now is an example of this principle at work.

It might be useful for you to remember that an added advantage of using a cloud storage service such as Amazon S3 over using your own on-premise storage is that with cloud storage, the price reductions that we regularly roll out apply not only to any new storage that you might add but also to the existing storage that you have. This could amount to considerable financial savings for many of you.

– Jeff;

Looking Back
In 2011 we added a total of seven edge locations to Amazon CloudFront and Route 53. We also added lots of new features, as I documented last year.

Looking Forward
Our newest edge locations are located in Milan, Italy and Osaka, Japan. This brings our total worldwide location count to 26 (see the CloudFront page for a complete list). Each new edge location helps lower latency and improves performance for your end users.

Making Plans
We have additional locations in the pipeline for 2012 and beyond. Our planning process takes a number of factors in to account including notes from our sales team and discussions on the Amazon CloudFront forum. We also collect latency measurements from a number of points around the globe to our current set of locations and correlate them with broadband Internet penetration and existing Amazon CloudFront usage in the area.

I would also like to invite you to participate in the Amazon CloudFront Edge Location Survey. We are very interested in your suggestions for additional locations. We'd also like to learn a bit more about the type of content that you deliver to your customers.

All Aboard
The CloudFront team is hiring. We need some Software Development Engineers, a Senior Systems Engineer,a Senior Software Development Manager,  a Product Manager, and a Business Development Representative.

– Jeff;

Today's guest blogger is Adam Gray. Adam is a Product Manager on the Elastic MapReduce Team.

– Jeff;

We’re always excited when we can bring features to our customers that make it easier for them to derive value from their data—so it’s been a fun month for the EMR team. Here is a sampling of the things we’ve been working on.

Free CloudWatch Metrics
Starting today customers can view graphs of 23 job flow metrics within the EMR Console by selecting the Monitoring tab in the Job Flow Details page. These metrics are pushed CloudWatch every five minutes at no cost to you and include information on:

• Job flow progress including metrics on the number of map and reduce tasks running and remaining in your job flow and the number of bytes read and written to S3 and HDFS.
• Job flow contention including metrics on HDFS utilization, map and reduce slots open, jobs running, and the ratio between map tasks remaining and map slots.
• Job flow health including metrics on whether your job flow is idle, if there are missing data blocks, and if there are any dead nodes.

Please watch this video to see how to view CloudWatch graphs in the EMR Console:

You can also learn more from the Viewing CloudWatch Metrics section of the EMR Developer Guide.

You can view the new metrics in the AWS Management Console:

Further, through the CloudWatch Console, API, or SDK you can set alarms to be notified via SNS if any of these metrics go outside of specified thresholds. For example, you can receive an email notification whenever a job flow is idle for more than 30 minutes, HDFS Utilization goes above 80%, or there are five times as many remaining map tasks as there are map slots, indicating that you may want to expand your cluster size.

Please watch this video to see how to set EMR alarms through the CloudWatch Console:

Hadoop 0.20.205, Pig 0.9.1, and AMI Versioning
EMR now supports running your job flows using Hadoop 0.20.205 and Pig 0.9.1. To simplify the upgrade process, we have also introduced the concept of AMI versions. You can now provide a specific AMI version to use at job flow launch or specify that you would like to use our “latest” AMI, ensuring that you are always using our most up-to-date features. The following AMI versions are now available:

• Version 2.0.x: Hadoop 0.20.205, Hive 0.7.1, Pig 0.9.1, Debian 6.0.2 (Squeeze)
• Version 1.0.x: Hadoop 0.18.3 and 0.20.2, Hive 0.5 and 0.7.1, Pig 0.3 and 0.6, Debian 5.0 (Lenny)

You can specify an AMI version when launching a job flow in the Ruby CLI using the –ami-version argument (note that you will have to download the latest version of the Ruby CLI):

Please visit the AMI Versioning section of the Elastic MapReduce Developer Guide for more information.

S3DistCp for Efficient Copy between S3 and HDFS
We have also made available S3DistCp, an extension of the open source Apache DistCp tool for distributed data copy, that has been optimized to work with Amazon S3. Using S3DistCp, you can efficiently copy large amounts of data between Amazon S3 and HDFS on your Amazon EMR job flow or copy files between Amazon S3 buckets. During data copy you can also optimize your files for Hadoop processing. This includes modifying compression schemes, concatenating small files, and creating partitions.

For example, you can load Amazon CloudFront logs from S3 into HDFS for processing while simultaneously modifying the compression format from Gzip (the Amazon CloudFront default) to LZO and combining all the logs for a given hour into a single file. As Hadoop jobs are more efficient processing a few, large, LZO-compressed files than processing many, small, Gzip-compressed files, this can improve performance significantly.

Please see Distributed Copy Using S3DistCp in the Amazon Elastic MapReduce documentation for more details and code examples.

cc2.8xlarge Support
Amazon Elastic MapReduce also now supports the new Amazon EC2 Cluster Compute instance, Cluster Compute Eight Extra Large (cc2.8xlarge). Like other Cluster Compute instances, cc2.8xlarge instances are optimized for high performance computing, giving customers very high CPU capabilities and the ability to launch instances within a high bandwidth, low latency, full bisection bandwidth network. cc2.8xlarge instances provide customers with more than 2.5 times the CPU performance of the first Cluster Compute instance (cc1.4xlarge) instance, more memory, and more local storage at a very compelling cost. Please visit the Instance Types section of the Amazon Elastic MapReduce detail page for more details.

In addition, we are pleased to announce an 18% reduction in Amazon Elastic MapReduce pricing for cc1.4xlarge instances, dropping the total per hour cost to $1.57. Please visit the Amazon Elastic MapReduce Pricing Page for more details.

VPC Support
Finally, we are excited to announce support for running job flows in an Amazon Virtual Private Cloud (Amazon VPC), making it easier for customers to:

• Process sensitive data – Launching a job flow on Amazon VPC is similar to launching the job flow on a private network and provides additional tools, such as routing tables and Network ACLs, for defining who has access to the network. If you are processing sensitive data in your job flow, you may find these additional access control tools useful.
• Access resources on an internal network – If your data is located on a private network, it may be impractical or undesirable to regularly upload that data into AWS for import into Amazon Elastic MapReduce, either because of the volume of data or because of its sensitive nature. Now you can launch your job flow on an Amazon VPC and connect to your data center directly through a VPN connection.

You can launch Amazon Elastic MapReduce job flows into your VPC through the Ruby CLI by using the –subnet argument and specifying the subnet address (note that you will have to download the latest version of the Ruby CLI):

Please visit the Running Job Flows on an Amazon VPC section in the Elastic MapReduce Developer Guide for more information.

– Adam Gray, Product Manager, Amazon Elastic MapReduce.

As of the end of 2011, there are 762 billion (762,000,000,000) objects in Amazon S3. We process over 500,000 requests per second for these objects at peak times.

Here's the annual growth chart:

This represents year-over-year growth of 192%; S3 grew faster last year than it did in any year since it launched in 2006.

Where are all of these objects coming from? Although we definitely made it easier for you to delete objects using Multi-Object Deletion and Object Expiration, we also gave you plenty of ways to upload new objects using Multipart upload, AWS Direct Connect, and AWS Import/Export.

As you can imagine, building, running, and adding new features to a system as large and as complex as S3 is no simple task. Here are some of the open positions on the S3 team:

• Software Development Engineer
• Senior Software Development Engineer
• Systems Engineer
• Senior Product Manager
• Director

– Jeff;

We have added two new benefits to the Gold and Platinum levels of AWS Premium Support. The following features are now in beta testing:

• We now offer third-party support for popular operating systems running on Amazon EC2. We also support a number of pieces of system software.
• The AWS Trusted Advisor monitors your use of AWS and recommends configuration changes and new services that may help save you money, improve system performance, and close security gaps.

Third-Party Support
If you have Gold or  Platinum Premium Support, you can now ask questions related to a number of popular operating systems including Microsoft Windows, Ubuntu, Red Hat Linux, SuSE Linux, and the Amazon Linux AMI. You can ask us about system software including the Apache and IIS web servers, the Amazon SDKs, Sendmail, Postfix, and FTP. A team of AWS support engineers is ready to help with setup, configuration, and troubleshooting of these important infrastructure components.

We are also enabling the use of desktop sharing software, giving you the option to share your desktop with a support engineer as needed.

AWS Trusted Advisor
AWS Trusted Advisor draws upon best practices learned from AWS’ aggregated operational history of serving hundreds of thousands of AWS customers. The AWS Trusted Advisor inspects your AWS environment and makes recommendations when opportunities exist  to save money, improve system performance, or close security gaps. The initial release of the AWS Trusted Advisor includes eight separate checks; we'll be adding more throughout 2012.

The checks are grouped into three families: fault tolerance checks, security audits, and cost optimizations. Here is the initial set of eight checks performed by AWS Trusted Advisor:

1. Security Group – Open Ports – This check inspects your security groups and classifies each open port into one of three categories. Green ports for common protocols such as SSH and HTTP, Red ports for protocols that don't usually need to be open on internet-facing servers (e.g. port 1443 for Microsoft SQL Server), and Yellow for all others.
2. Security Group – CIDR Rules – This check inspects your security groups for rules that have errors which might allow more access than may be intended. Some people (me included) often confuse "/0"and "/32" addresses.
3. Reserved Instance Recommendations - This check looks at your billing and instance utilization history and recommends optimizations that could be achieved by the purchase of Reserved Instances.
4. Unused Elastic IP Addresses – Elastic IP Addresses that are not attached to an Amazon EC2 instance will be flagged since you pay for them if you don't use them.
5. EBS Snapshots – This check looks for EBS volumes that don't have a snapshot, or which have only aged snapshots. The Red/Yellow/Green model is also used here: Red if there is no snapshot at all or if the most recent one is very old; Yellow if the most recent snapshot is somewhat old, and Green if the most recent snapshot is reasonably recent (we're still fine tuning the thresholds for these checks).
6. Amazon EC2 Availability Zone Balance – This check identifies situations where Amazon EC2 instances are not evenly distributed across Availability Zones, or if (even worse) they are all in the same Availability Zone. The Red/Yellow/Green model is used to characterize the situation.
7. Elastic Load Balancer Optimization – This check determines whether instance allocation across Availability Zones for each Load Balancer is balanced.
8. Service Limits – This check gives you visibility into the per-account limits and usage of things like instances, Elastic IP addresses, and other resources (in almost every case, limits can be raised using the appropriate online form).

All of the checks are made using a constrained subset of the existing set of documented AWS API calls. The AWS Trusted Advisor does not have access to your data.

Here's a diagram to show you how it works:

Advice from the AWS Trusted Advisor is made available in several different forms. For certain issues, we will proactively create support cases and notify you that a given check has identified an opportunity for improvement. The AWS Support Engineers are also available to review AWS Trusted Advisor recommendations any time you call in for support. In the future a regular scorecard report will be available, as will an AWS Trusted Advisor Console with support for viewing, running, customizing, and even opting out of certain checks as desired.

These new features are available for all Gold and Platinum customers. What do you think? Leave a comment and let me know.

– Jeff;