The Back-to-Basics Readings of 2012 - All Things Distributed

All Things Distributed

The Back-to-Basics Readings of 2012. By Werner Vogels on 18 December 2012 10:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Permalink. Comments (). After the AWS re: Invent conference I spent two weeks in Europe for the last customer visits of the year. I have since returned and am now in New York City enjoying a few days of winding down the last activities of the year before spending the holidays here with family.

An Album for Each Year - 2012 Version - All Things Distributed

All Things Distributed

An Album for Each Year - 2012 Version. By Werner Vogels on 22 December 2012 06:00 PM. My list stopped in 2007, so now that 2012 is almost at its end it is a good moment to add the next 5 years to the list. 2012: Jack White, Blunderbuss. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Permalink. Comments (). About 5 years ago I joined a challenge to list "a favorite album for every year of your life."

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

NoSQL Data Modeling Techniques

Highly Scalable

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. This aspect of NoSQL is well-studied both in practice and theory because specific non-functional properties are often the main justification for NoSQL usage and fundamental results on distributed systems like the CAP theorem apply well to NoSQL systems.

2012 Favorites

Tim Kadlec

It’s the end of the year which means that “Best of 2012” articles are all the craze. Here, then, are the five most popular posts on this site from 2012. ## WTFWG. Far be it from I too abstain from such memes. Where-in I get unusually riled up and rant about the picture vs. srcset situation and how it was handled. The conversation has evolved, but I still have concerns about how parts of the process were handled. ## Media Query & Asset Downloading Results.

Media 40

Distributed Algorithms in NoSQL Databases

Highly Scalable

Scalability is one of the main drivers of the NoSQL movement. As such, it encompasses distributed system coordination, failover, resource management and many other capabilities. It sounds like a big umbrella, and it is. Although it can hardly be said that NoSQL movement brought fundamentally new techniques into distributed data processing, it triggered an avalanche of practical studies and real-life trials of different combinations of protocols and algorithms.

On BDConf 2012: Dallas

Tim Kadlec

It wasn’t what I was expecting, but it ended up being just what I needed. When one of the people who attended Breaking Development in Dallas , told me that on the last day of the event, I couldn’t help but smile. Single-track events are awesome, but they’re always a little nerve-wrecking as well. How do you balance code and design, pragmatic and conceptual? Each of those discussions has to happen to move the discussion forward, but balancing can be a challenge.

Probabilistic Data Structures for Web Analytics and Data Mining

Highly Scalable

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases.

Countdown to What is Next in AWS - All Things Distributed

All Things Distributed

By Werner Vogels on 13 January 2012 08:00 AM. Join me at 9AM PST on Wednesday January 18, 2012 to find out what is next in the AWS Cloud. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Countdown to What is Next in AWS. Permalink. Comments (). Registration required. Comments. blog comments powered by Disqus. Contact Info. Werner Vogels. CTO - Amazon.com. Other places.

AWS 78

What I Read in 2012

Tim Kadlec

Here, then, is 2012’s list: Designing Devices by Dan Saffer. For the third year in a row, my book count is trending in the wrong direction. In 2009, it was 38. In 2010, it was 33. In 2011, that number fell to 29 and this past year, that number plummeted to 21. Not only do I think reading is important, but it’s also something I enjoy a bunch and probably the closest thing I have to a hobby. However this year the reality is that I just didn’t set aside much time for it.

Games 40

MapReduce Patterns, Algorithms, and Use Cases

Highly Scalable

In this article I digested a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web or scientific articles. Several practical case studies are also provided. All descriptions and code snippets use the standard Hadoop’s MapReduce model with Mappers, Reduces, Combiners, Partitioners, and sorting. This framework is depicted in the figure below. MapReduce Framework. Basic MapReduce Patterns. Counting and Summing.

Amazon DynamoDB ? a Fast and Scalable NoSQL Database.

All Things Distributed

By Werner Vogels on 18 January 2012 07:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon DynamoDB â?? a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Permalink. Comments (). Today is a very exciting day as we release Amazon DynamoDB , a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications.

Driving Compute Cost Down for AWS Customers - All Things.

All Things Distributed

By Werner Vogels on 05 March 2012 02:01 PM. AWS today announced a substantial price drop from March 1, 2012 for many of the Amazon EC2, Amazon RDS, and Amazon ElastiCache instances types around the world. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Driving Compute Cost Down for AWS Customers. Permalink. Comments ().

AWS 76

Back-to-Basics Weekend Reading - Staged Event-Driven Architecture

All Things Distributed

By Werner Vogels on 17 August 2012 07:00 PM. I am in São Paolo, Brazil for the 2012 AWS Latin America Summit and for The Next Web Latin America conference. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Staged Event-Driven Architecture. Permalink. Comments (). I will also be visiting Chile and Mexico on this trip and have the great fortune to meet many of our Latin American AWS customers.

Expanding the Cloud ? Introducing Amazon CloudSearch - All.

All Things Distributed

By Werner Vogels on 12 April 2012 12:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Introducing Amazon CloudSearch. Permalink. Comments (). Today Amazon Web Services is introducing Amazon CloudSearch , a new web service that brings the power of the Amazon.comâ??s s search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale.

Cloud 120

Two Sessions: C++ Concurrency and Parallelism – 2012 State of the Art (and Standard)

Sutter's Mill

I’m going to put them together especially and only for C++ and Beyond 2012 , and I’ll be giving them nowhere else this year: C++ Concurrency – 2012 State of the Art (and Standard). C++ Parallelism – 2012 State of the Art (and Standard). At these sessions of C&B 2012, I expect anyone who wasn’t personally at the SG1 meeting this month, even world-class experts, will learn something new in these talks. I hope to see many of you at C&B 2012.

C++ 40

Amazon DynamoDB - From the Super Bowl to WeatherBug - All.

All Things Distributed

By Werner Vogels on 21 June 2012 09:00 AM. ve already passed our 2012 targets. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon DynamoDB - From the Super Bowl to WeatherBug. Permalink. Comments (). Amazon DynamoDB is the fastest growing new service in the history of AWS. In the five months since it launched in January, DynamoDB , our fast and scalable NoSQL database service, has been setting AWS growth records.

Games 103

Expanding The Cloud ? High Performance I/O Instances for Amazon.

All Things Distributed

By Werner Vogels on 19 July 2012 12:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding The Cloud â?? High Performance I/O Instances for Amazon EC2. Permalink. Comments (). AWS customers are bringing their most demanding workloads onto the cloud. These include the likes of high performance computation, for which we introduced the Cluster Compute and Cluster GPU instance types.

Cloud 100

Speeding Up Hadoop Builds Using Distributed Unit Tests

Highly Scalable

We recently worked with one of the Hadoop vendors on the continuous integration system for Hadoop core and other Hadoop-related projects like Pig, Hive, HBase. One of the challenges we faced was very slow automatic tests — full unit/integration test suite takes more than 2 hours for Hadoop core and more than 9 hours for Apache Pig. Although there are different ways to alleviate this problem (divide tests into suites, optimize tests by tweaking timeouts and sleeps, etc.),

Speed 100

Fast Intersection of Sorted Lists Using SSE Instructions

Highly Scalable

Intersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. At GridDynamics, we recently worked on a custom database for realtime web analytics where fast intersection of very large lists of IDs was a must for good performance.

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

All Things Distributed

By Werner Vogels on 21 August 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Managing Cold Storage with Amazon Glacier. Permalink. Comments (). Managing long-term digital archiving is a challenge for almost every company. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage.

Expanding the Cloud ? The Amazon Simple Workflow Service - All.

All Things Distributed

By Werner Vogels on 22 February 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? The Amazon Simple Workflow Service. Permalink. Comments (). Today AWS launched an exciting new service for developers: the Amazon Simple Workflow Service. Amazon SWF is an orchestration service for building scalable distributed applications.

Cloud 102

Customer Conversations - How Intuit and Edmodo Innovate using.

All Things Distributed

By Werner Vogels on 06 April 2012 06:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Customer Conversations - How Intuit and Edmodo Innovate using Amazon RDS. Permalink. Comments (). From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud. Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS).

My Best Christmas Present ? Root Domain Support for Amazon S3.

All Things Distributed

By Werner Vogels on 27 December 2012 12:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. My Best Christmas Present â?? Root Domain Support for Amazon S3 Website Hosting. Permalink. Comments (). I have been a big fan of the Amazon S3 Static Website Hosting feature since its launch and this blog happily is being served from it.

Hierarchical Navigation and Faceted Search on Top of Oracle Coherence

Highly Scalable

Some time ago I participated in design of a backend for one large online retailer company. From the business logic point of view, this was a pretty typical eCommerce service for hierarchical and faceted navigation, although not without peculiarities, but high performance requirements led us to the quite advanced architecture and technical design. In particular, we built this system on top of Oracle Coherence and designed our own data structures and indexes.

Tricks with Direct Memory Access in Java

Highly Scalable

Java was initially designed as a safe, managed environment. Nevertheless, Java HotSpot VM contains a “backdoor” that provides a number of low-level operations to manipulate memory and threads directly. This backdoor – sun.misc.Unsafe – is widely used by JDK itself in the packages like java.nio or java.util.concurrent. It is hard to imagine a Java developer who uses this backdoor in any regular development because this API is extremely dangerous, non portable, and volatile.

Java 101

Expanding the Cloud ? introducing the Asia Pacific (Sydney) Region.

All Things Distributed

By Werner Vogels on 12 November 2012 05:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? introducing the Asia Pacific (Sydney) Region. Permalink. Comments (). Today, Amazon Web Services has greater worldwide coverage with the launch of a new AWS Region in Sydney, Australia.

Expanding the Cloud ? Provisioned IOPS for Amazon RDS - All.

All Things Distributed

By Werner Vogels on 25 September 2012 06:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Provisioned IOPS for Amazon RDS. Permalink. Comments ().

Total Cost of Ownership and the Return on Agility - All Things.

All Things Distributed

By Werner Vogels on 16 August 2012 10:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Total Cost of Ownership and the Return on Agility. Permalink. Comments ().

AWS 84

Dynamic Content Support in Amazon CloudFront - All Things.

All Things Distributed

By Werner Vogels on 14 May 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Dynamic Content Support in Amazon CloudFront. Permalink. Comments (). In the past three and a half years, Amazon CloudFront has changed the content delivery landscape.

Cache 83

Expanding the Cloud for Windows Developers - All Things Distributed

All Things Distributed

By Werner Vogels on 08 May 2012 02:00 PM. Amazon RDS currently supports SQL Server 2008 R2 and plans to add support for SQL Server 2012 later this year. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud for Windows Developers. Permalink. Comments (). The software that powers todayâ??s s world of Internet services has become incredibly diverse. Todayâ??s

Driving Storage Costs Down for AWS Customers - All Things.

All Things Distributed

By Werner Vogels on 06 February 2012 09:00 PM. AWS today announced a substantial price drop per February 1, 2012 for Amazon S3 standard storage to help customers drive their storage cost down. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Permalink. Comments ().

Expanding the Cloud ? Introducing AWS Marketplace - All Things.

All Things Distributed

By Werner Vogels on 19 April 2012 02:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Introducing AWS Marketplace. Permalink. Comments (). Today Amazon Web Services launched AWS Marketplace , an online store that makes it easy for you to find, buy, and immediately start using software and services that run on the AWS Cloud. You can use AWS Marketplaceâ??s

AWS 82

Expanding Flexibility - Introducing the Reserved Instance Marketplace

All Things Distributed

By Werner Vogels on 12 September 2012 01:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding Flexibility - Introducing the Reserved Instance Marketplace. Permalink. Comments (). Today we launched a new feature that enables you to buy and sell Amazon EC2 Reserved Instances. Reserved Instances are an important pricing option for AWS customers to drive cost down.

The AWS Storage Gateway - All Things Distributed

All Things Distributed

By Werner Vogels on 23 January 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - The AWS Storage Gateway. Permalink. Comments (). Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? on-premises applications.

Back-to-Basics Weekend Reading - The 5 Minute Rule - All Things.

All Things Distributed

By Werner Vogels on 24 August 2012 04:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The 5 Minute Rule. Permalink. Comments (). I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile. The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 per GB/month.

Improving the Cloud - More Efficient Queuing with SQS - All Things.

All Things Distributed

By Werner Vogels on 08 November 2012 03:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Improving the Cloud - More Efficient Queuing with SQS. Permalink. Comments (). The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that just works.

Back-to-Basics Weekend Reading - Leases

All Things Distributed

I was in Los Angeles this week for the Digital Media on AWS Summit and to visit many of the studios and production houses that are using AWS for production and post-production work.

AWS 78

The Andrew File System - All Things Distributed

All Things Distributed

By Werner Vogels on 17 November 2012 11:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The Andrew File System. Permalink. Comments (). This weekend I am heading to Brussels for meetings with the European Commission, specifically with Vice-president Neelie Kroes who owns the Digital Agenda for the EU, about how to accelerate cloud usage in both business and government in Europe.

Back-to-Basics Weekend Reading - Virtualizing Operating Systems.

All Things Distributed

By Werner Vogels on 20 July 2012 12:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Virtualizing Operating Systems. Permalink. Comments (). This weekends back-to-basics reading is on operating system virtualization. There are two papers that deserve the "classic" tag as they both form the basis for operating system virtualization that is in production today.

Back-to-Basics Weekend Reading - Hints for Computer Systems.

All Things Distributed

By Werner Vogels on 13 July 2012 09:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Hints for Computer Systems Design. Permalink. Comments (). For a while now I have been on a track to read one influential/fundamental Computer Science paper each weekend.

Register for AWS re: Invent - All Things Distributed

All Things Distributed

By Werner Vogels on 16 July 2012 09:00 AM. General registration opens up on July 25, 2012, however active AWS customers will get a chance to jump the line and start registering on July 20. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Register for AWS re: Invent. Permalink. Comments (). The first annual AWS user and partner conference will be held November 27-29 at The Venetian in Las Vega.

AWS 64

Back-to-Basics Weekend Reading - Weighted Voting for Replicated.

All Things Distributed

By Werner Vogels on 02 November 2012 11:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Weighted Voting for Replicated Data. Permalink. Comments (). The last two weeks in Europe and Israel (The image above is from Tel Aviv) were intense so I didnt get to do much reading, hence I didnt post any reading suggestions.

Back-to-Basics Weekend Reading - Load Sharing - All Things.

All Things Distributed

By Werner Vogels on 07 September 2012 03:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Load Sharing. Permalink. Comments (). One of the main reasons for picking some of these older papers as back-to-basics reading is that the first researchers on a topic had to develop the fundamental models and principles from scratch.