The Back-to-Basics Readings of 2012 - All Things Distributed

All Things Distributed

The Back-to-Basics Readings of 2012. By Werner Vogels on 18 December 2012 10:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Permalink. Comments (). After the AWS re: Invent conference I spent two weeks in Europe for the last customer visits of the year. I have since returned and am now in New York City enjoying a few days of winding down the last activities of the year before spending the holidays here with family.

NoSQL Data Modeling Techniques

Highly Scalable

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. This aspect of NoSQL is well-studied both in practice and theory because specific non-functional properties are often the main justification for NoSQL usage and fundamental results on distributed systems like the CAP theorem apply well to NoSQL systems.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

An Album for Each Year - 2012 Version - All Things Distributed

All Things Distributed

An Album for Each Year - 2012 Version. By Werner Vogels on 22 December 2012 06:00 PM. My list stopped in 2007, so now that 2012 is almost at its end it is a good moment to add the next 5 years to the list. 2012: Jack White, Blunderbuss. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Permalink. Comments (). About 5 years ago I joined a challenge to list "a favorite album for every year of your life."

2012 Favorites

Tim Kadlec

It’s the end of the year which means that “Best of 2012” articles are all the craze. Here, then, are the five most popular posts on this site from 2012. ## WTFWG. Far be it from I too abstain from such memes. Where-in I get unusually riled up and rant about the picture vs. srcset situation and how it was handled. The conversation has evolved, but I still have concerns about how parts of the process were handled. ## Media Query & Asset Downloading Results.

Media 40

Distributed Algorithms in NoSQL Databases

Highly Scalable

Scalability is one of the main drivers of the NoSQL movement. As such, it encompasses distributed system coordination, failover, resource management and many other capabilities. It sounds like a big umbrella, and it is. Although it can hardly be said that NoSQL movement brought fundamentally new techniques into distributed data processing, it triggered an avalanche of practical studies and real-life trials of different combinations of protocols and algorithms.

Probabilistic Data Structures for Web Analytics and Data Mining

Highly Scalable

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases.

On BDConf 2012: Dallas

Tim Kadlec

It wasn’t what I was expecting, but it ended up being just what I needed. When one of the people who attended Breaking Development in Dallas , told me that on the last day of the event, I couldn’t help but smile. Single-track events are awesome, but they’re always a little nerve-wrecking as well. How do you balance code and design, pragmatic and conceptual? Each of those discussions has to happen to move the discussion forward, but balancing can be a challenge.

MapReduce Patterns, Algorithms, and Use Cases

Highly Scalable

In this article I digested a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web or scientific articles. Several practical case studies are also provided. All descriptions and code snippets use the standard Hadoop’s MapReduce model with Mappers, Reduces, Combiners, Partitioners, and sorting. This framework is depicted in the figure below. MapReduce Framework. Basic MapReduce Patterns. Counting and Summing.

Two Sessions: C++ Concurrency and Parallelism – 2012 State of the Art (and Standard)

Sutter's Mill

I’m going to put them together especially and only for C++ and Beyond 2012 , and I’ll be giving them nowhere else this year: C++ Concurrency – 2012 State of the Art (and Standard). C++ Parallelism – 2012 State of the Art (and Standard). At these sessions of C&B 2012, I expect anyone who wasn’t personally at the SG1 meeting this month, even world-class experts, will learn something new in these talks. I hope to see many of you at C&B 2012.

C++ 40

Countdown to What is Next in AWS - All Things Distributed

All Things Distributed

By Werner Vogels on 13 January 2012 08:00 AM. Join me at 9AM PST on Wednesday January 18, 2012 to find out what is next in the AWS Cloud. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Countdown to What is Next in AWS. Permalink. Comments (). Registration required. Comments. blog comments powered by Disqus. Contact Info. Werner Vogels. CTO - Amazon.com. Other places.

AWS 63

Back-to-Basics Weekend Reading - Staged Event-Driven Architecture

All Things Distributed

By Werner Vogels on 17 August 2012 07:00 PM. I am in São Paolo, Brazil for the 2012 AWS Latin America Summit and for The Next Web Latin America conference. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Staged Event-Driven Architecture. Permalink. Comments (). I will also be visiting Chile and Mexico on this trip and have the great fortune to meet many of our Latin American AWS customers.

Amazon DynamoDB ? a Fast and Scalable NoSQL Database.

All Things Distributed

By Werner Vogels on 18 January 2012 07:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon DynamoDB â?? a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Permalink. Comments (). Today is a very exciting day as we release Amazon DynamoDB , a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications.

Speeding Up Hadoop Builds Using Distributed Unit Tests

Highly Scalable

We recently worked with one of the Hadoop vendors on the continuous integration system for Hadoop core and other Hadoop-related projects like Pig, Hive, HBase. One of the challenges we faced was very slow automatic tests — full unit/integration test suite takes more than 2 hours for Hadoop core and more than 9 hours for Apache Pig. Although there are different ways to alleviate this problem (divide tests into suites, optimize tests by tweaking timeouts and sleeps, etc.),

Speed 100

Fast Intersection of Sorted Lists Using SSE Instructions

Highly Scalable

Intersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. At GridDynamics, we recently worked on a custom database for realtime web analytics where fast intersection of very large lists of IDs was a must for good performance.

C++ 101

Driving Compute Cost Down for AWS Customers - All Things.

All Things Distributed

By Werner Vogels on 05 March 2012 02:01 PM. AWS today announced a substantial price drop from March 1, 2012 for many of the Amazon EC2, Amazon RDS, and Amazon ElastiCache instances types around the world. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Driving Compute Cost Down for AWS Customers. Permalink. Comments ().

AWS 63

Hierarchical Navigation and Faceted Search on Top of Oracle Coherence

Highly Scalable

Some time ago I participated in design of a backend for one large online retailer company. From the business logic point of view, this was a pretty typical eCommerce service for hierarchical and faceted navigation, although not without peculiarities, but high performance requirements led us to the quite advanced architecture and technical design. In particular, we built this system on top of Oracle Coherence and designed our own data structures and indexes.

Expanding the Cloud ? Announcing Amazon Redshift, a Petabyte.

All Things Distributed

By Werner Vogels on 28 November 2012 09:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Announcing Amazon Redshift, a Petabyte-scale Data Warehouse Service. Permalink. Comments (). Today, we are excited to announce the limited preview of Amazon Redshift , a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud.

Cloud 91

Tricks with Direct Memory Access in Java

Highly Scalable

Java was initially designed as a safe, managed environment. Nevertheless, Java HotSpot VM contains a “backdoor” that provides a number of low-level operations to manipulate memory and threads directly. This backdoor – sun.misc.Unsafe – is widely used by JDK itself in the packages like java.nio or java.util.concurrent. It is hard to imagine a Java developer who uses this backdoor in any regular development because this API is extremely dangerous, non portable, and volatile.

Java 101

Expanding the Cloud ? Introducing Amazon CloudSearch - All.

All Things Distributed

By Werner Vogels on 12 April 2012 12:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Introducing Amazon CloudSearch. Permalink. Comments (). Today Amazon Web Services is introducing Amazon CloudSearch , a new web service that brings the power of the Amazon.comâ??s s search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale.

Cloud 81

Back-to-Basics Weekend Reading - Leases

All Things Distributed

I was in Los Angeles this week for the Digital Media on AWS Summit and to visit many of the studios and production houses that are using AWS for production and post-production work.

AWS 78

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

All Things Distributed

By Werner Vogels on 21 August 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Managing Cold Storage with Amazon Glacier. Permalink. Comments (). Managing long-term digital archiving is a challenge for almost every company. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage.

My Best Christmas Present ? Root Domain Support for Amazon S3.

All Things Distributed

By Werner Vogels on 27 December 2012 12:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. My Best Christmas Present â?? Root Domain Support for Amazon S3 Website Hosting. Permalink. Comments (). I have been a big fan of the Amazon S3 Static Website Hosting feature since its launch and this blog happily is being served from it.

Customer Conversations - How Intuit and Edmodo Innovate using.

All Things Distributed

By Werner Vogels on 06 April 2012 06:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Customer Conversations - How Intuit and Edmodo Innovate using Amazon RDS. Permalink. Comments (). From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud. Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS).

Amazon DynamoDB - From the Super Bowl to WeatherBug - All.

All Things Distributed

By Werner Vogels on 21 June 2012 09:00 AM. ve already passed our 2012 targets. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon DynamoDB - From the Super Bowl to WeatherBug. Permalink. Comments (). Amazon DynamoDB is the fastest growing new service in the history of AWS. In the five months since it launched in January, DynamoDB , our fast and scalable NoSQL database service, has been setting AWS growth records.

Games 71

Expanding The Cloud ? High Performance I/O Instances for Amazon.

All Things Distributed

By Werner Vogels on 19 July 2012 12:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding The Cloud â?? High Performance I/O Instances for Amazon EC2. Permalink. Comments (). AWS customers are bringing their most demanding workloads onto the cloud. These include the likes of high performance computation, for which we introduced the Cluster Compute and Cluster GPU instance types.

Cloud 70

Expanding the Cloud - The AWS Storage Gateway

All Things Distributed

Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customers’ on-premises applications.

Expanding the Cloud ? introducing the Asia Pacific (Sydney) Region.

All Things Distributed

By Werner Vogels on 12 November 2012 05:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? introducing the Asia Pacific (Sydney) Region. Permalink. Comments (). Today, Amazon Web Services has greater worldwide coverage with the launch of a new AWS Region in Sydney, Australia.

Expanding the Cloud ? The Amazon Simple Workflow Service - All.

All Things Distributed

By Werner Vogels on 22 February 2012 12:01 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? The Amazon Simple Workflow Service. Permalink. Comments (). Today AWS launched an exciting new service for developers: the Amazon Simple Workflow Service. Amazon SWF is an orchestration service for building scalable distributed applications.

Cloud 71

The Andrew File System - All Things Distributed

All Things Distributed

By Werner Vogels on 17 November 2012 11:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The Andrew File System. Permalink. Comments (). This weekend I am heading to Brussels for meetings with the European Commission, specifically with Vice-president Neelie Kroes who owns the Digital Agenda for the EU, about how to accelerate cloud usage in both business and government in Europe.

Register for AWS re: Invent - All Things Distributed

All Things Distributed

By Werner Vogels on 16 July 2012 09:00 AM. General registration opens up on July 25, 2012, however active AWS customers will get a chance to jump the line and start registering on July 20. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Register for AWS re: Invent. Permalink. Comments (). The first annual AWS user and partner conference will be held November 27-29 at The Venetian in Las Vega.

AWS 60

Back-to-Basics Weekend Reading - Sparse Partitions - All Things.

All Things Distributed

By Werner Vogels on 30 November 2012 11:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Sparse Partitions. Permalink. Comments (). The amazing AWS re: Invent conference completed last night and I am on my way to Europe for a last visit to customers this year. I am carrying with me a more theoretical paper on the principles of distributed computing: Sparse Partitions by Awerbug and Peleg.

Back-to-Basics Weekend Reading - Weighted Voting for Replicated.

All Things Distributed

By Werner Vogels on 02 November 2012 11:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Weighted Voting for Replicated Data. Permalink. Comments (). The last two weeks in Europe and Israel (The image above is from Tel Aviv) were intense so I didnt get to do much reading, hence I didnt post any reading suggestions.

Expanding the Cloud ? Provisioned IOPS for Amazon RDS - All.

All Things Distributed

By Werner Vogels on 25 September 2012 06:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? Provisioned IOPS for Amazon RDS. Permalink. Comments ().

Back-to-Basics Weekend Reading - Using Encryption for - All Things.

All Things Distributed

By Werner Vogels on 09 November 2012 05:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Using Encryption for Authentication. Permalink. Comments (). Now that I am enjoying some time in Seattle with real weekends, I like to remind you that my reading list is called back-to-basics.

Back-to-Basics Weekend Reading - Load Sharing - All Things.

All Things Distributed

By Werner Vogels on 07 September 2012 03:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Load Sharing. Permalink. Comments (). One of the main reasons for picking some of these older papers as back-to-basics reading is that the first researchers on a topic had to develop the fundamental models and principles from scratch.

Driving Storage Costs Down for AWS Customers - All Things.

All Things Distributed

By Werner Vogels on 06 February 2012 09:00 PM. AWS today announced a substantial price drop per February 1, 2012 for Amazon S3 standard storage to help customers drive their storage cost down. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Permalink. Comments ().

Back-to-Basics Weekend Reading - The 5 Minute Rule - All Things.

All Things Distributed

By Werner Vogels on 24 August 2012 04:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The 5 Minute Rule. Permalink. Comments (). I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile. The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 per GB/month.

Back-to-the-Future Weekend Reading - CryptDB - All Things.

All Things Distributed

By Werner Vogels on 14 September 2012 05:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-the-Future Weekend Reading - CryptDB. Permalink. Comments (). This weekend I am not going back in time to fundamentals, but looking forward to the future. Encryption techniques to protect sensitive information such as personal identifiable information are becoming more and more practical, and ubiquitously.

Back-to-Basics Weekend Reading - Virtualizing Operating Systems.

All Things Distributed

By Werner Vogels on 20 July 2012 12:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Virtualizing Operating Systems. Permalink. Comments (). This weekends back-to-basics reading is on operating system virtualization. There are two papers that deserve the "classic" tag as they both form the basis for operating system virtualization that is in production today.

Back-to-Basics Weekend Reading - Counting Bloom Filters - All.

All Things Distributed

By Werner Vogels on 28 September 2012 06:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Counting Bloom Filters. Permalink. Comments (). I am in India for the AWS Summits in Mumbai, Chennai and Bangalore (next week).

Cache 60

Total Cost of Ownership and the Return on Agility - All Things.

All Things Distributed

By Werner Vogels on 16 August 2012 10:00 AM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Total Cost of Ownership and the Return on Agility. Permalink. Comments ().

AWS 65

Improving the Cloud - More Efficient Queuing with SQS - All Things.

All Things Distributed

By Werner Vogels on 08 November 2012 03:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Improving the Cloud - More Efficient Queuing with SQS. Permalink. Comments (). The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that just works.

Back-to-Basics Weekend Reading - Automatic Reconfiguration in.

All Things Distributed

By Werner Vogels on 12 October 2012 06:00 PM. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Automatic Reconfiguration in Autonet. Permalink. Comments (). I wrote this post last week but didnt get around to publishing it. I am heading to Europe this weekend for, among other things, Structure Europe and the AWS Summit in Tel Aviv. My time in India is almost over and I am heading back to Seattle.