Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. It became clear that real-time query processing and in-stream processing is the immediate need in many practical applications. In recent years, this idea got a lot of traction and a whole bunch of solutions like Twitter’s Storm, Yahoo’s S4, Cloudera’s Impala, Apache Spark, and Apache Tez appeared and joined the army of Big Data and NoSQL systems.

Big Data 154

Big Data Processing Lambda Database 154

All Things Distributed

NOVEMBER 26, 2013

'As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas , which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.6 and enable you to maintain a nearly up-to-date copy of your master database in a different AWS Region.

Cloud 147

Cloud AWS Traffic Latency 147

Tim Kadlec

JUNE 10, 2013

The topic of responsive images has been one of the most hotly debated topics amongst web developers for what feels like forever. I think Jason Grigsby was perhaps the first to publicly point out that simply setting a percentage width on images was not enough, you needed to resize these images as well. He showed that if you served appropriately sized images on the original responsive demo site, you could shave 78% off the weight of those images (about 162kB) on small screens.

Efficiency 76

Efficiency Processing Servers Design 76

The Agile Manager

JULY 30, 2013

Product management was never a formal responsibility; it just sort of happened. Early on, it was driven by what the technical wizards came up with. But the magic left the development team years ago: it had been gutted by several rounds of staff cuts that took the garrulous personalities and innovative thinkers. It took the wind from development's sails: those who were still on the payroll were just happy to have kept their jobs.

Design 65

Design Innovation Mobile Software 65

Sutter's Mill

JULY 9, 2013

I don’t often link to other articles, but this one is worth reading. Why mobile web apps are slow. by Drew Crawford. … So if you are trying to figure out exactly what brand of crazy all your native developer friends are on for continuing to write the evil native applications on the cusp of the open web revolution, or whatever, then bookmark this page, make yourself a cup of coffee, clear an afternoon, find a comfy chair, and then we’ll both be ready.

Mobile 53

Mobile C++ Code Servers 53

Alex Russell

NOVEMBER 6, 2013

Something irks me about the phrase “semantic HTML” The intent is clear enough — using HTML in ways that are readable, using plain language to describe things. But that’s not what “semantic” means. We might as well be saying “well written” or “copy edited” They’d be closer fits. What we describe today as “unsemantic markup” isn’t really; there is actual meaning to end-users, but it is conveyed with long-winded mar

Engineering 40

Engineering Technology Technology Systems 40

Speed Curve

NOVEMBER 5, 2013

I've redesigned the weekly email reports to provide more trending information so you can compare week on week performance improvements. They're nice simple visualizations for forwarding around your whole team. Let me know what you'd like to see added to these emails.

Performance 40

Performance 40

Nick Desaulniers

AUGUST 27, 2013

Node.js is a great runtime for writing applications in JavaScript, the language I primarily develop in. CoffeeScript is a programming language that compiles to JavaScript. Why would we write a reusable piece of code, a module , in CoffeeScript? CoffeeScript is a very high level language and beautifully brings together my favorite aspects of JavaScript, Ruby, and Python.

Open Source 40

Open Source Programming Code Development 40

All Things Distributed

SEPTEMBER 6, 2013

'This spring I travelled through Europe for the AWS Global Summit series. In my many conversations with customers, and with the media, I encountered surprise and excitement about the extent that European enterprises have already been using the Amazon Web Services for some time. Whether it is large telecommunications manufactures like Nokia Siemens Networks running their real-time data analytics for network operators on AWS, or a luxury hotel chain like Kempinski moving their core IT functions to

Cloud 130

Cloud Energy AWS Media 130

All Things Distributed

DECEMBER 23, 2013

'This year I have not been able to publish many back-to-basics readings, so I will not close the year with a recap of those. Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 " Hints for computer system design ". The presentation was part of the Heidelberg Laureate Forum helt in September of this year.

Design 129

Design Systems 129

All Things Distributed

APRIL 17, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud: Faster, More Flexible Queries with DynamoDB. By Werner Vogels on 17 April 2013 10:30 AM. | Permalink. | Comments (). Today, Iâ??m thrilled to announce that we have expanded the query capabilities of DynamoDB. We call the newest capability Local Secondary Indexes (LSI).

Cloud 116

Cloud Latency Games Scalability 116

All Things Distributed

JULY 7, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Exerting Fine Grain Control Over Your Cloud Resources. By Werner Vogels on 07 July 2013 06:30 PM. | Permalink. | Comments (). I am thrilled that now both Amazon EC2 and Amazon RDS support resource-level permissions. As customers move increasing amounts of compute and database workloads over to AWS, they have expressed an increased desire for finer grain control over their underlying resources.

Cloud 116

Cloud AWS Database Scalability 116

All Things Distributed

MAY 15, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. DynamoDB Keeps Getting Better (and cheaper!). By Werner Vogels on 15 May 2013 06:30 PM. | Permalink. | Comments (). We love getting feedback so we can deliver the improvements and new features that really matter to our customers. You can see from the pace at which we roll out new functionality that teams across AWS take this very seriously.

AWS 113

AWS Scalability Systems Performance 113

All Things Distributed

MAY 22, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon Redshift and Designing for Security. By Werner Vogels on 22 May 2013 08:30 AM. | Permalink. | Comments (). Itâ??s been a few months since I last wrote about Amazon Redshift and I thought Iâ??d update you on some of the things we are hearing from customers.

Design 112

Design AWS Hardware Media 112

All Things Distributed

MARCH 20, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. The Netflix OSS Cloud Prize. By Werner Vogels on 20 March 2013 05:00 PM. | Permalink. | Comments (). Netflix has over the years become one of the absolute best engineering powerhouses for building cloud-native applications. At AWS we are very proud to be their infrastructure partner and every day we learn from how they use our cloud services.

Cloud 112

Cloud AWS Open Source Infrastructure 112

All Things Distributed

AUGUST 13, 2013

'This year as I hosted AWS Summits in 12 different cities around the world, I met thousands of developers who are building powerful new applications for smartphones, tablets and other connected devices, all running mobile cloud backends on AWS. These developers want to engage their users with timely, dynamic content even when the users haven’t opened their mobile apps.

Mobile 110

Mobile Development AWS Games 110

All Things Distributed

OCTOBER 30, 2013

'Speed of development, scalability, and simplicity of management are among the critical needs of mobile developers. With the proliferation of mobile devices and users, and small agile teams that are tasked with building successful mobile apps that can grow from 100 users to 1 million users in a few days, scalability of the underlying infrastructure and simplicity of management are more important than ever.

Mobile 110

Mobile Games Infrastructure Scalability 110

All Things Distributed

MARCH 11, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Elastic Beanstalk a la Node. By Werner Vogels on 11 March 2013 04:00 PM. | Permalink. | Comments (). I spent a lot of time talking to AWS developers, many working in the gaming and mobile space, and most of them have been finding Node.js well suited for their web applications.

AWS 105

AWS Mobile Games Java 105

All Things Distributed

FEBRUARY 11, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud: The Amazon Elastic Transcoder. By Werner Vogels on 11 February 2013 04:00 PM. | Permalink. | Comments (). While I was returning from an exciting time in New Orleans watching the Super Bowl, AWS launched a very cool, brand new service: Amazon Elastic Transcoder.

AWS 105

AWS Media Scalability Storage 105

All Things Distributed

SEPTEMBER 5, 2013

'Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. The explosive growth in mobile devices and mobile apps is generating a huge amount of data, which has fueled the demand for big data services and for high scale databases. Meanwhile, mobile app developers have shown that they care a lot about getting to market quickly, the ability to easily scale their app from 100 users to 1 million users on day 1, and the extreme

Big Data 100

Big Data Mobile Latency Database 100

All Things Distributed

NOVEMBER 13, 2013

'Today we are kicking off AWS re:Invent 2013. Over the course of the next three days, we will host more than 200 sessions, training bootcamps, and hands on labs taught by expert AWS staff as well as dozens of our customers. This year’s conference kicks off with a keynote address by AWS Senior Vice President Andy Jassy, followed by my keynote on Thursday morning.

AWS 99

AWS Tuning Innovation Cloud 99

All Things Distributed

OCTOBER 25, 2013

'Many of you know Thorsten von Eicken as the founder of Rightscale , the company that has helped numerous organizations find their way onto AWS. In what seems almost a previous life by now Thorsten was one of the top young professors in Distributed Systems and I had the great pleasure of working with him at Cornell in the early 90''s. What set Thorsten aside from so many other system research academics was his desire to build practical, working systems, a path that I followed as well.

Network 99

Network Latency Operating System Speed 99

All Things Distributed

JULY 19, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Feeling the Customer Love for AWS. By Werner Vogels on 19 July 2013 11:00 AM. | Permalink. | Comments (). We work hard to meet our customers expectations and to continue to innovate on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage.

AWS 99

AWS Innovation Scalability Systems 99

All Things Distributed

APRIL 12, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Join Processing in Relational Databases. By Werner Vogels on 12 April 2013 04:00 AM. | Permalink. | Comments (). Joins are one of the fundamental relational database query operations. It is very hard to implement the join operation efficiently as there any many unknowns in the execution of the operation.

Processing 97

Processing Database Efficiency Scalability 97

All Things Distributed

AUGUST 30, 2013

'Storing and querying datasets that contain objects in a geometric space have always required special treatment. The choice of data structures and query algorithms can easily make the different between a query that runs in seconds or in days. Much of the fundamental work has been done in the late eighties and early nineties, for examples around topological relations (disjoint, meet, equal, overlap, contains, etc.), direction relations (north, north-east, etc.) and distance relations (far, near),

Database 90

Database Systems 90

All Things Distributed

SEPTEMBER 20, 2013

'Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Not everybody agreed that the "N-ary Storage Model" (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives.

Storage 90

Storage Hardware Cache C++ 90

All Things Distributed

JULY 17, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. AWS re:Invent 2013. By Werner Vogels on 17 July 2013 05:00 PM. | Permalink. | Comments (). The AWS re:Invent user conference last year in Las Vegas was by many described as the best technology conference they had been to in a long time. We had worked hard to give you great keynote sessions as well as deep technical content by AWS engineers, partners and customers.

AWS 89

AWS Strategy Scalability Cloud 89

All Things Distributed

JANUARY 25, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Epidemics. By Werner Vogels on 25 January 2013 06:00 PM. | Permalink. | Comments (). My paper to read this weekend was the Alan Demers seminal paper on epidemic techniques for database replication. I realized that in 2004, before my Amazon days, I already wrote a blog post about the fundamental publications in the area of epidemics, so this seems like a good moment

Database 89

Database Scalability C++ Systems 89

All Things Distributed

JULY 19, 2013

We work hard to meet our customer’s expectations and to continue to innovate on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage. Jayaram Gopinath Nagaraj and Kavitha Doraimaickam gave a truly electrifying presentation about how AWS has transformed their radio stations.

AWS 78

AWS Innovation 78

All Things Distributed

OCTOBER 4, 2013

'The anonymity routing network Tor is frequently in the news these days, which makes it a good case to read up on the fascinating technologies behind it. Tor stands for The Onion Router as its technology is based on the onion routing principles. These principles were first described by Goldschlag, et al., from the Naval Research Lab, in their 1996 paper on Hiding Routing Information.

Network 85

Network Technology Technology Systems 85

All Things Distributed

AUGUST 23, 2013

'The intense travels around the world in the spring have kept me from keeping up on the historical reading that I would like to do, as such there have not been that many suggesting for the back-to-basics reading list. The fall is going be not that much different but I will make an effort to get back into a reading habit. I want to kick off the fall readings not with an historical paper but with two that detail GraphLab , an excellent framework for high performance machine learning that originall

Cloud 83

Cloud Operating System Design Systems 83

All Things Distributed

MAY 10, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Using continuations to implement thread management and communication in operating systems. By Werner Vogels on 10 May 2013 09:30 AM. | Permalink. | Comments (). I have returned from a great series of AWS Summits in NYC and in Europe so it is time to get back to some weekend reading.

Operating System 73

Operating System AWS Systems Scalability 73

All Things Distributed

APRIL 5, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Practical Applications of Triggers and Constraints: Successes and Lingering Issues. By Werner Vogels on 05 April 2013 08:30 AM. | Permalink. | Comments (). At the end of the 80s Ceri and Widom were researching the fundamentals of integrity constraints in databases.

Database 73

Database Scalability Systems 73

All Things Distributed

MARCH 29, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Principles of Transaction-Oriented Database Recovery. By Werner Vogels on 29 March 2013 03:30 PM. | Permalink. | Comments (). I have been reading mainly newer papers in the beginning of this year, but it is time to get back to the basics and start reading some more historical papers again.

Database 72

Database Scalability Engineering Systems 72

Tim Kadlec

JULY 30, 2013

Back in 2011, Brad Frost wrote a post on Support vs. Optimization. One of the (many) smart things he said was: The power of the web is its ubiquity. It is the web’s superpower, and its omnipresence is what sets it apart from native platforms. This is what excites me about the web and it’s why web technology tends to be my focus. That ubiquity, that ability to get your information to anyone with a device connected to the web, is incredibly inspiring.

Network 66

Network Technology Technology 66

Tim Kadlec

JANUARY 27, 2013

Jason Grigsby once quipped that “We’ve remade the Internet in our image….obese.” He was right, of course. Average page weight and number of connections has been increasing at a rather alarming rate. This is why I’ve been so happy to see the recent rash of posts discussing performance as a fundamental component of design. The latest comes from Mr.

Performance 64

Performance Internet Internet Design 64

The Agile Manager

OCTOBER 31, 2013

The word "governance" is misused in IT, particularly in software development. There are two popular misconceptions. One is that it consists of a steering committee of senior executives with oversight responsibility for delivery; it's responsibilities are largely super-management tasks. The other is that it is primarily concerned with compliance with protocols, procedures or regulations, such as ITIL or Sarbanes-Oxley or even coding and architectural standards.

Government 58

Government Strategy Technology Technology 58