All Things Distributed

AUGUST 24, 2012

Werner Vogels weblog on building scalable and robust distributed systems. For this purpose I have picked work based on two papers by Jim Gray , the brilliant IBM / Tandem / Microsoft researcher, who won a Turing award for his contributions to data and transaction processing. The papers are from 1987, 1997 and 2007. Comments ().

Storage 110

Storage Hardware AWS Data Engineering 110

Adrian Cockcroft

JUNE 2, 2023

One example is that Netflix launched it’s streaming service in 2007, just at the point when the cost of streaming a movie over the network (which was dropping fast) became less than the cost of shipping a DVD. You need to be able to innovate fast enough to pivot or reinvent your business model and leverage the change.

Social Media 98

Social Media Innovation Open Source Entertainment 98

Particular Software

JANUARY 27, 2020

So as a company dedicated to making our customers better at building, maintaining, and running complex software systems, it's basically impossible for us to recommend building a new system using MSMQ. What if you have a current system already running on MSMQ? So what do I need to do? What do you do then? for more details.

Transportation 98

Transportation Azure Open Source Systems 98

Percona

JANUARY 14, 2019

To simulate text messages I have used ~3 billion of reddit comments (10 years from 2007 to 2017), downloaded from pushshift.io. Processed 4.15 Processed 4.15 Processed 8.19 Processed 8.19 Processed 4.15 Processed 3.05 No secondary indexes means that point selects (select by ID) tend to be very slow.

Database 40

Database Analytics Blockchain Healthcare 40

Percona

APRIL 12, 2023

MongoDB started out in 2007 as 10gen, a New York-based company looking to create a Platform as a Service (PaaS) solution. Instead of relational (SQL) databases defined primarily through a hierarchy of related sets via tables and columns, their non-relational structure used a system of collections and documents.

Open Source 79

Open Source Programming Database Servers 79

Rigor

SEPTEMBER 17, 2019

When working on a project for the Man Booker Prize in 2007 (now known as the Booker Prize), Howard saw first-hand how a heavy-load day could impact a website. To start the process of thinking about holiday readiness, he recommends that questions be asked such as, “How many people are we sending out to? Retrospectives.

Performance 68

Performance Traffic Website Performance Testing 68

Brendan Gregg

JANUARY 17, 2018

For large processes (>100 Gbytes), the delays in setting and reading pagemap data can be large, so I've added the Est(s) column to better reflect the real span of the WSS measurement. ### How it works My wss.pl I'll show them analyze the same 20 Gbyte process, along with wss.pl: #./wss-v1 . - PSS(MB): Proportional Set Size (Mbytes).

Cache 71

Cache Latency C++ Programming 71

Tasktop

APRIL 2, 2020

The first is the Installation Period , when a new technological approach and financial capital combine to create a “Cambrian explosion” of startups, disrupting companies and entire industries in the process. Calamity lies ahead for companies that still do not know how to measure and optimize the flow of value in a systemic and scalable way.

Government 109

Government Innovation Strategy Technology 109

SQL Performance

JULY 16, 2019

From 2007 until 2016, Intel was able to successfully execute their Tick-Tock release strategy, where they would introduce a new processor microarchitecture roughly every two years (a Tock release). Intel officially moved on to what they call " Process- Architecture-Optimization (PAO) " in early 2016. So, what has changed my mind?

Servers 46

Servers Hardware Architecture Retail 46

The Agile Manager

SEPTEMBER 30, 2013

Technocracy was, too: "best practice" was derived from people performing narrowly defined tasks in rigid processes that yielded poor quality products at a high cost. Wall Street Journal, 24 December 2007. Service businesses grew at scale, and the reaction to scale is hierarchy, process, and command & control.

Education 45

Education Best Practices Retail Network 45

O'Reilly Software

MARCH 7, 2018

Build a more scalable, composable, and functional architecture for interconnecting systems and applications. Welcome to a new world of data-driven systems. Today, data needs to be available at all times, serving its users—both humans and computer systems—across all time zones, continuously, in close to real time.

Transportation 72

Transportation Java Programming Architecture 72

Brendan Gregg

JANUARY 16, 2018

For large processes (>100 Gbytes), the delays in setting and reading pagemap data can be large, so I've added the Est(s) column to better reflect the real span of the WSS measurement. ### How it works My wss.pl I'll show them analyze the same 20 Gbyte process, along with wss.pl: #./wss-v1 . - PSS(MB): Proportional Set Size (Mbytes).

Cache 40

Cache Latency C++ Programming 40

Alex Russell

APRIL 29, 2021

After 20 years of neck-in-neck competition, often starting from common code lineages, there just isn't that much left to wring out of the system. Consistent improvement is the name of the game, and it can still have positive impacts, particularly as users lean on the system more heavily over time. PWA Install Prompts.

Media 145

Media Games Education Engineering 145

Highly Scalable

MARCH 10, 2015

Most of this article represents an overview of the results published by retailers and researchers who built practical decision making and optimization systems combining abstract economic models with data mining methods. The most typical use cases for this problem are recommender systems, personalized search results ranking, and targeted ads.

Retail 152

Retail C++ Analytics Metrics 152

The Netflix TechBlog

FEBRUARY 13, 2023

We accomplish this by paving the path to: Accessing and processing media data (e.g. To streamline this process, we standardized media assets with pre-processing steps that create and store dedicated quality-controlled derivatives with associated snapshotted metadata.

Media 290

Media Storage Infrastructure Systems 290

The Netflix TechBlog

MARCH 1, 2021

Background The Media Cloud Engineering and Encoding Technologies teams at Netflix jointly operate a system to process incoming media files from our partners and studios to make them playable on all devices. The first generation of this system went live with the streaming launch in 2007. Delivery?—?A

Serverless 228

Serverless Media Latency Social Media 228

Rigor

SEPTEMBER 11, 2019

Reading time 16 min Whether you’re a web performance expert, an evangelist for the culture of performance, a web engineer incorporating performance into your process, or someone new to the web performance entirely, you probably identify as curious, excited about new ideas, and always learning. Maximiliano Firtman. Maximiliano Firtman.

Performance 89

Performance Education Google Website 89

The Agile Manager

MAY 31, 2012

Assessments are based, not on whether the decisions made are any good, but on whether they were made in accordance with what is deemed to be an appropriate process. Companies invest in "process improvement" initiatives, with a focus on establishing a common methodology.

Government 45

Government Best Practices Social Media Innovation 45

The Agile Manager

JULY 29, 2007

While the systems and solutions it produces might automate tasks of the business, not to mention allow for complex tasks not otherwise practical to be manually executed, the production of those systems is a people-centric process. This presents both a dilemma and a competitive opportunity for companies today.

Government 40

Government Open Source Latency Code 40

The Agile Manager

AUGUST 23, 2007

When it all comes together, the overall benefit of the resulting system can be incredible, such as an increase in life expectancy. IT reward systems are also often based on individual performance tied to granular and highly focused statements of accomplishment. August 2007. 4 But that, too, isn’t enough. 4 White, Erin. “

Innovation 40

Innovation Technology Technology Latency 40

The Agile Manager

APRIL 26, 2010

Our Lean IT organization faces a two-phase exposure similar to the credit crisis that struck Wall Street in 2007. and the IT context (how do all these systems actually work together?) IT people master the situational complexity necessary to keep the systems that run the business humming. Initially, we face a liquidity crisis.

Government 49

Government Efficiency C++ Software 49

The Agile Manager

MARCH 31, 2021

With trade growing, the value proposition of ever-larger ships was self evident in 2007: “On a per-container basis, a larger vessel cost less to build and operate than a smaller one, allowing the owner to undercut competitors’ cargo rates and still earn a healthy profit.” The systemic fragility was as plain as day.

Transportation 52

Transportation Retail Traffic Efficiency 52

All Things Distributed

OCTOBER 2, 2017

With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.

Internet 130

Internet Internet AWS Performance 130

The Agile Manager

AUGUST 31, 2010

Efficiency, by comparison, makes us more vulnerable to one-way risk by introducing greater fragility into our processes. Common sense and statistical analysis both conclude that an optimized system is sensitive to the tiniest of variations. Without redundancy in capability, the entire line is put at risk.

Government 40

Government Efficiency Technology Technology 40

The Agile Manager

JUNE 23, 2007

This might be a data centre where hardware uptime is guaranteed to process transactions, a timesheeting capability that is available on-demand, or development of a custom application to analyse asset backed securities. Look for another Capital Markets paper in July 2007.

Technology 40

Technology Technology Energy Innovation 40

The Agile Manager

DECEMBER 28, 2007

The business processes that need to be defined, modeled and automated to fulfill that market opportunity will not be established at the front end of a project. Management Leaders Turn Attention to Followers ” The Wall Street Journal, 24 December 2007. They implode on themselves. Ballard was quoted by Anders, George.

Government 40

Government Retail Best Practices Technology 40

High Scalability

NOVEMBER 25, 2019

He and his colleagues spend their productive hours scaling large distributed file systems. Your Laptop has a filesystem used by hundreds of processes. It is limited by the disk space; it can’t expand storage elastically; it chokes if you run few I/O intensive processes or try collaborating with 100 other users.

Architecture 52

Architecture Cache Azure Storage 52