Brendan Gregg

MAY 16, 2017

I've now worked at Netflix for over three years. Time flies! I previously wrote about Netflix in [2015] and [2016], and if you are interested in what it's like to work here, I already covered much in those posts. As before, no one at Netflix has asked me to write this, and this is my personal blog and not a company post. I'll start with some exciting news, describe what my job is really like, the culture and mission, and some work updates. ## 100 Million Subscribers!

Entertainment 64

Entertainment Java Engineering Scalability 64

Brendan Gregg

NOVEMBER 29, 2017

Xen AWS 2017 In 2015, AWS launched c4, which used hardware virtualization for EBS volumes. This finally came to storage-optimized instance types in 2017 with the [i3 instance type], which used SR-IOV and the nvme storage driver. AWS Nitro 2017. --> As was announced at AWS re:Invent last night, and covered in Anthony Liguori's talk today ([CMP332]: video ), and the bare metal talk ([CMP330]: video ), the [c5 instance type] uses a new hypervisor called Nitro.

Virtualization 73

Virtualization AWS Hardware Storage 73

SQL Server According to Bob

OCTOBER 26, 2017

Please check out these new resources for SQL Server 2017 and SQL Server on Linux and Docker: Brk2045 upgrade sql server 2017 (on prem, iaa-s and paas) from Bob Ward. With the 2017 PASS Summit approaching next week, I thought it would be helpful to provide for you my list of resources when it comes to SQL Server 2017, which we announced for General Availability on October 2nd. SQL Server 2017 Release Notes. bobsql blog posts on SQL Server 2017.

Azure 40

Azure Servers Database Open Source 40

SQL Server According to Bob

SEPTEMBER 25, 2017

Scott Guthrie and Rohan Kumar officially made the announcement today at Ignite that SQL Server 2017 will be generally available on October 2nd, 2017. Andrew Brust also wrote a very nice article detailing out all the great new features that come with SQL Server 2017. Stay tuned to hear more about SQL Server 2017 over the next several weeks both on our main SQL Server blog [link] and here at the bobsql blog: [link]. SQL Server 2017

Tuning 40

Tuning Availability Servers Database 40

J. Paul Reed

JANUARY 24, 2017

I mentioned last week I’d have 2017 predictions for you… well, last week. So, what do I think 2017 will have in store for us? Prediction #1: DevOps Will Be Declared 1.0-STABLE This prediction was originally published in a larger T echBeacon piece on 2017 DevOps predictions. But we’ll see this mechanism happen in 2017. So ultimately, in 2017, Release Engineering will need to (continue?)

IoT 40

IoT C++ Infrastructure Internet 40

Brendan Gregg

SEPTEMBER 5, 2017

Many people have contacted me recently about switching from Solaris (or illumos) to Linux, especially since most of the Solaris kernel team were let go this year (including my former colleagues, I'm sorry to hear). This includes many great engineers who I'm sure will excel in whatever they choose to work on next. They have been asking me about Linux because I've worked for years on each platform: Solaris, illumos, and Linux, in all cases full time and as a subject matter expert.

Virtualization 64

Virtualization AWS Network Entertainment 64

SQL Server According to Bob

SEPTEMBER 27, 2017

Please check out a new blog post I created detailing how SQL Server 2017 is Fast, Built-In. Look for more details on my blog post and the SQL Server Blog about some of these concepts and new features for performance for SQL Server 2017. I just presented on this topic at Ignite and will repeat this same session at the upcoming SQL 2017 PASS Summit. SQL Server 2017 Fast and the Furious

Tuning 40

Tuning Servers Database Cloud 40

Brendan Gregg

DECEMBER 31, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. WARNING: These tunables were developed in late 2017, for Ubuntu Xenial instances on EC2.** There was so much demand for it this year that I had three overflow rooms streaming it, and people still couldn't get in. (I I shouldn't let this go to my head, as there were 42,000 attendees at re:Invent looking for something to see!)

Tuning 62

Tuning AWS Best Practices Games 62

Nick Desaulniers

SEPTEMBER 5, 2017

Martin had data comparing gcc to llvm from Q4 2015, but I wanted to see what the data looked like now in Q3 2017 and wanted to share my findings; simply rerunning the numbers. A blog post from a few years ago that really stuck with me was Martin Olsson’s Browser Engines 2015: Commit Rates and Active Developer Counts , where he shows information about the number of authors and commits to popular web browsers.

C++ 63

C++ Open Source Development Engineering 63

Brendan Gregg

AUGUST 24, 2017

While all eyes and instruments were aimed at the total eclipse of August 21st, I had a video camera pointed in the opposite direction. I was hoping to catch changes in the landscape – perhaps the Moon's shadow as it raced by. This was filmed from on a road north east of Madras, Oregon, almost right on the center line. It was just an experiment that I wasn't going to share, but when I saw it afterwards I thought it might interest some.

Engineering 40

Engineering 40

The Symphonia

DECEMBER 6, 2017

Serverless at re:Invent 2017 was originally published in The Symphonium on Medium, where people are continuing the conversation by highlighting and responding to this story. The light show at the re:Play party Another re:Invent has come and gone, and us mere AWS-using mortals are now rapidly trying to sort the wheat from the chaff of a heady harvest of announcements. It’s funny to think that AWS Lambda was announced at re:Invent only 3 years ago?—?the

Serverless 40

Serverless Traffic Azure Virtualization 40

Alex Podelko

OCTOBER 3, 2017

This year it would be held on November 6-9, 2017 in New Orleans, LA. A lot of events are scheduled: Performance and Capacity Hackathon (which is $10 / free for the conference attendees), Performance Testing Workshop, CMG Training / Badge Program, and the CMG imPACt 2017 conference itself. I am very excited by the upcoming CMG imPACt performance and capacity conference.

IoT 109

IoT Programming Performance Performance Testing 109

Brendan Gregg

DECEMBER 30, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. WARNING: These tunables were developed in late 2017, for Ubuntu Xenial instances on EC2.**

Tuning 52

Tuning Best Practices AWS Games 52

Alex Podelko

AUGUST 24, 2017

There was a rather heated discussion around A Context-Driven Approach to Automation in Testing by James Bach and Michael Bolton (referred below as the article and the authors – other references would be explicit). Here are a couple of places where it got fiercely attacked Reviewing “Context Driven Approach to Automation in Testing” by Chris McMahon. and Open letter to “CDT Test Automation” reviewers as well as in many other different places.

Testing 126

Testing Performance Testing Tools Systems 126

The Symphonia

DECEMBER 19, 2017

Hello everyone, and welcome to the last Serverless Insights of 2017! Symphonia’s Serverless Insights — December 2017 was originally published in The Symphonium on Medium, where people are continuing the conversation by highlighting and responding to this story. This is the latest edition of our newsletter. To receive the email version please subscribe here.

Serverless 40

Serverless Lambda DevOps Availability 40

Brendan Gregg

NOVEMBER 28, 2017

Xen AWS 2017 In 2015, AWS launched c4, which used hardware virtualization for EBS volumes. This finally came to storage-optimized instance types in 2017 with the [i3 instance type], which used SR-IOV and the nvme storage driver. AWS Nitro 2017. --> As was announced at AWS re:Invent last night, and covered in Anthony Liguori's talk today ([CMP332]: video ), and the bare metal talk ([CMP330]: video ), the [c5 instance type] uses a new hypervisor called Nitro.

Virtualization 40

Virtualization Hardware Storage AWS 40

All Things Distributed

SEPTEMBER 25, 2017

2017 continues a busy year for AWS in the Middle East. They are also migrating 700 government websites, with more than 50 TB of data, onto AWS, helping them to meet their goal of decommissioning their hosting platform by the end of 2017. Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East!

AWS 106

AWS Cloud Education Government 106

Jos

JUNE 17, 2017

Image taken from the Chrome 60 blog post , which first appeared in “Web Performance: Leveraging the Metrics that Most Affect User Experience” at Google I/O 2017 Up until now we have been measuring performance through other metrics, mostly using the Navigation Timing API , which is also what Google Analytics uses for their Site Speed report.

Google 100

Google Metrics Analytics Monitoring 100

Software Architecture

AUGUST 24, 2017

Memcached is in-memory key value store whereas Redis is in-memory data structures store. Memcached supports only string data type which is ideal for storing read-only data. Redis supports almost all types of data. Redis can also be used as a messaging system such as pubsub. Memcached cannot be used as a pubsub system. Memcached is more memory efficient than Redis. Redis is more memory efficient only after you use hashes. Memcached is volatile store and Redis is non-volatile.

Traffic 100

Traffic Software Architecture Efficiency Website 100

Software Architecture

AUGUST 24, 2017

For a quick introduction on what Apache Cassandra is, take a look here. Consistency is a significantly large topic to cover in one part. So I’ll be completing it in 3 parts. This first part defines consistency in general, write consistency, read consistency, consistency levels (CL), immediate, eventual and tunable consistency. Consistency The topic and concept of consistency is very important when you work with a distributed database like Cassandra.

Software Architecture 100

Software Architecture Tuning Servers Speed 100

Software Architecture

AUGUST 24, 2017

In part 1 , I introduced the basics of consistency in general, write consistency, read consistency, consistency levels (CL), immediate, eventual and tunable consistency. In this part 2, I will talk about how to achieve immediate and eventual consistency using different write and read consistency levels.

Tuning 100

Tuning Software Architecture C++ Architecture 100

Software Architecture

AUGUST 24, 2017

In part 2 , I explained how to achieve immediate and eventual consistency using different write and read consistency levels. In this part, I’ll go a bit deeper into understanding different configuration settings and consistency levels. Immediate Consistency with Write CL = ALL, Read CL = ONE The write request is sent to all replicas. Complete write operation is considered successful when all replicas respond a write success.

Software Architecture 100

Software Architecture Architecture Availability Systems 100

All Things Distributed

OCTOBER 2, 2017

In support of Amazon Prime Day 2017, the biggest day in Amazon retail history, DynamoDB served over 12.9 Today marks the 10 year anniversary of Amazon's Dynamo whitepaper , a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations.

Internet 98

Internet Internet AWS Database 98

Alex Podelko

AUGUST 15, 2017

An interesting discussion started around a very good post Open Source Load Testing Tool Review by Ragnar Lönn. Somehow the latest comment, which I got in my e-mail, didn’t get published. So I decided to copy the discussion as is and publish the comment (and my answer) here. Check the original post (as well as many other great posts) following the link above – but the discussion below is somewhat independent.

Testing Tools 100

Testing Tools Testing Website Open Source 100

All Things Distributed

SEPTEMBER 4, 2017

This article titled " Wie Unternehmen vom Vormarsch des maschinellen Lernens profitieren können " appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. When a technology has its breakthrough, can often only be determined in hindsight. In the case of artificial intelligence (AI) and machine learning (ML), this is different. ML is that part of AI that describes rules and recognizes patterns from large amounts of data in order to predict future data.

Artificial Intelligence 103

Artificial Intelligence Airlines Transportation Education 103

All Things Distributed

JUNE 30, 2017

So much so, that in March 2017, we announced Amazon Connect, which is the result of nearly ten years of work to build cloud-based contact centers at scale to power customer service for more than 50 Amazon teams and subsidiaries, including Amazon.com, Zappos, and Audible. Customer service is central to the overall customer experience that all consumers are familiar with when communicating with companies.

Artificial Intelligence 98

Artificial Intelligence Lambda AWS Virtualization 98

All Things Distributed

JUNE 20, 2017

Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. The new AWS Asia Pacific (Hong Kong) Region will have three Availability Zones and be ready for customers for use in 2018. Over the past decade, we have seen tremendous growth at AWS.

Logistics 103

Logistics AWS Social Media Cloud 103

All Things Distributed

NOVEMBER 21, 2017

Amazon ElastiCache embodies much of what makes fast data a reality for customers looking to process high volume data at incredible rates, faster than traditional databases can manage. Developers love the performance, simplicity, and in-memory capabilities of Redis, making it among the most popular NoSQL key-value stores. Redis's microsecond latency has made it a de facto choice for caching.

Games 94

Games Education Retail Latency 94

All Things Distributed

MAY 19, 2017

Amazon Aurora: Design Considerations for HighThroughput Cloud-Native Relational Databases ", Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, Xiaofeng Bao, in SIGMOD '17 Proceedings of the 2017 ACM International Conference on Management of Data, Pages 1041-1052 May 14 – 19, 2017, Chicago, IL, USA

Design 88

Design Database Storage AWS 88

All Things Distributed

JUNE 21, 2017

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX) , a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console – no application rewrites required.

Speed 94

Speed Cache AWS Latency 94

All Things Distributed

APRIL 12, 2017

I am excited to announce that AWS is opening its first Mentor's Office at STATION F in Paris! The Mentor's Office is a workplace exclusively dedicated to meetings between AWS experts and the startups. STATION F is the world's biggest startup campus. With this special offer starting at the end of June, at the campus opening, AWS increases the support already available to startup customers in France.

AWS 96

AWS Innovation Programming Cloud 96

All Things Distributed

JUNE 28, 2017

This article titled " Wartet nicht auf Perfektion – lernt aus euren Fehlern! appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. "Man errs as long as he doth strive." " Goethe, the German prince of poets, knew that already more than 200 years ago. His words still ring true today, but with a crucial difference: Striving alone is not enough. You have to strive faster than the rest.

Innovation 93

Innovation Website Servers Systems 93

All Things Distributed

APRIL 4, 2017

Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden. The new AWS EU (Stockholm) Region will have three Availability Zones and will be ready for customers to use in 2018. Over the past decade, we have seen tremendous growth at AWS.

Airlines 96

Airlines AWS Games Latency 96

All Things Distributed

MARCH 4, 2017

"Everything fails, all the time." A humble computer scientist once said. With all the resources we have today, it is easier for us to achieve fault-tolerance than it was many decades ago when computers began playing a role in critical systems such as health care, air traffic control and financial market systems. In the early days, the thinking was to use a hardware approach to achieve fault-tolerance. It was not until the mid-nineties that software fault-tolerance became more acceptable.

Hardware 96

Hardware Traffic Systems Software 96

All Things Distributed

MARCH 15, 2017

Amazon today announced a new program that will make it free for tens of thousands of Alexa developers to build and host most Alexa skills using Amazon Web Services (AWS). Many Alexa skill developers currently take advantage of the AWS Free Tier, which offers one million AWS Lambda requests and up to 750 hours of Amazon Elastic Compute Cloud (Amazon EC2) compute time per month at no charge. However, if developers exceed the AWS Free Tier limits, they may incur AWS usage fees each month.

Games 96

Games Development AWS Lambda 96

All Things Distributed

MARCH 10, 2017

More and more we see stories appearing, like this one in HBR by MIT Media Lab's Joi Ito and crew. It praises the power of blockchain as a disruptive technology, on par with how "the internet" changed everything. I am always surprised to see that these far-reaching predictions are made, without diving into the technology itself. This weekend I would like to read about some of the technologies that predate blockchain, as they are its fundamental building blocks.

Blockchain 95

Blockchain Media Technology Technology 95

All Things Distributed

MARCH 30, 2017

This weekend I am travelling to Australia for the first AWS Summit of 2017. I find on such a long trip, to keep me from getting distracted, I need an exciting paper that is easy to read. Last week's 'Deep Learning' overview would have not met those requirements. One topic that always gets me excited is how to take computer science research and implement it in production systems. There are often so many obstacles that we do not see much of this work happening.

Systems 83

Systems AWS Design Technology 83

All Things Distributed

MARCH 24, 2017

In the past few years, we have seen an explosion in the use of Deep Learning as its software platforms mature and the supporting hardware, especially GPUs with larger memories, become widely available. Even though this is a recent development, Deep Learning has deep historical roots, tracing back all the way to the sixties, or maybe even earlier.

Network 93

Network Hardware Availability Software 93

All Things Distributed

DECEMBER 18, 2017

Today, I'm happy to announce that the AWS EU (Paris) Region, our 18th technology infrastructure Region globally, is now generally available for use by customers worldwide. With this launch, AWS now provides 49 Availability Zones, with another 12 Availability Zones and four Regions in Bahrain, Hong Kong, Sweden, and a second AWS GovCloud (US) Region expected to come online by early 2019.

Education 83

Education Artificial Intelligence Energy AWS 83

All Things Distributed

FEBRUARY 3, 2017

Listening to the " Algorithms to Live By " audio on my commute this morning, once again I was struck by the beauty of Bloom Filters. So, I decided it is time to resurrect the 'Back-to-Basics Weekend Reading' series, as I will be re-reading some fundamental CS papers this weekend. In the past, I have done some weekend reading about Counting Bloom Filters , but now I am going even more fundamental, and I invite you to join me.

Cache 93

Cache Efficiency Code Systems 93

All Things Distributed

FEBRUARY 24, 2017

In Reliable Distribution Systems, we need to handle different failure scenarios. Many of those deal with message loss and process failure. However, there is a class of scenarios that deal with malfunctioning processes, which send out conflicting information. The challenge is developing algorithms that can reach an agreement in the presence of these failures.

Programming 92

Programming Systems Processing Design 92

All Things Distributed

JUNE 7, 2017

Unlocking the value of data is a primary goal that AWS helps our customers to pursue. In recent years, an explosion of intelligent devices have created oceans of new data across many industries. We have seen that such devices can benefit greatly from the elastic resources of the cloud. This is because data gets more valuable when it can be processed together with other data. At the same time, it can be valuable to process some data right at the source where it is generated.

AWS 86

AWS IoT Cloud Lambda 86

All Things Distributed

DECEMBER 20, 2017

This article titled " Daten müssen strategischer Teil des Geschäfts werden " appeared in German last week in the "IT und Datenproduktion" column of Wirtschaftwoche. How companies can use ideas from mass production to create business with data. Strategically, IT doesn't matter. That was the provocative thesis of a much-talked-about article from 2003 in the Harvard Business Review by the US publicist Nicolas Carr.

Artificial Intelligence 75

Artificial Intelligence Social Media Logistics Software 75