Working at Netflix 2017

Brendan Gregg

I've now worked at Netflix for over three years. Time flies! I previously wrote about Netflix in [2015] and [2016], and if you are interested in what it's like to work here, I already covered much in those posts. As before, no one at Netflix has asked me to write this, and this is my personal blog and not a company post. I'll start with some exciting news, describe what my job is really like, the culture and mission, and some work updates. ## 100 Million Subscribers!

AWS EC2 Virtualization 2017: Introducing Nitro

Brendan Gregg

Xen AWS 2017 In 2015, AWS launched c4, which used hardware virtualization for EBS volumes. This finally came to storage-optimized instance types in 2017 with the [i3 instance type], which used SR-IOV and the nvme storage driver. AWS Nitro 2017. --> As was announced at AWS re:Invent last night, and covered in Anthony Liguori's talk today ([CMP332]: video ), and the bare metal talk ([CMP330]: video ), the [c5 instance type] uses a new hypervisor called Nitro.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Experience SQL Server 2017: Start your journey here

SQL Server According to Bob

Please check out these new resources for SQL Server 2017 and SQL Server on Linux and Docker: Brk2045 upgrade sql server 2017 (on prem, iaa-s and paas) from Bob Ward. With the 2017 PASS Summit approaching next week, I thought it would be helpful to provide for you my list of resources when it comes to SQL Server 2017, which we announced for General Availability on October 2nd. SQL Server 2017 Release Notes. bobsql blog posts on SQL Server 2017.

Azure 40

SQL Server 2017 generally available on October 2nd…

SQL Server According to Bob

Scott Guthrie and Rohan Kumar officially made the announcement today at Ignite that SQL Server 2017 will be generally available on October 2nd, 2017. Andrew Brust also wrote a very nice article detailing out all the great new features that come with SQL Server 2017. Stay tuned to hear more about SQL Server 2017 over the next several weeks both on our main SQL Server blog [link] and here at the bobsql blog: [link]. SQL Server 2017

2017 Wheel of Fortune

J. Paul Reed

I mentioned last week I’d have 2017 predictions for you… well, last week. So, what do I think 2017 will have in store for us? Prediction #1: DevOps Will Be Declared 1.0-STABLE This prediction was originally published in a larger T echBeacon piece on 2017 DevOps predictions. But we’ll see this mechanism happen in 2017. So ultimately, in 2017, Release Engineering will need to (continue?)

IoT 40

Solaris to Linux Migration 2017

Brendan Gregg

Many people have contacted me recently about switching from Solaris (or illumos) to Linux, especially since most of the Solaris kernel team were let go this year (including my former colleagues, I'm sorry to hear). This includes many great engineers who I'm sure will excel in whatever they choose to work on next. They have been asking me about Linux because I've worked for years on each platform: Solaris, illumos, and Linux, in all cases full time and as a subject matter expert.

Experience SQL Server 2017: The Fast and the Furious

SQL Server According to Bob

Please check out a new blog post I created detailing how SQL Server 2017 is Fast, Built-In. Look for more details on my blog post and the SQL Server Blog about some of these concepts and new features for performance for SQL Server 2017. I just presented on this topic at Ignite and will repeat this same session at the upcoming SQL 2017 PASS Summit. SQL Server 2017 Fast and the Furious

AWS re:Invent 2017: How Netflix Tunes EC2

Brendan Gregg

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. WARNING: These tunables were developed in late 2017, for Ubuntu Xenial instances on EC2.** There was so much demand for it this year that I had three overflow rooms streaming it, and people still couldn't get in. (I I shouldn't let this go to my head, as there were 42,000 attendees at re:Invent looking for something to see!)

GCC vs LLVM Q3 2017: Active Developer Counts

Nick Desaulniers

Martin had data comparing gcc to llvm from Q4 2015, but I wanted to see what the data looked like now in Q3 2017 and wanted to share my findings; simply rerunning the numbers. A blog post from a few years ago that really stuck with me was Martin Olsson’s Browser Engines 2015: Commit Rates and Active Developer Counts , where he shows information about the number of authors and commits to popular web browsers.

C++ 63

Total Solar Eclipse 2017

Brendan Gregg

While all eyes and instruments were aimed at the total eclipse of August 21st, I had a video camera pointed in the opposite direction. I was hoping to catch changes in the landscape – perhaps the Moon's shadow as it raced by. This was filmed from on a road north east of Madras, Oregon, almost right on the center line. It was just an experiment that I wasn't going to share, but when I saw it afterwards I thought it might interest some.

Serverless at re:Invent 2017

The Symphonia

Serverless at re:Invent 2017 was originally published in The Symphonium on Medium, where people are continuing the conversation by highlighting and responding to this story. The light show at the re:Play party Another re:Invent has come and gone, and us mere AWS-using mortals are now rapidly trying to sort the wheat from the chaff of a heady harvest of announcements. It’s funny to think that AWS Lambda was announced at re:Invent only 3 years ago?—?the

Excited by the Upcoming CMG imPACt Performance and Capacity Conference

Alex Podelko

This year it would be held on November 6-9, 2017 in New Orleans, LA. A lot of events are scheduled: Performance and Capacity Hackathon (which is $10 / free for the conference attendees), Performance Testing Workshop, CMG Training / Badge Program, and the CMG imPACt 2017 conference itself. I am very excited by the upcoming CMG imPACt performance and capacity conference.

IoT 109

AWS re:Invent 2017: How Netflix Tunes EC2

Brendan Gregg

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. WARNING: These tunables were developed in late 2017, for Ubuntu Xenial instances on EC2.**

Follow up on the Test Automation Discussion – Bringing in the Performance View

Alex Podelko

There was a rather heated discussion around A Context-Driven Approach to Automation in Testing by James Bach and Michael Bolton (referred below as the article and the authors – other references would be explicit). Here are a couple of places where it got fiercely attacked Reviewing “Context Driven Approach to Automation in Testing” by Chris McMahon. and Open letter to “CDT Test Automation” reviewers as well as in many other different places.

Symphonia’s Serverless Insights — December 2017

The Symphonia

Hello everyone, and welcome to the last Serverless Insights of 2017! Symphonia’s Serverless Insights — December 2017 was originally published in The Symphonium on Medium, where people are continuing the conversation by highlighting and responding to this story. This is the latest edition of our newsletter. To receive the email version please subscribe here.

AWS EC2 Virtualization 2017: Introducing Nitro

Brendan Gregg

Xen AWS 2017 In 2015, AWS launched c4, which used hardware virtualization for EBS volumes. This finally came to storage-optimized instance types in 2017 with the [i3 instance type], which used SR-IOV and the nvme storage driver. AWS Nitro 2017. --> As was announced at AWS re:Invent last night, and covered in Anthony Liguori's talk today ([CMP332]: video ), and the bare metal talk ([CMP330]: video ), the [c5 instance type] uses a new hypervisor called Nitro.

As-Salaam-Alaikum: The cloud arrives in the Middle East!

All Things Distributed

2017 continues a busy year for AWS in the Middle East. They are also migrating 700 government websites, with more than 50 TB of data, onto AWS, helping them to meet their goal of decommissioning their hosting platform by the end of 2017. Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East!

AWS 106

PerformanceObserver and Paint Timing API

Jos

Image taken from the Chrome 60 blog post , which first appeared in “Web Performance: Leveraging the Metrics that Most Affect User Experience” at Google I/O 2017 Up until now we have been measuring performance through other metrics, mostly using the Navigation Timing API , which is also what Google Analytics uses for their Site Speed report.

Google 100

Redis vs Memcached

Software Architecture

Memcached is in-memory key value store whereas Redis is in-memory data structures store. Memcached supports only string data type which is ideal for storing read-only data. Redis supports almost all types of data. Redis can also be used as a messaging system such as pubsub. Memcached cannot be used as a pubsub system. Memcached is more memory efficient than Redis. Redis is more memory efficient only after you use hashes. Memcached is volatile store and Redis is non-volatile.

Data Consistency in Apache Cassandra — Part 1

Software Architecture

For a quick introduction on what Apache Cassandra is, take a look here. Consistency is a significantly large topic to cover in one part. So I’ll be completing it in 3 parts. This first part defines consistency in general, write consistency, read consistency, consistency levels (CL), immediate, eventual and tunable consistency. Consistency The topic and concept of consistency is very important when you work with a distributed database like Cassandra.

Data Consistency in Apache Cassandra — Part 2

Software Architecture

In part 1 , I introduced the basics of consistency in general, write consistency, read consistency, consistency levels (CL), immediate, eventual and tunable consistency. In this part 2, I will talk about how to achieve immediate and eventual consistency using different write and read consistency levels.

Tuning 100

Data Consistency in Apache Cassandra — Part 3

Software Architecture

In part 2 , I explained how to achieve immediate and eventual consistency using different write and read consistency levels. In this part, I’ll go a bit deeper into understanding different configuration settings and consistency levels. Immediate Consistency with Write CL = ALL, Read CL = ONE The write request is sent to all replicas. Complete write operation is considered successful when all replicas respond a write success.

A Decade of Dynamo: Powering the next wave of high-performance, internet-scale applications

All Things Distributed

In support of Amazon Prime Day 2017, the biggest day in Amazon retail history, DynamoDB served over 12.9 Today marks the 10 year anniversary of Amazon's Dynamo whitepaper , a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations.

The Role of Recording in Load Testing Tools

Alex Podelko

An interesting discussion started around a very good post Open Source Load Testing Tool Review by Ragnar Lönn. Somehow the latest comment, which I got in my e-mail, didn’t get published. So I decided to copy the discussion as is and publish the comment (and my answer) here. Check the original post (as well as many other great posts) following the link above – but the discussion below is somewhat independent.

AI for everyone - How companies can benefit from the advance of machine learning

All Things Distributed

This article titled " Wie Unternehmen vom Vormarsch des maschinellen Lernens profitieren können " appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. When a technology has its breakthrough, can often only be determined in hindsight. In the case of artificial intelligence (AI) and machine learning (ML), this is different. ML is that part of AI that describes rules and recognizes patterns from large amounts of data in order to predict future data.

Improving Customer Service with Amazon Connect and Amazon Lex

All Things Distributed

So much so, that in March 2017, we announced Amazon Connect, which is the result of nearly ten years of work to build cloud-based contact centers at scale to power customer service for more than 50 Amazon teams and subsidiaries, including Amazon.com, Zappos, and Audible. Customer service is central to the overall customer experience that all consumers are familiar with when communicating with companies.

Expanding the Cloud – An AWS Region is coming to Hong Kong

All Things Distributed

Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. The new AWS Asia Pacific (Hong Kong) Region will have three Availability Zones and be ready for customers for use in 2018. Over the past decade, we have seen tremendous growth at AWS.

Scaling Amazon ElastiCache for Redis with Online Cluster Resizing

All Things Distributed

Amazon ElastiCache embodies much of what makes fast data a reality for customers looking to process high volume data at incredible rates, faster than traditional databases can manage. Developers love the performance, simplicity, and in-memory capabilities of Redis, making it among the most popular NoSQL key-value stores. Redis's microsecond latency has made it a de facto choice for caching.

Games 94

Weekend Reading: Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases.

All Things Distributed

Amazon Aurora: Design Considerations for HighThroughput Cloud-Native Relational Databases ", Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, Xiaofeng Bao, in SIGMOD '17 Proceedings of the 2017 ACM International Conference on Management of Data, Pages 1041-1052 May 14 – 19, 2017, Chicago, IL, USA

Amazon DynamoDB Accelerator (DAX): Speed Up DynamoDB Response Times from Milliseconds to Microseconds without Application Rewrite.

All Things Distributed

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX) , a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console – no application rewrites required.

Speed 94

Coming to STATION F: The first Mentor's Office powered by AWS!

All Things Distributed

I am excited to announce that AWS is opening its first Mentor's Office at STATION F in Paris! The Mentor's Office is a workplace exclusively dedicated to meetings between AWS experts and the startups. STATION F is the world's biggest startup campus. With this special offer starting at the end of June, at the campus opening, AWS increases the support already available to startup customers in France.

AWS 96

Stop waiting for perfection and learn from your mistakes

All Things Distributed

This article titled " Wartet nicht auf Perfektion – lernt aus euren Fehlern! appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. "Man errs as long as he doth strive." " Goethe, the German prince of poets, knew that already more than 200 years ago. His words still ring true today, but with a crucial difference: Striving alone is not enough. You have to strive faster than the rest.

Välkommen till Stockholm – An AWS Region is coming to the Nordics

All Things Distributed

Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden. The new AWS EU (Stockholm) Region will have three Availability Zones and will be ready for customers to use in 2018. Over the past decade, we have seen tremendous growth at AWS.

Back-to-Basic Weekend Reading: Why Do Computers Stop and What Can Be Done About It?

All Things Distributed

"Everything fails, all the time." A humble computer scientist once said. With all the resources we have today, it is easier for us to achieve fault-tolerance than it was many decades ago when computers began playing a role in critical systems such as health care, air traffic control and financial market systems. In the early days, the thinking was to use a hardware approach to achieve fault-tolerance. It was not until the mid-nineties that software fault-tolerance became more acceptable.

Amazon Makes it Free for Developers to Build and Host Most Alexa Skills Using AWS

All Things Distributed

Amazon today announced a new program that will make it free for tens of thousands of Alexa developers to build and host most Alexa skills using Amazon Web Services (AWS). Many Alexa skill developers currently take advantage of the AWS Free Tier, which offers one million AWS Lambda requests and up to 750 hours of Amazon Elastic Compute Cloud (Amazon EC2) compute time per month at no charge. However, if developers exceed the AWS Free Tier limits, they may incur AWS usage fees each month.

Games 96

Back-to-Basics Weekend Reading: The Foundations of Blockchain

All Things Distributed

More and more we see stories appearing, like this one in HBR by MIT Media Lab's Joi Ito and crew. It praises the power of blockchain as a disruptive technology, on par with how "the internet" changed everything. I am always surprised to see that these far-reaching predictions are made, without diving into the technology itself. This weekend I would like to read about some of the technologies that predate blockchain, as they are its fundamental building blocks.

Back-to-Basics Weekend Reading: An Implementation of a Log-Structured File System

All Things Distributed

This weekend I am travelling to Australia for the first AWS Summit of 2017. I find on such a long trip, to keep me from getting distracted, I need an exciting paper that is easy to read. Last week's 'Deep Learning' overview would have not met those requirements. One topic that always gets me excited is how to take computer science research and implement it in production systems. There are often so many obstacles that we do not see much of this work happening.

Back-to-Basics Weekend Reading: Deep learning in neural networks

All Things Distributed

In the past few years, we have seen an explosion in the use of Deep Learning as its software platforms mature and the supporting hardware, especially GPUs with larger memories, become widely available. Even though this is a recent development, Deep Learning has deep historical roots, tracing back all the way to the sixties, or maybe even earlier.

'Paris s'éveille'! Introducing the AWS EU (Paris) Region

All Things Distributed

Today, I'm happy to announce that the AWS EU (Paris) Region, our 18th technology infrastructure Region globally, is now generally available for use by customers worldwide. With this launch, AWS now provides 49 Availability Zones, with another 12 Availability Zones and four Regions in Bahrain, Hong Kong, Sweden, and a second AWS GovCloud (US) Region expected to come online by early 2019.

Back-to-Basics Weekend Reading - Bloom Filters

All Things Distributed

Listening to the " Algorithms to Live By " audio on my commute this morning, once again I was struck by the beauty of Bloom Filters. So, I decided it is time to resurrect the 'Back-to-Basics Weekend Reading' series, as I will be re-reading some fundamental CS papers this weekend. In the past, I have done some weekend reading about Counting Bloom Filters , but now I am going even more fundamental, and I invite you to join me.

Cache 93

Back-to-Basic Weekend Reading: Byzantine Generals

All Things Distributed

In Reliable Distribution Systems, we need to handle different failure scenarios. Many of those deal with message loss and process failure. However, there is a class of scenarios that deal with malfunctioning processes, which send out conflicting information. The challenge is developing algorithms that can reach an agreement in the presence of these failures.

Unlocking the Value of Device Data with AWS Greengrass.

All Things Distributed

Unlocking the value of data is a primary goal that AWS helps our customers to pursue. In recent years, an explosion of intelligent devices have created oceans of new data across many industries. We have seen that such devices can benefit greatly from the elastic resources of the cloud. This is because data gets more valuable when it can be processed together with other data. At the same time, it can be valuable to process some data right at the source where it is generated.

AWS 86

Rethinking the 'production' of data

All Things Distributed

This article titled " Daten müssen strategischer Teil des Geschäfts werden " appeared in German last week in the "IT und Datenproduktion" column of Wirtschaftwoche. How companies can use ideas from mass production to create business with data. Strategically, IT doesn't matter. That was the provocative thesis of a much-talked-about article from 2003 in the Harvard Business Review by the US publicist Nicolas Carr.