The Netflix TechBlog

OCTOBER 19, 2020

By 2017, open source projects like Open-Tracing and Open-Zipkin were mature enough for use in polyglot runtime environments at Netflix. Our distributed tracing infrastructure is grouped into three sections: tracer library instrumentation, stream processing, and storage. Storage: don’t break the bank!

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

The Netflix TechBlog

OCTOBER 28, 2021

In 2017, I attended the Women in Big Data conference at Netflix and met with several amazing women in data engineering, including our VP. The efficiency achieved in processing and storage layers brought us big savings on computing and storage costs. You can learn more about it from my talk at the Flink forward conference.

Data Engineering 173

Data Engineering Engineering Big Data Software Engineering 173

All Things Distributed

MAY 19, 2017

We believe the central constraint in high throughput data processing has moved from compute and storage to the network. Aurora brings a novel architecture to the relational database to address this constraint, most notably by pushing redo processing to a multi-tenant scaleout storage service, purpose-built for Aurora.

Database 117

Database Design Cloud Storage 117

Dynatrace

MAY 6, 2021

Contact Dynatrace ONE if you wish to enable Cluster-side screenshot storage on pre-1.216 fresh-installed Clusters. Linux: Amazon Linux AMI 2017.x. Consumption and storage now returned correctly when using Environment API with paging. Operating systems support. Future Dynatrace Managed operating systems support changes.

Operating System 232

Operating System AWS Metrics Storage 232

Brendan Gregg

SEPTEMBER 5, 2017

Appliance manufacturers hire kernel engineers to develop custom features, including storage appliances. Large companies (like the one I work for) have OS teams to look after patching, releases, and performance.

Virtualization 76

Virtualization AWS Engineering Hardware 76

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on.

Performance 52

Performance Latency Hardware Storage 52

Brendan Gregg

FEBRUARY 28, 2023

Titus, the Netflix container management platform, is now open source,” [link] , Apr 2018 [Cutress 19] Dr. [link] , 2021 [Trader 21] Tiffany Trader, “Cerebras Second-Gen 7nm Wafer Scale Engine Doubles AI Performance Over First-Gen Chip ,” [link] , Apr 2021 [Ghigoff 21] Yoann Ghigoff, et al.,

Performance 52

Performance Latency Cache Virtualization 52

SQL Server According to Bob

DECEMBER 18, 2018

Durability is a cornerstone of any database system and starting with SQL Server 2017 on Linux Cumulative Update 6 (CU6), SQL Server on Linux enables “Forced Flush” behavior as described in this article , improving durability on non-Fua optimized systems. “Be Any storage device that can survive a power outage. Starting with the Linux 4.18

Servers 90

Servers Media Cache Storage 90

Percona

JUNE 4, 2023

This blog post was originally published in November 2017 and was updated in June 2023. MariaDB retains compatibility with MySQL, offers support for different programming languages, including Python, PHP, Java, and Perl, and works with all major open source storage engines such as MyRocks, Aria, and InnoDB. 3 GA 23 May 2017 10.2.6

Open Source 52

Open Source Servers Storage Scalability 52

All Things Distributed

OCTOBER 17, 2016

In the remainder of 2016 and inq 2017, we will launch another four AWS regions in Canada, China, the United Kingdom, and France, adding another nine AZs to our global infrastructure footprint. It brings the worldwide total of AWS Availability Zones (AZs) to 38, and the number of regions globally to 14.

AWS 127

AWS Cloud Infrastructure Analytics 127

Percona

AUGUST 17, 2023

StackOverflow statistics show that 26% of developers preferred it in 2017, 34% in 2019, and 40% in 2021. PostgreSQL has powerful and advanced features, including asynchronous replication, full-text searches of the database, and native support for JSON-style storage, key-value storage, and XML.

Open Source 83

Open Source Healthcare Social Media Database 83

Scalegrid

JULY 13, 2020

Recognized as the fastest growing database by popularity, PostgreSQL was named the DBMS of the year in both 2018 and 2017 by DB-Engines, and continues to grow in popularity in 2019. pg_repack – reorganizes tables online to reclaim storage. In fact, PostgreSQL is so popular, 11.5%

Open Source 244

Open Source Tuning C++ Database 244

SQL Performance

SEPTEMBER 5, 2019

This table uses columnstore for its primary storage to produce batch mode execution later on. You can skip the WHERE clauses if you are running SQL Server 2017. OPTION ( MAXDOP 1 ) ; This time, only the COUNT_BIG aggregate achieves aggregate pushdown (SQL Server 2017 only). DROP TABLE IF EXISTS #T ; GO. c1 = @Start + N.

Performance 81

Performance C++ Storage Servers 81

Alex Russell

MARCH 6, 2021

TL;DR: A lot has changed since 2017 when we last estimated a global baseline resource per-page resource budget of 130-170KiB. A then-representative $200USD device had 4-8 slow (in-order, low-cache) cores, ~2GiB of RAM, and relatively slow MLC NAND flash storage. The Moto G4 , for example. Here begins our 2021 adventure. Hard Reset.

Performance 124

Performance Network Cache Metrics 124

The Morning Paper

NOVEMBER 8, 2020

Back in 2017 the engineering team at Facebook had a problem. replacing Paxos with Raft), or they could be shims over external storage systems. A minimal Loglet needs to provide totally ordered, durable storage via the shared log API. Every little helps! The NativeLoglet.

Virtualization 126

Virtualization Latency Storage Systems 126

Percona

MARCH 30, 2023

Further, open source databases can be modified in infinite ways, enabling institutions to meet their specific needs for data storage, retrieval, and processing. Non-relational databases: Instead of tables, non-relational (NoSQL) databases use document-based data storage, column-oriented storage, and graph databases.

Open Source 79

Open Source Database Storage Scalability 79

Brendan Gregg

APRIL 14, 2022

offer letter logo (2014) flame graphs (2014) eBPF tools (2014-2019) PMC analysis (2017) my pandemic-abandoned desk (2020); office wall I joined Netflix in 2014, a company at the forefront of cloud computing with an attractive work culture. Netflix has been the best job of my career so far, and I'll miss my colleagues and the culture.

Java 52

Java Cache Cloud AWS 52

Percona

JANUARY 17, 2019

now has a version which will support parallelism for SELECT queries (utilizing the read capacity of storage nodes underneath the Aurora cluster). Aurora PQ works by doing a full table scan (parallel reads are done on the storage level). sec) mysql> explain select count(*) from ontime where flightdate > '2017-01-01'G 1.

Cache 48

Cache C++ AWS Airlines 48

SQL Server According to Bob

MAY 19, 2019

For example, Azure SQL Database Edge can already use TSQL Predict Machine Learning (ML) and Column Storage. The reduction effort started when we shipped SQL Server 2017 ( [link].) SQL Server 2017. SQL Server 2017 uses function address comparisons which the enablement of COMDAT would break. SQL Server 2019. linker version.

IoT 61

IoT Servers Development Cache 61

SQL Performance

AUGUST 27, 2019

TPC-H Benchmark Results with SQL Server 2017. HPE has already submitted a new official TPC-H result on SQL Server 2017 for a single-socket, HPE Proliant DL325 Gen10 1U server that was 743,750 QphH@1000GB. TPC-E Benchmark Results with SQL Server 2017. AMD Rome Second Generation EPYC Review: 2x 64-core Benchmarked.

Servers 52

Servers Benchmarking Virtualization Architecture 52

Percona

JULY 7, 2023

Backup and continuous archiving: PostgreSQL asynchronous replication relies on the write-ahead log (WAL) stream, and it’s vital to have centralized storage that all nodes can access. StackOverflow statistics show that 26% of developers preferred it in 2017, 34% in 2019, and 40% in 2021.

Availability 88

Availability Open Source Architecture Database 88

Percona

AUGUST 5, 2023

This blog was originally published in July 2017 and was updated in August 2023. Otherwise, the storage engine does a scatter-gather and queries ALL partitions in a UNION that is not concurrent. This method distributes data evenly across partitions to achieve balanced storage and optimal query performance.

Strategy 40

Strategy Storage Database Best Practices 40

SQL Performance

APRIL 8, 2019

Back on December 5, 2017, Microsoft announced that they were using AMD EPYC 7551 processors in their storage-optimized Lv2-Series virtual machines. They feature low latency, local NVMe storage that can directly leverage the 128 PCIe 3.0 Microsoft handles the storage account management for you when you use Azure Managed Disks.

Azure 42

Azure Virtualization Storage Benchmarking 42

Percona

SEPTEMBER 1, 2023

This blog was originally published in October 2017 and was updated in September 2023. 3) Storage engine limitations There are a few storage engine limitations that can be a bottleneck in your use case. Database architects working with MongoDB encounter specific challenges related to database systems and system growth.

Best Practices 75

Best Practices Traffic Database Servers 75

SQL Performance

MAY 1, 2020

I was a bit perplexed by this issue, since the system was certainly no slouch — plenty of cores, 3TB of memory, and XtremIO storage. Everything was on SQL Server 2017, so why was this setting so prevalent? writes/sec average throughput: 46.96 Here is a sample of the results: Partial results from CMS query.

C++ 58

C++ Database Servers Storage 58

Particular Software

MARCH 15, 2021

In it, we now offer a new tool for generating scripts for NServiceBus tables, enable support for SQL Server Always Encrypted, have begun to expand our testing to MariaDB and Amazon Aurora, have solved a problem with ever-expanding outbox storage, and added controls for managing transaction isolation level.

Transportation 52

Transportation Azure Database Servers 52

Tim Kadlec

JANUARY 9, 2019

I recently profiled a page on a Pixel 2 (released in 2017), and an Alcatel 1x (released in 2018). Ballooning bandwidth and storage have fostered complacency that we can do without. The web’s growth is being pushed forward primarily by low-cost, underpowered Android devices that frequently struggle with today’s web.

Performance 111

Performance Energy Internet Internet 111

SQL Performance

MARCH 17, 2020

It comes set to SQL Server 2008 compatibility (level 100), but we will start with a more modern setting of SQL Server 2017 (level 140): ALTER DATABASE StackOverflow2013 SET COMPATIBILITY_LEVEL = 140 ; The tests were performed on my laptop using SQL Server 2019 CU 2. The serial aggregate push down result of 13ms is equally impressive.

Servers 86

Servers Tuning Database Efficiency 86

SQL Performance

FEBRUARY 3, 2020

The downside to the latter is that, in SQL Server 2017, you cannot run this online… but you will be able to in 2019. If you want to maximize compression with zero disruption , you can approach 90% storage reduction without any disruption at all, using page compression online.

Speed 113

Speed Processing Servers Storage 113

Enprowess

JANUARY 12, 2022

Other benefits: It has other benefits like a Quicker launch to the market, Easier distribution, saving device power and storage, seamless maintenance, and updating. Twitter Lite PWA : It was integrated as the standard User Interface in 2017. Blockchain technology is an encrypted database storage system. Famous PWA Use Examples.

Blockchain 59

Blockchain IoT Development Serverless 59

Percona

JANUARY 14, 2019

To simulate text messages I have used ~3 billion of reddit comments (10 years from 2007 to 2017), downloaded from pushshift.io. In a partitioned massively parallel database system, the storage format and sorting algorithm may not be optimized for that operation as we are reading multiple partitions in parallel.

Database 40

Database Analytics Blockchain Healthcare 40

CSS Wizardry

APRIL 2, 2018

Here are my top five most-played tracks of 2017’s 70,000 minutes: However, after recently discussing one of my all-time favourite tracks with a friend, and, bizarrely enough, with Nu:Tone himself, I decided that I wanted to start taking my listening a little more seriously. At home, I’m used to just streaming Spotify through an Amazon Echo.

Website 108

Website Media Hardware Monitoring 108

SQL Performance

APRIL 23, 2019

Columnstore storage. To understand this, it is necessary to first review how columnstore storage works at a high level: A compressed row group contains a column segment for each column. Value encoding can reduce the number of bits required for storage by translating raw values using a base offset and magnitude modifier.

Servers 67

Servers Testing Metrics Storage 67

Tasktop

JUNE 25, 2020

When I joined Tasktop back in October of 2017, Tasktop was doing really well. Flow Fabric feeds a schemaless immutable storage of digital artifacts, across disciplines and functions. In the grand tradition of our sages, I’ve been contemplating existential questions about Tasktop and wrote a Bar Mitzvah speech of my own. .

Metrics 52

Metrics Education Innovation Software 52

The Morning Paper

JANUARY 19, 2020

STELLA: report from the SNAFU-catchers workshop on coping with complexity , Woods 2017, Coping with Complexity workshop. Today’s choice is a report from a 2017 workshop convened with that title, and recommended to me by John Allspaw – thank you John! Take a trivial example with three components A, B, and C.

C++ 50

C++ Systems Software Software 50

SQL Performance

AUGUST 23, 2019

The details given here apply to SQL Server 2017 — the most recently released version at the time of writing. This article is about batch mode bitmaps, which have a very different implementation. There have been major improvements since the first appearance of the batch mode execution engine in SQL Server 2012. The Query Optimizer.

Servers 75

Servers Engineering Efficiency Retail 75

Alex Russell

OCTOBER 22, 2017

— Alex Russell (@slightlylate) October 4, 2017. We now have everything we need to create a ballpark perf budget for a product in 2017. Time required to read bytes from disk (it’s not zero, even on flash-based storage!). Maybe "ambush by JS"? TTI under 2 seconds for subsequent loads. First Load.

Performance 68

Performance Benchmarking Network Mobile 68