Scalegrid

MAY 13, 2020

It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. This feature-packed database provides powerful and rapid analytics on data that scales up to petabyte volumes.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

The Netflix TechBlog

FEBRUARY 9, 2022

JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. Netflix homegrown CLI tool for data pipeline management. task, an atomic unit of data transformation logic, a non-separable execution block in the workflow chain.

Storage 201

Storage Data Engineering Testing Code 201

Scalegrid

JULY 17, 2020

It is an open standard format which organizes data into key/value pairs and arrays detailed in RFC 7159. JSON is the most common format used by web services to exchange data, store documents, unstructured data, etc. You can also check out our Working with JSON Data in PostgreSQL vs. JSONB Patterns & Antipatterns.

Storage 321

Storage Database Efficiency Processing 321

Dynatrace

SEPTEMBER 7, 2022

Dynatrace and the Dynatrace Intelligent Observability Platform have added support for the newly introduced Amazon VPC Flow Logs to Amazon Kinesis Data Firehose. This support enables customers to define specific endpoint delivery of real-time streaming data to platforms such as Dynatrace. What is VPC Flow Logs? Why Dynatrace?

Traffic 215

Traffic AWS Network Cloud 215

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Efficiency 103

Efficiency Engineering Design Storage 103

The Netflix TechBlog

FEBRUARY 9, 2021

Growth Engineering at Netflix?—?Automated In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on the topic: Growth Engineering at Netflix? Accelerating Innovation.

Engineering 165

Engineering Storage Latency Entertainment 165

DZone

MAY 5, 2023

This blog post explores the benefits of combining both open-source frameworks, shows unique differentiators of Flink versus Kafka, and discusses when to use a Kafka-native streaming engine like Kafka Streams instead of Flink.

Open Source 246

Open Source Storage Innovation Engineering 246

Dynatrace

MAY 7, 2024

Fluent Bit is a telemetry agent designed to receive data (logs, traces, and metrics), process or modify it, and export it to a destination. Fluent Bit can serve as a proxy before you send data to Dynatrace or similar. However, you can also use Fluent Bit as a processor because you can perform various actions on the data.

Best Practices 231

Best Practices IoT Metrics Storage 231

Dynatrace

NOVEMBER 28, 2023

In the previous installment of this blog series , we explored how to set up Dynatrace as a build-stage orchestrator to effectively address the challenges faced by Site Reliability Engineers (SREs). Davis AI can leverage this data to enable predictive analysis. This approach effectively combats configuration drift.

Traffic 262

Traffic Best Practices Strategy Engineering 262

Dynatrace

MARCH 23, 2020

A decade ago, while working for a large hosting provider, I led a team that was thrown into turmoil over the purchasing of server and storage hardware in preparation for a multi-million dollar super-bowl ad campaign. Our procurement decisions were based on trace data that was pulled from a handful of fragmented monitoring solutions.

Google 193

Google Engineering Metrics Hardware 193

The Netflix TechBlog

JUNE 25, 2021

By Gim Mahasintunan on behalf of Data Platform Engineering. Supporting a rapidly growing base of engineers of varied backgrounds using different data stores can be challenging in any organization. In this blog post, we are thrilled to share that we are open-sourcing one such tool: the Netflix Data Explorer.

Metrics 152

Metrics Best Practices Strategy Design 152

The Netflix TechBlog

APRIL 27, 2023

Engineers want their alerting system to be realtime, reliable, and actionable. A few years ago, we were paged by our SRE team due to our Metrics Alerting System falling behind — critical application health alerts reached engineers 45 minutes late! Data expressions define what data needs to be sourced in order to evaluate a query.

Storage 288

Storage Cache Metrics Database 288

The Netflix TechBlog

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. The pandemic, however, posed unique challenges in conducting a conventional in-lab subjective test with HDR encodes.

Open Source 252

Open Source Software Engineering Internet Internet 252

Dynatrace

APRIL 8, 2022

Fluentd is an open-source data collector that unifies log collection, processing, and consumption. Output plugins deliver logs to storage solutions, analytics tools, and observability platforms like Dynatrace. Built-in resiliency ensures data completeness and consistency even if Fluentd or an endpoint service goes down temporarily.

Metrics 179

Metrics Analytics Software Architecture Open Source 179

Percona

DECEMBER 19, 2023

PG_TDE (PostgreSQL Transparent Data Encryption) is an experimental transparent data encryption access method for PostgreSQL 16, available on GitHub as an extension for vanilla PostgreSQL. Today, we’d like to share more details about the extension, including information about its internal architecture.

Open Source 85

Open Source Storage Database C++ 85

Percona

AUGUST 9, 2023

MySQL 8 brought a significant architectural transformation by replacing the traditional MyISAM-based system tables with the Transaction Data Dictionary (TDD), a more efficient and reliable approach. The primary goal of the Transaction Data Dictionary (TDD) is to enhance the overall performance, stability, and scalability of MySQL databases.

Database 80

Database Storage Scalability Games 80

Percona

SEPTEMBER 14, 2023

However, modern business is very much dependent on analytical data. I recommend you read our previous blog, Using ClickHouse as an Analytic Extension for MySQL , from Vadim to know more about this. In this blog post, I will show you how to replicate MySQL data in real-time to ClickHouse. We can do the filter as well.)

Engineering 91

Engineering Database Servers Analytics 91

Dynatrace

JULY 29, 2022

Log analytics is the process of viewing, interpreting, and querying log data so developers and IT teams can quickly detect and resolve application and system issues. At the same time, log analytics can present challenges as data volumes explode, particularly in traditional environments that lack end-to-end observability solutions.

Analytics 214

Analytics Storage Retail Monitoring 214

Dynatrace

OCTOBER 4, 2022

Several pain points have made it difficult for organizations to manage their data efficiently and create actual value. Limited data availability constrains value creation. Modern IT environments — whether multicloud, on-premises, or hybrid-cloud architectures — generate exponentially increasing data volumes.

Analytics 234

Analytics Artificial Intelligence Storage Serverless 234

Percona

MARCH 21, 2023

In this blog, I’ll discuss the use case for replication. We want to improve our ability to replicate your data and limit replication to row-based events securely, wherein we do not have control over the source(s). PRIVILEGE CHECKS USER is helpful in multi-source replication to carefully gather data from various sources.

Storage 89

Storage Open Source Database Engineering 89

Dynatrace

JUNE 25, 2020

With Dynatrace actively managing business-critical applications, some of our globally distributed enterprise customers require Dynatrace Managed to continue operating even when an entire data center goes down. Near-zero RPO and RTO—monitoring continues seamlessly and without data loss in failover scenarios.

Availability 247

Availability Hardware Latency Traffic 247

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which In our previous blog post we introduced Edgar, our troubleshooting tool for streaming sessions. We needed to increase engineering productivity via distributed request tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

JULY 29, 2022

Log analytics is the process of viewing, interpreting, and querying log data so developers and IT teams can quickly detect and resolve application and system issues. At the same time, log analytics can present challenges as data volumes explode, particularly in traditional environments that lack end-to-end observability solutions.

Analytics 181

Analytics Storage Retail Monitoring 181

The Netflix TechBlog

SEPTEMBER 24, 2021

Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side. As described by the white paper Apple ProRes ( link ), the target data rate of the Apple ProRes HQ for 1920x1080 at 29.97 is 220 Mbps.

Cloud 237

Cloud Media Storage Cache 237

Dynatrace

APRIL 8, 2022

Fluentd is an open-source data collector that unifies log collection, processing, and consumption. Output plugins deliver logs to storage solutions, analytics tools, and observability platforms like Dynatrace. Built-in resiliency ensures data completeness and consistency even if Fluentd or an endpoint service goes down temporarily.

Metrics 130

Metrics Analytics Software Architecture Open Source 130

Percona

APRIL 20, 2023

Data-at-rest encryption is essential for compliance with regulations that require the protection of sensitive data. It is also critical for securing sensitive data and avoiding data breaches. PostgreSQL does not natively support Transparent Data Encryption (TDE). apiVersion: storage.k8s.io/v1

Storage 123

Storage AWS Cloud Database 123

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. Data Store. Improving data processing.

Serverless 280

Serverless AWS Lambda Storage 280

Dynatrace

FEBRUARY 10, 2023

Data proliferation—as well as a growing need for data analysis—has accelerated. According to recent Dynatrace data, 59% of CIOs say the increasing complexity of their technology stack could soon overload their teams without a more automated approach to IT operations. Digital transformation shows no signs of slowing down.

Efficiency 276

Efficiency Performance Analytics DevOps 276

Percona

MARCH 7, 2023

One approach is to separate compute and storage to allow for independent scaling. By separating storage and compute, Neon replaces the PostgreSQL storage layer with data nodes, and compute nodes are distributed across a cluster of nodes. Compute nodes are stateless PostgreSQL nodes backed by the Neon storage engine.

Serverless 113

Serverless Storage Open Source Architecture 113

Percona

MARCH 29, 2023

Compression in any database is necessary as it has many advantages, like storage reduction, data transmission time, etc. Storage reduction alone results in significant cost savings, and we can save more data in the same space. In this blog, we will discuss both data and network-level compression offered in MongoDB.

Storage 111

Storage Network Open Source Latency 111

Percona

OCTOBER 13, 2023

A common practice among DBAs and developers is to copy table data and.frm files from the data dictionary. In the case of InnoDB tablespaces, this information is incorporated into the tablespace itself, creating a fusion of metadata and data primarily to enhance performance. In MySQL 8.0, ibd files), general tablespace files (.ibd

Storage 96

Storage Database Engineering Open Source 96

Percona

JANUARY 5, 2023

Space constraint has been an endless and painstaking process for many of us, especially in systems that have a high number of transactions and data growth exceeding hundreds of GBs in a matter of days. MySQL won’t shrink tablespace and storage won’t be released to the file system. What is partitioning? 1 mysql mysql 9.0G

Traffic 61

Traffic Engineering Speed Storage 61

The Netflix TechBlog

MARCH 10, 2023

are stored in secure storage layers. In this blog, we will be focusing on how we utilize Elasticsearch for indexing and search the assets. Amsterdam is built on top of three storage layers. This is the layer we’d like to focus on in this blog. The first layer, Cassandra , is the source of truth for us.

Strategy 258

Strategy Cache Analytics Storage 258

The Netflix TechBlog

MARCH 14, 2023

Earlier we shared the details of one of these algorithms , introduced how our platform team is evolving the media-specific machine learning ecosystem , and discussed how data from these algorithms gets stored in our annotation service. Dawn Chenette , Design Lead This approach had several benefits for product engineering. Incredible!”

Media 291

Media Innovation Energy Architecture 291

Percona

AUGUST 29, 2023

Its query engine not only parallelizes incoming SQL queries for time series but also has the ability to create column-wise tables making it ideal for OLAP analysis duties. The distributed query engine can route and parallelize SELECT, DML, and other operations on distributed tables across the cluster.

C++ 82

C++ Database Storage Testing 82

Dynatrace

MAY 26, 2021

From May 17 to May 18, 2021, the Open-Source Engineering team at Dynatrace attended the virtual observability conference, o11yfest. This blog dives into what we learned by listening to experts in the field and the key takeaways we gathered from the talks. A recurring theme was Site Reliability Engineering. We second that.

AWS 212

AWS Strategy Storage Metrics 212

The Netflix TechBlog

SEPTEMBER 2, 2020

For engineers, instead of whodunit, the question is often “what failed and why?” An engineer can find herself digging through logs, poring over traces, and staring at dozens of dashboards. In an earlier blog post, we discussed Telltale , our health monitoring system. The more complex a system, the more places to look for clues.

Latency 296

Latency Transportation Engineering Traffic 296