The Netflix TechBlog

JUNE 15, 2022

With a goal of abstracting users away from the complexity of interacting with Elasticsearch directly, we landed on a custom Studio Search DSL reminiscent of SQL. This also leaves the possibility of swapping Elasticsearch with a different search engine in the future. But, we are not quite done yet!

Engineering 183

Engineering Architecture Availability Tuning 183

DZone

FEBRUARY 6, 2024

The metrics can be stored in various DBs, and Grafana supports most of them, like Prometheus , Zabbix, Graphite, MySQL, PostgreSQL, Elasticsearch, etc. It can query those metrics, send alerts, and can be actively used for monitoring and observability, making it a popular tool for gaining insights.

Big Data 161

Big Data Open Source Virtualization Metrics 161

Dynatrace

APRIL 11, 2022

To take advantage of performance improvements, the latest security-vulnerability enhancements, and bug fixes, we’ve upgraded JRE for Cassandra and Elasticsearch. Both, Cassandra and Elasticsearch will now use JRE 8u322. Other components will now use JRE 11.0.14.

Storage 203

Storage Metrics Performance 203

Percona Community

OCTOBER 3, 2018

For Percona Live Europe, I’ll be presenting the tutorial Elasticsearch 101 alongside my colleagues and fellow presenters from ObjectRocket Alex Cercel, DBA, and Mihai Aldoiu, Data Engineer. In this tutorial, we’ll give you a hands-on introduction to Elasticsearch and give you a glimpse at some of the fundamental concepts.

Data Engineering 52

Data Engineering Scalability Engineering 52

The Netflix TechBlog

JUNE 1, 2023

We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch. We used Elasticsearch dashboards to analyze results. It also included metadata about ads, such as ad placement and impression-tracking events.

Traffic 342

Traffic Best Practices Systems Testing 342

DZone

MAY 19, 2023

Logs often take up the majority of a company's data assets. Examples of logs include business logs (such as user activity logs) and Operation and Maintenance logs of servers, databases, and network or IoT devices. Logs are the guardian angel of business.

IoT 130

IoT Analytics Network Database 130

The Netflix TechBlog

JANUARY 25, 2023

For supporting different search requirements we chose ElasticSearch. Different types of search queries are supported using the Elasticsearch as a backend search engine. We use ElasticSearch to support stem search for 34 different languages. We support semantic search using Open Distro for ElasticSearch .

Scalability 283

Scalability Latency Media Architecture 283

Dynatrace

JUNE 29, 2021

To take advantage of performance improvements, the latest security-vulnerability enhancements, and bug fixes, we’ve upgraded the following cluster node components: Cassandra and Elasticsearch will now use JRE 8u292. Elasticsearch engine is now based on version 7.10. Other components will now use JRE 11.0.11.

Engineering 198

Engineering Systems Performance 198

Dynatrace

JULY 1, 2022

To take advantage of performance improvements, the latest security-vulnerability enhancements, and bug fixes, we’ve upgraded JRE for the following cluster node components: Cassandra and Elasticsearch will now use JRE 8u332. Other components will now use JRE 11.0.15. Storage service upgrade to Cassandra 3.11.13.

Operating System 199

Operating System Storage Metrics Systems 199

DZone

JULY 13, 2021

Splunk, when it was first released, was known as Google Search for logs, and it was followed by others such as Elasticsearch and Sumo Logic. Then came log managers, providing log aggregation and making logs easier to search.

Google 255

Google Architecture Efficiency Cloud 255

Dynatrace

FEBRUARY 8, 2022

To take advantage of performance improvements, the latest security-vulnerability enhancements, and bug fixes, we’ve upgraded JRE for the following cluster node components: Cassandra and Elasticsearch will now use JRE 8u312. Other components will now use JRE 11.0.13. OpenID Connect (SSO).

Monitoring 209

Monitoring Performance 209

DZone

JANUARY 8, 2020

Data related to monitoring is stored in RAM and LevelDB nevertheless data can be stored to other storage systems such as ElasticSearch, InfluxDb, and others, [link]. Prometheus is an open-source system monitoring and alerting toolkit.

Monitoring 179

Monitoring Open Source Storage Systems 179

The Netflix TechBlog

MARCH 10, 2023

In this blog, we will be focusing on how we utilize Elasticsearch for indexing and search the assets. The second layer is Elasticsearch , which is used to discover assets based on user queries. One of the first decisions when integrating with Elasticsearch is designing the indices, their settings and mappings.

Strategy 258

Strategy Cache Analytics Storage 258

The Netflix TechBlog

JULY 30, 2019

Decoupling Elasticsearch from Persistence This inter-dependency was removed by moving the indexing layer into separate persistence modules, exposing a property ( workflow.elasticsearch.instanceType ) to choose the type of indexing engine. Bulk APIs enable the operators to have macro-level control on the workflows executing within the system.

Lambda 189

Lambda Media Open Source Metrics 189

DZone

OCTOBER 1, 2023

Elasticsearch is an open-source search engine and analytics store used by a variety of applications from search in e-commerce stores, to internal log management tools using the ELK stack (short for “Elasticsearch, Logstash, Kibana”).

Open Source 247

Open Source Tuning Analytics Scalability 247

The Netflix TechBlog

SEPTEMBER 29, 2022

Secondary indexes For observability purposes, we capture information about incoming messages and their transition between states in two secondary indexes maintained on Elasticsearch. Two Flink jobs — one for each type of index we maintain — consume the events from the corresponding Kafka topics, and update the indexes in Elasticsearch.

Latency 212

Latency Systems Media Serverless 212

Dynatrace

OCTOBER 20, 2020

To free disk space, log files related to metric and configuration storage, in addition to Elasticsearch log files, are now automatically deleted after two months. Backup improvements: To prevent full disk space utilization, Elasticsearch data backups are now paused until Dynatrace removes older data snapshots. Dynatrace news.

Storage 102

Storage Metrics Systems 102

DZone

DECEMBER 3, 2019

You may also like: Using the Telegraf Elasticsearch Input Plugin. Telegraf is an agent that runs on your operating system of choice, schedules gathering metrics and events from various sources and then sends them to one or more sinks, such as InfluxDB or Kafka. For InfluxDB, version 1.x, as well as InfluxDB Cloud are supported.

Hardware 147

Hardware Operating System Metrics Cloud 147

Dynatrace

NOVEMBER 19, 2021

Reduce Elasticsearch backup snapshots retention to 5 days. Dynatrace news. New features and enhancements. Old snapshots will be automatically removed after the upgrade. Problems API v2 now includes event evidence data as part of the event details. With this release, svchost.exe is excluded from injection.

Metrics 201

Metrics Monitoring Traffic Servers 201

The Netflix TechBlog

OCTOBER 19, 2020

We started with Elasticsearch as our data store due to its flexible data model and querying capabilities. The increased operational burden of scaling ElasticSearch clusters due to high data write rate became painful for us. Storage: don’t break the bank! The high data ingestion rate eventually degraded both read and write operations.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

SEPTEMBER 21, 2020

Cassandra and Elasticsearch will now use JRE 8u265. The JRE was updated to take advantage of performance improvements , the latest security-vulnerability enhancements and bug fixes. All services will be automatically restarted during the upgrade. Other components will now use JRE 11.0.8.

Monitoring 105

Monitoring Performance 105

Dynatrace

JANUARY 16, 2023

Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch. Organizations increasingly use the flexibility and elasticity of Kubernetes to run CI and CD jobs as well as their control planes. Security : For security, organizations mostly use policy checkers and enforcers, such as Gatekeeper.

Open Source 340

Open Source Java Operating System Programming 340

Dynatrace

JULY 30, 2020

Cassandra, Elasticsearch, ActiveGate, Server, and NodeKeeper get their own dedicated JRE. Multi-environment ActiveGates are now displayed in a dedicated tile on the CMC Home page and are available on the Deployment status page. Installation and upgrade. Requirements on binary data are now increased by 1GB. NGINX was updated to version 1.17.8.5

Hardware 104

Hardware Java Innovation Metrics 104

Dynatrace

JUNE 8, 2020

CPU usage and suspension rates for cluster node processes (namely Server, Cassandra, (embedded) ActiveGate, and Elasticsearch) are also visualized. Metrics are provided for general host info like CPU usage and memory consumption, OneAgent traffic, and network latency.

Software 165

Software Software Programming Metrics 165

The Netflix TechBlog

OCTOBER 15, 2019

providing advanced search capabilities (ElasticSearch etc.), Also, certain systems like ElasticSearch, do not support XA or any other heterogeneous transaction model. caching (Memcached etc.), Typically when using multiple datastores, one of them acts as the primary store, and the others as derived stores.

Transportation 192

Transportation Architecture Processing Storage 192

Dynatrace

MAY 15, 2019

Revised cleanup strategy for Elasticsearch snapshots. Previously, we retained hourly snapshots of Elasticsearch for 24 hours, daily snapshots for a week, and weekly snapshots for a month. This enables you to collect more unique metrics within a shorter period of time without increasing your overall licensed quota.

Storage 102

Storage Metrics Strategy Java 102

The Netflix TechBlog

JANUARY 13, 2022

And most of the Flink jobs run under a managed platform called Keystone , which abstracts out the underlying Flink job details and allows users to consume data from Apache Kafka streams and publish them to different data stores like Elasticsearch and Apache Iceberg on AWS S3.

Big Data 238

Big Data Infrastructure Metrics Hardware 238

Scalegrid

OCTOBER 2, 2019

According to the 2019 Open Source Database Report , a majority of Redis™ deployments are used in conjunction with MySQL, and over half of Redis™ deployments are used with either PostgreSQL, MongoDB, and Elasticsearch.

Azure 100

Azure Availability Open Source Virtualization 100

Percona

APRIL 12, 2023

licensed source code in Elasticsearch and Kibana to be dual licensed under the Elastic License and Server Side Public License (SSPL), giving users the choice of which license to apply. For instance, Elastic announced the following ( Elastic blog, Doubling down on open, Part II, January 2021 ): “We are moving our Apache 2.0-licensed

Open Source 83

Open Source Programming Database Servers 83

Dynatrace

JUNE 7, 2019

Before each upgrade, we perform sanity checks for Cassandra, Elasticsearch, and transaction datastores to ensure that they have at least 1 GB of free space available. The previous, lower values are still used during upgrades for backward compatibility. The new, higher values are applied for fresh installations. Other improvements.

Hardware 102

Hardware Traffic Servers Monitoring 102

Alex Podelko

JUNE 12, 2019

Often they get integrated with the TICK stack (Telegraf, InfluxDB, Chronohraf, Kapasitor) , the ELK stack (Elasticsearch, Logstash, Kibana) , Prometheus , and Grafana. Another trend is to integrate them with other products to provide needed functionality – such as monitoring, analysis, and reporting.

Testing 243

Testing Open Source Testing Tools Performance Testing 243

The Netflix TechBlog

MARCH 7, 2023

Instead our challenge was to implement this feature on top of Cassandra and ElasticSearch databases because that’s what Marken uses. We use Cassandra as a source of truth where we store the annotations while we index annotations in ElasticSearch to provide rich search functionalities. Write algo runs into files.

Media 264

Media Latency Architecture Database 264

Dynatrace

NOVEMBER 8, 2019

Let’s say you have a Java application monitored by OneAgent, but your Elasticsearch cloud database service can’t be monitored by OneAgent. Now, with the ActiveGate plugin framework, you can write your own plugin to integrate your Elasticsearch cloud database service with Dynatrace. Davis also found a responsible deployment event.

Java 35

Java Metrics Monitoring Cloud 35

The Netflix TechBlog

APRIL 12, 2022

Using the type information present in the GraphQL query template and the user specified index configuration we were able to create an index template with a set of custom Elasticsearch text analyzers that generalized well across domains. Similarly the generation of the Elasticsearch template was time consuming and error prone.

Engineering 263

Engineering Architecture Java Infrastructure 263

SQL Performance

JULY 8, 2021

Elasticsearch ). What To Use Instead. SQL Server’s Full-Text Search seems like it could help, but I’ve personally never used it. In practice, I’ve only seen success in solutions outside of SQL Server (e.g. Conclusion.

Design 92

Design Software Architecture Database Servers 92

All Things Distributed

DECEMBER 8, 2016

The fast-growing, Canadian startup uses Amazon EC2, Amazon S3, Amazon Elasticsearch Service, Amazon RDS, and AWS OpsWorks to scale quickly while securely delivering seamless video upload and playback services for its customers. AWS as your strategic cloud provider.

AWS 155

AWS Cloud Lambda Innovation 155

Strategic Tech

JANUARY 19, 2020

If your product relied on advanced search capabilities to differentiate from the competition, the arrival of state-of-the-art open source and SaaS search engines like Elasticsearch would have destroyed your advantage, giving any potential competitor the capabilities to compete with you. An example of a commoditised core is a search engine.

Open Source 87

Open Source Games Innovation Strategy 87