article thumbnail

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

The Netflix TechBlog

Meson was based on a single leader architecture with high availability. We want users to rely on shared templates and reuse their workflow definitions across their team, saving time and effort on creating the same functionality. Figure 1 shows the high-level architecture. With the high growth of workflows in the past few years?

Java 202
article thumbnail

Reinventing Performance Testing: New Architectures

Alex Podelko

New Architectures (this post). Cloud seriously impacts system architectures that has a lot of performance-related consequences. Cloud and virtualization triggered appearance dynamic, auto-scaling architectures, which significantly impact getting and analyzing feedback. – Cloud. – Agile. – New Technologies.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

PostgreSQL Connection Pooling: Part 2 – PgBouncer

Scalegrid

pgBouncer Architecture | Source. It recommends using other common linux tools like HAProxy to create an architecture which does support these features. Take a look at the sample PostgreSQL architecture for load-balanced reads below: PgBouncer architecture for load balanced reads | Source. Pool definitions.

Cache 246
article thumbnail

Evolving from Rule-based Classifier: Machine Learning Powered Auto Remediation in Netflix Data…

The Netflix TechBlog

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. We have also noted a great potential for further improvement by model tuning (see the section of Rollout in Production).

Tuning 210
article thumbnail

Conducting log analysis with an observability platform and full data context

Dynatrace

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Seamless integration.

Analytics 193
article thumbnail

Rebuilding Netflix Video Processing Pipeline with Microservices

The Netflix TechBlog

Since then, the video pipeline has undergone substantial improvements and broad expansions: Starting with Standard Dynamic Range (SDR) at Standard-Definitions , we expanded the encoding pipeline to 4K and High Dynamic Range (HDR) which enabled support for our premium offering. depending on the use case.

article thumbnail

How Netflix Content Engineering makes a federated graph searchable

The Netflix TechBlog

In a federated graph architecture, how can we answer such a query given that each entity is served by its own service? Architecture Putting this all together, below you can see a simplified view of the architecture. By transacting with a database which is monitored by a CDC connector that creates events, or b.