article thumbnail

Write Optimized Spark Code for Big Data Applications

DZone

Apache Spark is a powerful open-source distributed computing framework that provides a variety of APIs to support big data processing. In addition, pySpark applications can be tuned to optimize performance and achieve better execution time, scalability, and resource utilization.

Big Data 161
article thumbnail

How Netflix uses eBPF flow logs at scale for network insight

The Netflix TechBlog

By Alok Tiagi , Hariharan Ananthakrishnan , Ivan Porto Carrero and Keerti Lakshminarayan Netflix has developed a network observability sidecar called Flow Exporter that uses eBPF tracepoints to capture TCP flows at near real time. Without having network visibility, it’s difficult to improve our reliability, security and capacity posture.

Network 325
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Hyper Scale VPC Flow Logs enrichment to provide Network Insight

The Netflix TechBlog

Without having network visibility, it’s not possible to improve our reliability, security and capacity posture. Network Availability: The expected continued growth of our ecosystem makes it difficult to understand our network bottlenecks and potential limits we may be reaching. 43416 5001 52.213.180.42

Network 150
article thumbnail

Python at Netflix

The Netflix TechBlog

Open Connect Open Connect is Netflix’s content delivery network (CDN). video streaming) takes place in the Open Connect network. The network devices that underlie a large portion of the CDN are mostly managed by Python applications. If any of this interests you, check out the jobs site or find us at PyCon. are you logged in?

article thumbnail

Optimizing data warehouse storage

The Netflix TechBlog

Orient: Gather tuning parameters for a particular table that changed. AutoAnalyze In short, AutoAnalyze finds the best tuning/configuration parameters for a table. AutoOptimize reduces end to end lag in data processing by optimizing as we go. Also, respond to ad-hoc requests created manually by end-users.

Storage 203
article thumbnail

Web Performance Bookshelf

Rigor

Take, for example, The Web Almanac , the golden collection of Big Data combined with the collective intelligence from most of the authors listed below, brilliantly spearheaded by Google’s @rick_viscomi. High Performance Browser Networking. Web Performance Tuning. ” – Andy King, 2003. Time is Money.

article thumbnail

Bringing the Magic of Amazon AI and Alexa to Apps on AWS.

All Things Distributed

in ML and neural networks) and access to vast amounts of data. Effectively applying AI involves extensive manual effort to develop and tune many different types of machine learning and deep learning algorithms (e.g. This technology enables high naturalness, consistent intonation, and accurate processing of texts.

AWS 165