High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Who's Hiring?

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

APRIL 28, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Try out their platform.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MARCH 30, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Try out their platform.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MAY 26, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Try out their platform.

Education 100

Education AWS Software Engineering Scalability 100

InfoQ

APRIL 15, 2024

He specifically delved into Venice DB, the NoSQL data store used for feature persistence. At the QCon London 2024 conference, Félix GV from LinkedIn discussed the AI/ML platform powering the company’s products. By Rafal Gancarz

Artificial Intelligence 105

Artificial Intelligence Big Data Data Engineering Latency 105

Abhishek Tiwari

DECEMBER 22, 2017

Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse. Classic ETL. Different audience. Challenges.

Big Data 40

Big Data Retail Storage Google 40

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Netflix TechBlog

NOVEMBER 20, 2023

For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing. Backfill: Backfilling datasets is a common operation in big data processing. data arrives too late to be useful).

Processing 210

Processing Big Data Efficiency Engineering 210

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 82

Cache Latency Traffic Database 82

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

MAY 26, 2020

These characteristics allow for an on-call response time that is relaxed and more in line with traditional big data analytical pipelines. The enriched data allows us to analyze networks across a variety of dimensions (e.g. And excellent logging is needed for debugging purposes and supportability.

Network 150

Network Tuning AWS Big Data 150

All Things Distributed

OCTOBER 7, 2015

Today, I am excited to share with you a brand new service called Amazon QuickSight that aims to simplify the process of deriving insights from a wide variety of data sources in a fast and affordable manner. Big data challenges. and “Why is my most profitable region not growing?” Enter Amazon QuickSight.

Cloud 138

Cloud Big Data AWS Analytics 138

Abhishek Tiwari

DECEMBER 27, 2017

Unified data management architecture. A unified data management (UDM) system combines the best of data warehouses, data lakes, and streaming without expensive and error-prone ETL. In contrast, Alluxio a middleware for data access - think Alluxio storage layer as fast cache. Machine learning meets data integration.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 202

Java Scalability Traffic Architecture 202

The Netflix TechBlog

SEPTEMBER 10, 2019

You can look at ABlaze (our centralized A/B testing platform) and take a quick look at how it’s performing. Note that the new encodes perform well in the lower quantiles but worse in the higher ones You notice that the first new encode (Cell 2?—?Encode Sometimes statistical models are expensive to run even on compressed data.

Metrics 215

Metrics Architecture Infrastructure Innovation 215

All Things Distributed

NOVEMBER 15, 2016

The data infrastructure to collect, store, and process data is geared primarily towards developers and IT professionals whereas insights need to be derived by not just technical professionals but also non-technical business users. Auto-discovery : One of the challenges with BI is discovering and accessing the data.

Analytics 152

Analytics Availability Media Social Media 152