Data Engineering Database Development Infrastructure 
The Netflix TechBlog
MARCH 25, 2019
Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can
The Netflix TechBlog
MARCH 5, 2019
Central engineering teams enable this operational model by reducing the cognitive burden on innovation teams through solutions related to securing, scaling and strengthening (resilience) the infrastructure. All these micro-services are currently operated in AWS cloud infrastructure.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Netflix TechBlog
DECEMBER 3, 2022
Dataflow Dataflow is a command line utility built to improve experience and to streamline the data pipeline development at Netflix. The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. test_sparksql_write.py
O'Reilly
FEBRUARY 18, 2020
This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. Software architecture, infrastructure, and operations are each changing rapidly. Also: infrastructure and operations is trending up, while DevOps is trending down.
The Netflix TechBlog
DECEMBER 21, 2020
At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. Some of the optimizations are prerequisites for a high-performance data warehouse. Both automatic (event-driven) as well as manual (ad-hoc) optimization.
All Things Distributed
NOVEMBER 15, 2016
As I mentioned, we live in a world where massive volumes of data are being generated, every day, from connected devices, websites, mobile apps, and customer applications running on top of AWS infrastructure. Put simply, data is not always readily available and accessible to organizational end users.
All Things Distributed
OCTOBER 7, 2015
However, the data infrastructure to collect, store and process data is geared toward developers (e.g., Amazon Redshift, DynamoDB, Amazon EMR) whereas insights need to be derived by not just developers but also non-technical business users. SPICE automatically replicates data for high availability and performance.
Expert insights. Personalized for you.
219 
Let's personalize your content