AWS Data Engineering Processing Scalability 
The Netflix TechBlog
MARCH 7, 2024
For ETL and other heavy lifting of data, we mainly rely on Apache Spark. In addition to Spark, we want to support last-mile data processing in Python, addressing use cases such as feature transformations, batch inference, and training. We use Apache Arrow to decode Parquet and to host an in-memory representation of data.
The Netflix TechBlog
NOVEMBER 22, 2019
by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Netflix TechBlog
NOVEMBER 22, 2019
by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.
The Netflix TechBlog
MARCH 5, 2019
All these micro-services are currently operated in AWS cloud infrastructure. As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. Give us a holler if you are interested in a thought exchange.
High Scalability
MAY 26, 2020
Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years. The result is a process that doesn't get you the best employees you could.
The Netflix TechBlog
NOVEMBER 22, 2019
by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.
All Things Distributed
AUGUST 24, 2012
Werner Vogels weblog on building scalable and robust distributed systems. The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. Comments ().
Expert insights. Personalized for you.
226 
Let's personalize your content