Data Engineering Related Topics 
DZone
FEBRUARY 27, 2023
It's midnight in the dim and cluttered office of The New York Times, currently serving as the "situation room." A powerful surge of traffic is inevitable. During every major election, the wave would crest and crash against our overwhelmed systems before receding, allowing us to assess the damage
The Netflix TechBlog
JULY 15, 2021
Data Engineers of Netflix?—?Interview Interview with Kevin Wylie This post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Data Engineers of Netflix?—?Interview
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Netflix TechBlog
OCTOBER 28, 2021
Data Engineers of Netflix?—?Interview Interview with Pallavi Phadnis This post is part of our “ Data Engineers of Netflix ” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix.
The Netflix TechBlog
JUNE 1, 2021
Data Engineers of Netflix?—?Interview Interview with Dhevi Rajendran Dhevi Rajendran This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix.
The Netflix TechBlog
JUNE 1, 2021
Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix.
The Netflix TechBlog
DECEMBER 3, 2022
by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. A large number of our data users employ SparkSQL, pyspark, and Scala.
DZone
JANUARY 13, 2023
In this tutorial, you are going to learn about QuestDB SQL extensions which prove to be very useful with time-series data. Using some sample data sets, you will learn how designated timestamps work and how to use extended SQL syntax to write queries on time-series data. Introduction Traditionally, SQL has been used for relational databases and data warehouses. sql databases data engineering
DZone
FEBRUARY 8, 2022
While increasing both the precision and the recall of our secrets detection engine, we felt the need to keep a close eye on speed. So it wasn’t a surprise to find that our engine had the same problem: more power, less speed. performance data engineering benchmarking scanning
Simform
JANUARY 8, 2023
Data ingestion is the foremost layer in a data engineering pipeline, acting as a vital pillar in the overall analytics architecture. Thus, it is essential to implement data ingestion just right. Data Engineering
The Netflix TechBlog
FEBRUARY 9, 2022
JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. Netflix homegrown CLI tool for data pipeline management. by Sam Setegne, Jai Balani, Olek Gorajek Glossary asset ?—?any
Simform
FEBRUARY 14, 2023
Stream processing has become a core part of enterprise data architecture today due to the explosive growth of data from sources such as IoT sensors, security logs, and web applications. Data Engineering
The Netflix TechBlog
SEPTEMBER 18, 2020
But there is far less agreement on what that term “data analytics” actually means?—?or Even within Netflix, we have many groups that do some form of data analysis, including business strategy and consumer insights. When you think about data at Netflix, what comes to mind?
Simform
JANUARY 25, 2023
Businesses can unlock the value of data only after it is transformed into actionable insights and when those insights are delivered promptly. But implementing such robust data pipelines can be complex and challenging. Data Engineering
DZone
DECEMBER 21, 2020
Setting up a data warehouse is the first step towards fully utilizing big data analysis. Still, it is one of many that need to be taken before you can generate value from the data you gather. An important step in that chain of the process is data modeling and transformation. This is where data is extracted, transformed, and loaded (ETL) or extracted, loaded, and transformed (ELT).
The Netflix TechBlog
SEPTEMBER 18, 2020
and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley.
The Netflix TechBlog
MAY 26, 2020
By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, Custom Exporter Agents, etc, we can provide Network Insight to users through multiple data visualization techniques like Lumen , Atlas , etc.
The Netflix TechBlog
DECEMBER 18, 2020
I wasn’t even entirely sure what the right role fit would be and originally applied for a different position, before being redirected to the Analytics Engineer role. Working in Studio Data Science & Engineering (“Studio DSE”) was basically a dream come true.
The Netflix TechBlog
OCTOBER 30, 2020
I’m a Senior Analytics Engineer on the Content and Marketing Analytics Research team. Being an Analytics Engineer is like being a hybrid of a librarian ?? One of my favorite things about being an Analytics Engineer is the variety.
Uber Engineering
AUGUST 14, 2019
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Engineering Data Infrastructure data science Data Warehouse Engineering Efficiency
InfoQ
APRIL 7, 2021
Recently, Google announced A2 Virtual Machines (VMs)' general availability based on the NVIDIA Ampere A100 Tensor Core GPUs in Compute Engine.
DZone
JANUARY 27, 2023
Data lineage, an automated visualization of the relationships for how data flows across tables and other data assets, is a must-have in the data engineering toolbox. Not only is it helpful for data governance and compliance use cases, but it also plays a starring role as one of the five pillars of data observability.
Abhishek Tiwari
JULY 12, 2018
Building data pipelines can offer strategic advantages to the business. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects. In this post, we will discuss why you should avoid building data pipelines in first place. If built correctly, data pipelines can offer strategic advantages to the business. Depending on frameworks, data processing units (a.k.a
Abhishek Tiwari
DECEMBER 27, 2017
Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Containerized data workloads running on Kubernetes offer several advantages over traditional virtual machine/bare metal based data workloads including but not limited to.
Abhishek Tiwari
DECEMBER 27, 2017
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures.
Abhishek Tiwari
APRIL 14, 2018
Developing Extract–transform–load (ETL) workflow is a time-consuming activity yet a very important component of data warehousing process. It enables data warehouse teams to ask questions like how good is the current or proposed ETL workflow design, is the workflow resilient to occasional failures, what part of the workflow can be parallelized, are there any variants of ETL workflow, and if so is one variant is better than other. Query-based Data Warehousing Tool ??.
Abhishek Tiwari
DECEMBER 23, 2017
Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships. This can be time-consuming and bit challenging if you are dealing with hundreds of data sources and thousands of event types (see my recent article on ELT architecture ). Various data integration solution providers are trying to capitalize on this gap by offering various machine learning based features to overcome these challenges.
Abhishek Tiwari
DECEMBER 22, 2017
Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse. Often post-extraction data is staged in intermediate tables which is followed by transformation and load steps to migrate data into a target database or data warehouse.
DZone
FEBRUARY 20, 2023
In software engineering, we've learned that building robust and stable applications has a direct correlation with overall organization performance. The data community is striving to incorporate the core concepts of engineering rigor found in software communities but still has further to go.
O'Reilly
FEBRUARY 18, 2020
It’s also the data source for our annual usage study, which examines the most-used topics and the top search terms. [1]. This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers.
O'Reilly
JULY 15, 2020
Software engineers comprise the survey audience’s single largest cluster, over one quarter (27%) of respondents (Figure 1). Adding architects and engineers, we see that roughly 55% of the respondents are directly involved in software development. Microservices seem to be everywhere.
Percona Community
OCTOBER 14, 2020
Percona Live Online Agenda Slot: Tue 20 Oct • New York 12:30 p.m. London 5:30 p.m. • New Delhi 10:00 p.m. Singapore 12:30 a.m.
High Scalability
JUNE 15, 2020
This is a guest post by Eunice Do , Data Engineer at TripleLift , a technology company leading the next generation of programmatic advertising. The system is the data pipeline at TripleLift.
O'Reilly
FEBRUARY 2, 2020
First, the behavior of an AI application depends on a model , which is built from source code and training data. A model isn’t source code, and it isn’t data; it’s an artifact built from the two. You need a repository for models and for the training data.
Percona Community
OCTOBER 3, 2018
For Percona Live Europe, I’ll be presenting the tutorial Elasticsearch 101 alongside my colleagues and fellow presenters from ObjectRocket Alex Cercel, DBA, and Mihai Aldoiu, Data Engineer.
The Netflix TechBlog
JULY 21, 2022
We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. Secondly, and more importantly, the sheer volume of the runtime data is a lot.
The Netflix TechBlog
DECEMBER 21, 2020
By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.
The Netflix TechBlog
JANUARY 11, 2022
Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation. What other data and intuition can I bring to the problem?”
The Netflix TechBlog
JUNE 6, 2022
Our customers are product and engineering teams at Netflix that build these software services and platforms. This became the foundation for our current org structure with teams focused on Appsec Partnerships and Appsec Engineering.
The Netflix TechBlog
MARCH 2, 2021
Stephanie Lane , Wenjing Zheng , Mihir Tendulkar Source credit: Netflix Within the rapid expansion of data-related roles in the last decade, the title Data Scientist has emerged as an umbrella term for myriad skills and areas of business focus. Learning through data is in Netflix’s DNA.
Adrian Cockcroft
FEBRUARY 13, 2023
SUS206 Sustainability and AWS silicon — Kamran Khan AWS Senior Product Manager Inferential/Trainium/FPGA, David Chaiken Pinterest Chief Architect, and Paul Mazurkiewicz AWS Senior Principal Engineer. Building a data lake of detailed information about energy use of many physical devices.
The Netflix TechBlog
OCTOBER 18, 2022
As Big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have increasingly become more important for our data scientists and the company. a data table or a spreadsheet) to another periodically.
Abhishek Tiwari
FEBRUARY 26, 2020
The evolution of your technology architecture should depend on the size, culture, and skill set of your engineering organization. Skills: Induct Full-stack engineers. Introduce site-reliability engineering best-practices (SLI/SLOs).
The Netflix TechBlog
MARCH 5, 2019
Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility and they are free to operate with freedom to satisfy their mission. Central engineering teams enable this operational model by reducing the cognitive burden on innovation teams through solutions related to securing, scaling and strengthening (resilience) the infrastructure.
Expert insights. Personalized for you.
182 
Let's personalize your content