DZone

AUGUST 16, 2023

From a data engineer's point of view, financial risk management is a series of data analysis activities on financial data. The financial sector imposes its unique requirements on data engineering.

Engineering 130

Engineering Latency Data Engineering FinTech 130

DZone

FEBRUARY 20, 2023

The data community is striving to incorporate the core concepts of engineering rigor found in software communities but still has further to go. This is achieved through practices like Infrastructure as Code for deployments, automated testing, application observability, and end-to-end application lifecycle ownership.

Software Engineering 269

Software Engineering Engineering Software Software 269

DZone

JANUARY 27, 2023

Data lineage, an automated visualization of the relationships for how data flows across tables and other data assets, is a must-have in the data engineering toolbox.

Data Engineering 180

Data Engineering Government Engineering 180

High Scalability

JUNE 15, 2020

This is a guest post by Eunice Do , Data Engineer at TripleLift , a technology company leading the next generation of programmatic advertising. The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis.

Processing 224

Processing Data Engineering Engineering Systems 224

Simform

JANUARY 8, 2023

Data ingestion is the foremost layer in a data engineering pipeline, acting as a vital pillar in the overall analytics architecture. Thus, it is essential to implement data ingestion just right. Here is everything you need to know to take the first step toward a flawless data pipeline.

Data Engineering 52

Data Engineering Architecture Engineering Analytics 52

O'Reilly

FEBRUARY 18, 2020

The results for data-related topics are both predictable and—there’s no other way to put it—confusing. Starting with data engineering, the backbone of all data work (the category includes titles covering data management, i.e., relational databases, Spark, Hadoop, SQL, NoSQL, etc.). This follows a 3% drop in 2018.

DevOps 135

DevOps Software Architecture Data Engineering Architecture 135

The Netflix TechBlog

SEPTEMBER 18, 2020

The Engineer enjoys making data available by piping it in from new sources in optimal ways, building robust data models, prototyping systems, and doing project-specific engineering.

Analytics 245

Analytics Engineering Data Engineering Metrics 245

The Netflix TechBlog

FEBRUARY 9, 2022

Let’s define some requirements that we are interested in delivering to the Netflix data engineers or anyone who would like to schedule a workflow with some external assets in it. Conclusions This new method available for Netflix data engineers makes workflow management easier, more transparent and more reliable.

Testing 205

Testing Code Storage Data Engineering 205

The Netflix TechBlog

DECEMBER 3, 2022

is.null(title_id) & event_count > 0) |> select(title_id, country_code, event_type) |> group_by(title_id, country_code) |> summarize(event_count = sum(event_type, na.rm = TRUE)) ranked_title_activity_by_country <- title_activity_by_country |> group_by(country_code) |> mutate(title_rank = rank(desc(event_count))) top_25_title_by_country (..)

Code 360

Code Testing Best Practices Engineering 360

The Netflix TechBlog

DECEMBER 21, 2020

Some of the optimizations are prerequisites for a high-performance data warehouse. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster.

Storage 218

Storage Latency Data Engineering Efficiency 218

Percona Community

OCTOBER 14, 2020

next day) Abstract Not so long ago, operations specialists worked much like today’s data engineers do: with specialized skills, they were the people who kept sites running, who responded to emergencies, and who—unfortunately—spent much of their time dealing with incidents and other “fires. London 5:30 p.m. •

Engineering 52

Engineering DevOps Data Engineering 52

The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. Some nuances while creating this dataset come from the on-field domain knowledge of our engineers.

Cache 194

Cache Big Data Engineering Data Engineering 194

Dynatrace

JULY 6, 2022

This requires significant data engineering efforts, as well as work to build machine-learning models. While automating IT processes without integrated AIOps can create challenges, the approach to artificial intelligence itself can also introduce potential issues. AI that is based on machine learning needs to be trained.

DevOps 177

DevOps Monitoring Tuning Infrastructure 177

The Netflix TechBlog

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. Give us a holler if you are interested in a thought exchange.

Infrastructure 152

Infrastructure Scalability Cloud AWS 152

The Netflix TechBlog

JUNE 6, 2022

Our focus has been on improving overall security assurance as opposed to just vulnerability prevention. We are now expanding this approach to more parts of our ecosystem.

Software Engineering 182

Software Engineering Scalability Engineering Software 182

Percona Community

OCTOBER 3, 2018

For Percona Live Europe, I’ll be presenting the tutorial Elasticsearch 101 alongside my colleagues and fellow presenters from ObjectRocket Alex Cercel, DBA, and Mihai Aldoiu, Data Engineer. Here’s a brief overview of our tutorial.

Data Engineering 52

Data Engineering Scalability Engineering 52

Abhishek Tiwari

JULY 7, 2023

Airflow provides rich scheduling and execution semantics enabling data engineers to easily define complex pipelines, running at regular intervals. In reality, a DAG lacks the necessary workflow context, and relying solely on it can result in incomplete solutions and missed opportunities.

Logistics 52

Logistics Scalability Data Engineering Processing 52

The Netflix TechBlog

JANUARY 11, 2022

To learn about Analytics and Viz Engineering, have a look at Analytics at Netflix: Who We Are and What We Do by Molly Jackman & Meghana Reddy and How Our Paths Brought Us to Data and Netflix by Julie Beckley & Chris Pham. Curious to learn about what it’s like to be a Data Engineer at Netflix?

Testing 246

Testing Metrics Engineering Innovation 246

The Netflix TechBlog

SEPTEMBER 10, 2019

Our data scientists faced numerous challenges in our previous infrastructure. Complex business logic was embedded directly into the ETL pipelines by data engineers. In order to replicate results, scientists had to delve deep into the data, code, and documentation.

Metrics 213

Metrics Architecture Testing Infrastructure 213

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Engineering 222

Engineering Testing Analytics C++ 222

The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” Please share your experience by adding your comments below and stay tuned for more on data lineage at Netflix in the follow up blog posts. .

Infrastructure 224

Infrastructure Scalability Architecture Engineering 224

Abhishek Tiwari

FEBRUARY 26, 2020

Now building data-driven apps and data services and where necessary incorporating machine learning web service offered by public cloud providers. Introduce site-reliability engineering best-practices (SLI/SLOs). Skills++: Induct Site-reliability engineers, Data engineers.

Architecture 81

Architecture Engineering Technology Technology 81

The Agile Manager

JULY 31, 2022

There are shadow IT teams of developers or data engineers that spring up in areas like operations or marketing because the captive IT function is slow, if not outright incapable, of responding to internal customer demand. IT shadows appear in a lot of different forms.

Code 59

Code Testing Engineering Data Engineering 59

High Scalability

MARCH 17, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 107

Education Servers Engineering Java 107

High Scalability

FEBRUARY 18, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 118

Education Servers Engineering Java 118

High Scalability

MARCH 3, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 116

Education Servers Engineering Java 116

High Scalability

APRIL 14, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 108

Education Servers Scalability Software Engineering 108

High Scalability

FEBRUARY 9, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 111

Education Servers Engineering Java 111

High Scalability

MAY 12, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 104

Education Servers Scalability Software Engineering 104

High Scalability

MARCH 24, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 107

Education Servers Software Engineering Engineering 107

High Scalability

APRIL 28, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 104

Education Servers Scalability Software Engineering 104

High Scalability

MARCH 30, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 108

Education Servers Software Engineering Engineering 108

High Scalability

MAY 26, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

AWS 100

AWS Education Servers Scalability 100

All Things Distributed

AUGUST 24, 2012

Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. For this purpose I have picked work based on two papers by Jim Gray , the brilliant IBM / Tandem / Microsoft researcher, who won a Turing award for his contributions to data and transaction processing.

Storage 112

Storage AWS Data Engineering Scalability 112

InfoQ

SEPTEMBER 25, 2023

When it comes to deploying Large Language Models (LLMs) in production, the two major challenges originate from the huge amount of parameters they require and the necessity of handling very long input sequences to represent contextual information.

Scalability 120

Scalability Data Engineering Performance Engineering 120

The Netflix TechBlog

OCTOBER 18, 2022

It is a general-purpose workflow orchestrator that provides a fully managed workflow-as-a-service (WAAS) to the data platform at Netflix. It serves thousands of users, including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts, for various use cases.

Scalability 218

Scalability Java Engineering Architecture 218

O'Reilly

FEBRUARY 2, 2020

Collaboration between AI developers and operations teams will lead to growing pains on both sides, especially since many data scientists and AI researchers have had limited exposure to, or knowledge of, software engineering. It’s going to be an interesting few years as operations assimilates AI.

Code 69

Code Testing Monitoring Software Architecture 69