All Things Distributed

AUGUST 24, 2012

Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. For this purpose I have picked work based on two papers by Jim Gray , the brilliant IBM / Tandem / Microsoft researcher, who won a Turing award for his contributions to data and transaction processing.

Storage 108

Storage Hardware AWS Data Engineering 108

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next.

Big Data 63

Big Data Storage Benchmarking Hardware 63

O'Reilly

SEPTEMBER 15, 2021

The results are biased by the survey’s recipients (subscribers to O’Reilly’s Data & AI Newsletter ). Our audience is particularly strong in the software (20% of respondents), computer hardware (4%), and computer security (2%) industries—over 25% of the total. Average salary change vs. type of training. The Last Word.

Azure 145

Azure Social Media Programming AWS 145

Abhishek Tiwari

JULY 12, 2018

Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Not everyone is operating at Netflix or Spotify scale data engineering function. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Latency 63

Latency Analytics Scalability Engineering 63

All Things Distributed

OCTOBER 7, 2015

While BI solutions have existed for decades, customers have told us that it takes an enormous amount of time, engineering effort, and money to bridge this gap. These solutions lack interactive data exploration and visualization capabilities, limiting most business users to canned reports and pre-selected queries.

Cloud 137

Cloud Big Data AWS Analytics 137

Abhishek Tiwari

DECEMBER 27, 2017

Because supported big data frameworks and applications can utilize the same internal memory format, they can avoid data serialization and deserialization to convert data between various formats. In contrast, Alluxio a middleware for data access - think Alluxio storage layer as fast cache.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

O'Reilly

JULY 15, 2020

Technical roles represented in the “Other” category include IT managers, data engineers, DevOps practitioners, data scientists, systems engineers, and systems administrators. Combined, technology verticals—software, computers/hardware, and telecommunications—account for about 35% of the audience (Figure 2).

Database 135

Database Architecture Education Systems 135