Architecture, Big Data, Scalability and Technology

In-Stream Big Data Processing

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. In addition, we survey the current and emerging technologies and provide a few implementation tips. The article is based on a research project developed at Grid Dynamics Labs.

Big Data

Big Data Processing Lambda Database

Kubernetes in the wild report 2023

Dynatrace

JANUARY 16, 2023

While Kubernetes is still a relatively young technology, a large majority of global enterprises use it to run business-critical applications in production. Findings provide insights into Kubernetes practitioners’ infrastructure preferences and how they use advanced Kubernetes platform technologies. Java, Go, and Node.js

Open Source

Open Source Java Operating System Programming

What is container orchestration?

Dynatrace

MARCH 24, 2023

Container technology enables organizations to efficiently develop cloud-native applications or to modernize legacy applications to take advantage of cloud services. Apache Mesos with the Marathon DC/OS is popular for large-scale production clusters running existing workloads on big data systems, such as Hadoop, Kafka, and Spark.

Infrastructure

Infrastructure Open Source Operating System Cloud

How Netflix uses eBPF flow logs at scale for network insight

The Netflix TechBlog

JUNE 7, 2021

By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs , ELB Access Logs, eBPF flow logs on the instances, etc, we can provide network insight to users and central teams through multiple data visualization techniques like Lumen , Atlas , etc. What is BPF?

Network

Network Transportation AWS Cloud

Migrating Critical Traffic At Scale with No Downtime?—?Part 1

The Netflix TechBlog

MAY 4, 2023

When undertaking system migrations, one of the main challenges is establishing confidence and seamlessly transitioning the traffic to the upgraded architecture without adversely impacting the customer experience. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Traffic

Traffic Latency Tuning Systems

The Need for Real-Time Device Tracking

ScaleOut Software

JULY 19, 2021

Today’s streaming analytics architectures are not equipped to make sense of this rapidly changing information and react to it as it arrives. This data is also periodically uploaded to a data lake for offline batch analysis that calculates key statistics and looks for big trends that can help optimize operations.

IoT

IoT Analytics Big Data Architecture

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

ACM Sigarch

MAY 31, 2023

Introduction Memory systems are evolving into heterogeneous and composable architectures. Heterogeneous and Composable Memory (HCM) offers a feasible solution for terabyte- or petabyte-scale systems, addressing the performance and efficiency demands of emerging big-data applications. The recently announced CXL3.0

Latency

Latency Hardware Cache Architecture

Use Digital Twins for the Next Generation in Telematics

ScaleOut Software

NOVEMBER 24, 2020

Real-Time Digital Twins Can Add Important New Capabilities to Telematics Systems and Eliminate Scalability Bottlenecks. However, telematics architectures face challenges in responding to telemetry in real time. Current Telematics Architecture. Challenges for Current Architectures.

Analytics

Analytics Architecture Scalability Software Architecture

What is a Distributed Storage System

Scalegrid

FEBRUARY 8, 2024

This guide delves into how these systems work, the challenges they solve, and their essential role in businesses and technology. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise. Variations within these storage systems are called distributed file systems.

Storage

Storage Systems Big Data Azure

Expanding the AWS Cloud: Introducing the AWS Canada (Central) Region

All Things Distributed

DECEMBER 8, 2016

Canada has set forth a bold innovation agenda grounded in entrepreneurship, scientific research, growing small and medium-sized businesses with a focus on environmentally friendly technologies, and the transition to a digital economy. Scalability. For more information about AWS efforts, see AWS & Sustainability.

AWS

AWS Cloud Lambda Innovation

Dutch Enterprises and The Cloud

All Things Distributed

SEPTEMBER 6, 2013

Shell leverages AWS for big data analytics to help achieve these goals. Due to the exponential growth of the biology and informatics fields, Unilever needs to maintain this new program within a highly-scalable environment that supports parallel computation and heavy data storage demands.

Cloud

Cloud Energy AWS Healthcare

Track Thousands of Assets in a Time of Crisis Using Real-Time Digital Twins

ScaleOut Software

APRIL 3, 2020

An innovative new software approach called “real-time digital twins” running on a cloud-hosted, highly scalable, in-memory computing platform can help address this challenge. The computing system also has the ability to perform aggregate analytics in seconds on the continuously evolving data held in the twins.

Logistics

Logistics Analytics Scalability Cloud

Track Thousands of Assets in a Time of Crisis Using Real-Time Digital Twins

ScaleOut Software

APRIL 3, 2020

An innovative new software approach called “real-time digital twins” running on a cloud-hosted, highly scalable, in-memory computing platform can help address this challenge. The computing system also has the ability to perform aggregate analytics in seconds on the continuously evolving data held in the twins.

Logistics

Logistics Analytics Scalability Cloud

Redis vs Memcached in 2024

Scalegrid

MARCH 28, 2024

In this comparison of Redis vs Memcached, we strip away the complexity, focusing on each in-memory data store’s performance, scalability, and unique features. Redis is better suited for complex data models, and Memcached is better suited for high-throughput, string-based caching scenarios.

Cache

Cache Storage Scalability Architecture

Mastering Hybrid Cloud Strategy

Scalegrid

MARCH 14, 2024

This approach allows companies to combine the security and control of private clouds with public clouds’ scalability and innovation potential. Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution. A hybrid cloud strategy could be your answer.

Strategy

Strategy Cloud Artificial Intelligence Infrastructure

What is IT operations analytics? Extract more data insights from more sources

Dynatrace

MAY 1, 2023

Then, big data analytics technologies, such as Hadoop, NoSQL, Spark, or Grail, the Dynatrace data lakehouse technology, interpret this information. Here are the six steps of a typical ITOA process : Define the data infrastructure strategy. Provide data literacy for stakeholders. Why is ITOA important?

Analytics

Analytics Artificial Intelligence Big Data Open Source

Job Openings in AWS - Senior Leader in Database Services - All.

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. This team is constantly rethinking the assumptions behind how traditional databases were built and constantly working on building the right database architectures suited for the Cloud environment. Driving down the cost of Big-Data analytics.

AWS

AWS Database Storage Scalability

Using Real-Time Digital Twins for Aggregate Analytics

ScaleOut Software

JUNE 15, 2020

Instead, most applications just sift through the telemetry for patterns that might indicate exceptional conditions and forward the bulk of incoming messages to a data lake for offline scrubbing with a big data tool such as Spark. Maintain State Information for Each Data Source.

Analytics

Analytics IoT Lambda Big Data

Using Real-Time Digital Twins for Aggregate Analytics

ScaleOut Software

JUNE 15, 2020

Instead, most applications just sift through the telemetry for patterns that might indicate exceptional conditions and forward the bulk of incoming messages to a data lake for offline scrubbing with a big data tool such as Spark. Maintain State Information for Each Data Source.

Analytics

Analytics IoT Lambda Big Data

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. During my academic career, I spent many years working on HPC technologies such as user-level networking interfaces, large scale high-speed interconnects, HPC software stacks, etc. Driving down the cost of Big-Data analytics. All Things Distributed.

Cloud

Cloud AWS Automotive Latency

Around the World in 28 Days - All Things Distributed

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup. Mumbai on October 20 with Patni , Inxs Technologies and India Games.

AWS

AWS Storage Cloud Best Practices

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. Understanding Throughput-Oriented Architectures - background article in CACM on massively parallel and throughput vs latency oriented architectures. Comments ().

AWS

AWS Cloud Benchmarking Storage

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Often these namespaces are hierarchical in nature such that it becomes easier to manage them and to decentralize control, which makes the system more scalable. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications.

Cloud

Cloud Internet Internet AWS

Any analysis, any time: Dynatrace Log Management and Analytics powered by Grail

Dynatrace

OCTOBER 4, 2022

Limited data availability constrains value creation. Modern IT environments — whether multicloud, on-premises, or hybrid-cloud architectures — generate exponentially increasing data volumes. Stop worrying about log data ingest and storage — start creating value instead. What’s next for Grail?

Analytics

Analytics Artificial Intelligence Storage Serverless

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” Nonetheless, Netflix data landscape (see below) is complex and many teams collaborate effectively for sharing the responsibility of our data system management.

Infrastructure

Infrastructure Big Data Transportation Architecture

The Amazon.com 2010 Shareholder Letter Focusses on Technology.

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The Amazon.com 2010 Shareholder Letter Focusses on Technology. In the 2010 Shareholder Letter Jeff Bezos writes about the unique technologies developed at Amazon.com over the years. All Things Distributed. By Werner Vogels on 27 April 2011 12:51 AM.

Technology

Technology Technology AWS Storage

Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value Stores

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3. Moving data with Bulldozer at Netflix.

Latency

Latency Storage Big Data Tuning

Music to my Ears - All Things Distributed

All Things Distributed

MARCH 28, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The scalability, reliability and durability requirements for Cloud Drive are very high which is why they decided to make use of the Amazon Simple Storage Service (S3) as the core component of their service. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Storage Internet

Amazon EC2 Cluster GPU Instances - All Things Distributed

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. These trade-offs have even impacted the way the lowest level building blocks in our computer architectures have been designed. All Things Distributed. Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances. Comments ().

AWS

AWS Latency Programming Architecture

The Winds of Architecture Changes at the USENIX ATC 2019

ACM Sigarch

NOVEMBER 1, 2019

This blog post gives a glimpse of the computer systems research papers presented at the USENIX Annual Technical Conference (ATC) 2019, with an emphasis on systems that use new hardware architectures. As a consequence, the vast majority of the papers in the past has usually focused on conventional X86 or GPU-accelerated architectures.

Architecture

Architecture Hardware Cache Storage

Technology Performance Pulse

In-Stream Big Data Processing

Kubernetes in the wild report 2023

Trending Sources

What is container orchestration?

How Netflix uses eBPF flow logs at scale for network insight

Migrating Critical Traffic At Scale with No Downtime?—?Part 1

The Need for Real-Time Device Tracking

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

Use Digital Twins for the Next Generation in Telematics

What is a Distributed Storage System

Expanding the AWS Cloud: Introducing the AWS Canada (Central) Region

Dutch Enterprises and The Cloud

Track Thousands of Assets in a Time of Crisis Using Real-Time Digital Twins

Track Thousands of Assets in a Time of Crisis Using Real-Time Digital Twins

Redis vs Memcached in 2024

Mastering Hybrid Cloud Strategy

What is IT operations analytics? Extract more data insights from more sources

Job Openings in AWS - Senior Leader in Database Services - All.

Using Real-Time Digital Twins for Aggregate Analytics

Using Real-Time Digital Twins for Aggregate Analytics

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

Around the World in 28 Days - All Things Distributed

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

Any analysis, any time: Dynatrace Log Management and Analytics powered by Grail

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Amazon.com 2010 Shareholder Letter Focusses on Technology.

Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value Stores

Music to my Ears - All Things Distributed

Amazon EC2 Cluster GPU Instances - All Things Distributed

The Winds of Architecture Changes at the USENIX ATC 2019

Stay Connected