Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. It became clear that real-time query processing and in-stream processing is the immediate need in many practical applications. Modularity and flexibility.

Big Data 154

Big Data Processing Lambda Database 154

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. An increasingly large fraction of data in modern workloads comes from less predictable and highly variable sources. joins) during query processing.

Storage 112

Storage Engineering Cache Serverless 112

Dynatrace

DECEMBER 15, 2022

Complex cloud computing environments are increasingly replacing traditional data centers. In fact, Gartner estimates that 80% of enterprises will shut down their on-premises data centers by 2025. So, what is ITOps? Why is IT operations important? How does ITOps create business value? ITOps vs. DevOps and DevSecOps.

Artificial Intelligence 195

Artificial Intelligence DevOps Hardware Virtualization 195

Dynatrace

AUGUST 26, 2020

“ Dynatrace just makes observability easy—it works out-of-the-box, no silos of data, no DIY stitching together tools, no wasted time, and no wasted resources.” There’s no other competing software that can provide this level of value with minimum effort and optimal hardware utilization that can scale up to web-scale!

Performance 149

Performance Hardware Java Storage 149

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Dynatrace

JANUARY 16, 2023

The study analyzes factual Kubernetes production data from thousands of organizations worldwide that are using the Dynatrace Software Intelligence Platform to keep their Kubernetes clusters secure, healthy, and high performing. Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch.

Open Source 337

Open Source Java Operating System Programming 337

Dynatrace

AUGUST 5, 2022

In a distributed processing environment, message queuing is similar, although the speed and volume of messages are much greater. A producer creates the message, and a consumer processes it. Consumers store messages in a queue — usually in a buffer or on a storage medium — until they can process and delete them.

Monitoring 192

Monitoring Serverless Programming Speed 192

Dynatrace

AUGUST 5, 2022

In a distributed processing environment, message queuing is similar, although the speed and volume of messages are much greater. A producer creates the message, and a consumer processes it. Consumers store messages in a queue — usually in a buffer or on a storage medium — until they can process and delete them.

Monitoring 191

Monitoring Serverless Programming Speed 191

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Percona

SEPTEMBER 1, 2023

This results in expedited query execution, reduced resource utilization, and more efficient exploitation of the available hardware resources. A finely tuned database processes queries more efficiently, leading to swifter results. An unoptimized indexing strategy can impede data insertion and retrieval operations.

Tuning 52

Tuning Database Performance Hardware 52

ACM Sigarch

AUGUST 3, 2023

Embodied carbon is due to Scope 3, e.g., manufacturing IT hardware and datacenter construction. The differences across vendors and processes are increasingly well tracked as the community is starting to build carbon quantification tools for embodied carbon, e.g., Greenchip , ACT and IMEC SSTS.

Energy 98

Energy Architecture Hardware Metrics 98

Scalegrid

JANUARY 25, 2024

Effective management of memory stores with policies like LRU/LFU proactive monitoring of the replication process and advanced metrics such as cache hit ratio and persistence indicators are crucial for ensuring data integrity and optimizing Redis’s performance. All these contribute significantly towards ensuring smooth functioning.

Metrics 130

Metrics Monitoring Latency Cache 130

Percona

JUNE 22, 2023

Database uptime and availability Monitoring database uptime and availability is crucial as it directly impacts the availability of critical data and the performance of applications or websites that rely on the MySQL database. Disk space usage Monitor the disk space usage of MySQL data files, log files, and temporary files.

Performance 104

Performance Monitoring Traffic Database 104

Abhishek Tiwari

JULY 12, 2018

Building data pipelines can offer strategic advantages to the business. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects. In this post, we will discuss why you should avoid building data pipelines in first place.

Latency 63

Latency Analytics Scalability Engineering 63

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 87

Storage Engineering Database Hardware 87

Percona

JANUARY 19, 2023

With disks being faster nowadays and CPU and memory resources being cheaper, we could easily say MySQL can handle TBs of data with good performance. For instance, in Percona Managed Services , we have many clients with TBs worth of data that are well performant. There are many compression tools and algorithms for data out there.

Open Source 87

Open Source Storage Database Big Data 87

HammerDB

APRIL 19, 2023

As database performance is heavily influenced by the performance of storage, network, memory, and processors, we must understand the upper limit of these key components. For storage, FIO is generally used. Storage: The system has a SATA drive for the operating system and one NVMe (Intel SSD D7-P5510 (3.84 Database: MySQL 8.0.31

Performance 54

Performance Benchmarking Cache Storage 54

Percona

DECEMBER 27, 2023

In our conversations with developers and engineers during the conference, we confirmed the CNCF GItOps Microsurvey data – 91% are already using GitOps. It comprises numerous organizations from various sectors, including software, hardware, nonprofit, public, and academic. You mentioned ‘More Simple Kubernetes’?

Cloud 79

Cloud Open Source Strategy Best Practices 79

Percona

MAY 1, 2023

Key areas include: Configuration parameter tuning : This tuning involves altering variables such as memory allocation, disk I/O settings, and concurrent connections based on specific hardware and requirements. Slow queries and data retrieval can lead to frustrating delays that impact the user experience. Nobody has time for that!

Tuning 52

Tuning Efficiency Database Performance 52

The Morning Paper

FEBRUARY 18, 2020

When it comes to leveraging ML in enterprise applications, especially in regulated environments, the level of scrutiny for data handling, model fairness, user privacy, and debuggability will be substantially higher than in the first wave of ML applications. The key insight this led to is that ML models are software , derived from data.

Government 95

Government Software Engineering Storage Database 95

The Morning Paper

MAY 27, 2019

Last week we saw the benefits of rethinking memory and pointer models at the hardware level when it came to object storage and compression ( Zippads ). The protections are hardware implemented and cannot be forged in software. ASPLOS’19. code is not given access to excessive capabilities. CHERI implementation.

C++ 61

C++ Hardware Virtualization Benchmarking 61

All Things Distributed

AUGUST 24, 2012

The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile. per GB/month.

Storage 108

Storage Hardware AWS Data Engineering 108

Brendan Gregg

FEBRUARY 28, 2023

Make sure your system can handle next-generation DRAM,” [link] Nov 2011 - [Hruska 12] Joel Hruska, “The future of CPU scaling: Exploring options on the cutting edge,” [link] Feb 2012 - [Gregg 13] Brendan Gregg, “Blazing Performance with Flame Graphs,” [link] 2013 - [Shimpi 13] Anand Lal Shimpi, “Seagate to Ship 5TB HDD in 2014 using Shingled Magnetic (..)

Performance 110

Performance Latency Cache Virtualization 110

Scalegrid

MAY 13, 2020

Greenplum Database is a massively parallel processing (MPP) SQL database that is built and based on PostgreSQL. It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Scalegrid

FEBRUARY 1, 2024

Regarding MySQL backups , knowing how to secure your data is crucial. Dive in to uncover the essentials of different backup types, command-line tools, and strategic practices for robust data protection. MySQL is a popular open-source relational database management system for online applications and data warehousing.

Best Practices 130

Best Practices Storage Strategy Database 130

O'Reilly

SEPTEMBER 19, 2023

It progressed from “raw compute and storage” to “reimplementing key services in push-button fashion” to “becoming the backbone of AI work”—all under the umbrella of “renting time and storage on someone else’s computers.” Cloud computing? And Hadoop rolled in. Until it wasn’t.

Hardware 99

Hardware Storage Big Data Blockchain 99

Testsigma

JULY 3, 2021

More control: While performing on-premise testing, organizations have more control over configurations, setup, hardware, and software. Data security: Since all the tools and machines are deployed within the premises — data is quite secure. This may require virtual and physical storage spaces, data centers, etc.

Cloud 57

Cloud Testing Storage DevOps 57

The Morning Paper

OCTOBER 4, 2020

As is expected to be common during the roll-out and transition phase of 5G, this network adopts the NSA (Non-Standalone Access) deployment model whereby the 5G radio is used for the data plane, but the control plane relies on existing 4G. The bulk of this latency comes from on-device frame processing, not network transmission delay.

Energy 130

Energy Latency Performance Network 130

Dynatrace

JUNE 25, 2020

With Dynatrace actively managing business-critical applications, some of our globally distributed enterprise customers require Dynatrace Managed to continue operating even when an entire data center goes down. Our Premium High Availability comes with the following features: Active-active deployment model for optimum hardware utilization.

Availability 241

Availability Hardware Latency Traffic 241

All Things Distributed

MARCH 11, 2015

From retail recommendations to genomics based product development, from financial risk management to start-ups measuring the effect of their new products, from digital marketing to fast processing of clinical trial data, all are taken to the next level by cloud based analytics. Cloud enables self-service analytics.

Analytics 135

Analytics Cloud Healthcare Retail 135

Percona

AUGUST 30, 2023

These days, “Data is king,” and a lot more. Data powers everything, and unlike coal and coal combustion, data and databases aren’t going away. Any organization that uses personal and other sensitive data must have a firm, proven plan for business continuity in the event of a disaster or cyberattack.

Database 87

Database Hardware Open Source Strategy 87

All Things Distributed

AUGUST 16, 2012

An apples to apples comparison of the costs associated with running various usage patterns on-premises and with AWS requires more than a simple comparison of hardware expense versus always-on utility pricing for compute and storage. Saving money.

AWS 118

AWS Hardware Traffic Best Practices 118

Percona

DECEMBER 27, 2022

Encrypting data at rest in a database management system (DBMS) refers to securing data by encrypting it when it is not being used or accessed. This is often done to protect sensitive data from unauthorized access or theft. Disk-level encryption is a security measure that encrypts all data stored on a disk or storage device.

Database 93

Database Storage Servers Strategy 93

SQL Performance

MARCH 17, 2020

Each test result represents the best of ten runs, with all required data and index pages in memory. Remember this is the best time over ten runs with all the data in memory. The Sort only has 19 rows to process, so it consumes only 1ms or so. We can do even better by providing a column store representation of the data.

Servers 86

Servers Tuning Database Efficiency 86

Brendan Gregg

FEBRUARY 28, 2023

Make sure your system can handle next-generation DRAM,” [link] , Nov 2011 [Hruska 12] Joel Hruska, “The future of CPU scaling: Exploring options on the cutting edge,” [link] , Feb 2012 [Gregg 13] Brendan Gregg, “Blazing Performance with Flame Graphs,” [link] , 2013 [Shimpi 13] Anand Lal Shimpi, “Seagate to Ship 5TB HDD in 2014 using Shingled Magnetic (..)

Performance 52

Performance Latency Cache Virtualization 52

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Instead of worrying about infrastructure management functions, such as capacity provisioning and hardware maintenance, teams can focus on application design, deployment, and delivery. Data Store.

Serverless 273

Serverless AWS Lambda Storage 273