n0derunner

SEPTEMBER 15, 2023

Background For the past few months engineers from Nutanix have been participating in the MLPerftm Storage benchmark which is designed to measure the storage performance required for ML training workloads. appeared first on n0derunner.

Benchmarking 59

Benchmarking Storage Servers Engineering 59

InfoQ

OCTOBER 14, 2023

times the storage per node than before” without a price change. Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 By Steef-Jan Wiggers

Google 99

Google Storage Cloud Database 99

Dynatrace

MAY 7, 2024

Fluent Bit is a telemetry agent designed to receive data (logs, traces, and metrics), process or modify it, and export it to a destination. Fluent Bit was designed to help you adjust your data and add the proper context, which can be helpful in the observability backend. What’s the difference between Fluent Bit and Fluentd?

Best Practices 232

Best Practices IoT Metrics Storage 232

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. have altered the many assumptions that guided the design and optimization of the Snowflake system. From shared-nothing to disaggregation.

Storage 112

Storage Engineering Cache Serverless 112

Percona

FEBRUARY 1, 2023

This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability. Therefore, TOAST is a storage technique used in PostgreSQL to handle large data objects such as images, videos, and audio files.

Storage 94

Storage Scalability Strategy Performance 94

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Cloud 99

Smashing Magazine

DECEMBER 6, 2021

Designing Better Links For Websites And Emails: A Guideline. Designing Better Links For Websites And Emails: A Guideline. if the information is read-only or editable); The file size is crucial for people with costly internet, slow connection, or limited local storage. Slava Shestopalov. 2021-12-06T17:30:00+00:00.

Website 141

Website Design Internet Internet 141

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Speed Database 52

Particular Software

APRIL 12, 2021

Azure Storage Queues is a basic yet robust queueing service available on the Azure platform. For example, it lacks concepts like topics and subscriptions to build publisher and subscriber topologies—these are capabilities NServiceBus adds on top of Azure Storage Queues. Here’s how it works.

Azure 52

Azure Storage Transportation Java 52

Scalegrid

MAY 13, 2020

It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. Greenplum uses an MPP database design that can help you develop a scalable, high performance deployment. Greenplum Architectural Design.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Tech News Gather

MARCH 16, 2023

8 factors to consider while iOS app designing App designing is a rapidly growing industry because users spend more time in apps than in browsers. The user engagement, customer satisfaction, and additional exposure that a good app design provides for your product all go up. Our purpose is not to dispute Apple’s assertions.

Design 52

Design Mobile Storage Systems 52

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Instead of worrying about infrastructure management functions, such as capacity provisioning and hardware maintenance, teams can focus on application design, deployment, and delivery. Reliability.

Serverless 280

Serverless AWS Lambda Storage 280

Dynatrace

JANUARY 17, 2024

AI requires more compute and storage. Training AI data is resource-intensive and costly, again, because of increased computational and storage requirements. As a result, AI observability supports cloud FinOps efforts by identifying how AI adoption spikes costs because of increased usage of storage and compute resources.

Strategy 223

Strategy Artificial Intelligence Storage Cloud 223

Dynatrace

JANUARY 31, 2024

Designed to reach beyond petabyte (PB) scale, OpenPipeline will—at its launch— quintuple the ingest throughput from 100 TB a day per tenant to 500 TB a day per tenant. OpenPipeline high-performance filtering and preprocessing provides full ingest and storage control for the Dynatrace platform. Commitment to privacy.

Analytics 195

Analytics Processing Transportation Storage 195

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Efficiency 103

Efficiency Engineering Design Storage 103

Scalegrid

MARCH 28, 2024

This article will explore how they handle data storage and scalability, perform in different scenarios, and, most importantly, how these factors influence your choice. Redis Data Types and Structures The design of Redis’s data structures emphasizes versatility. Memcached’s primary strength lies in its simplicity.

Cache 130

Cache Storage Scalability Architecture 130

InfoQ

JULY 7, 2023

Facundo Agriel dives into the architecture of Magic Pocket, some early key design patterns, and the challenges of operating such a system at this scale. By Facundo Agriel

Systems 52

Systems Storage Architecture Design 52

Percona

APRIL 17, 2023

Since MongoDB usually works with a replica set, it is possible to increase this parameter to get better performance: # edit /etc/mongod.conf storage: journal: enabled: true commitIntervalMs: 300 WiredTiger cache For dedicated servers, it is possible to increase the WiredTiger(WT) cache. By default, it uses 50% of the memory + 1 GB.

Best Practices 52

Best Practices Design Tuning Database 52

Dynatrace

OCTOBER 17, 2023

Dynatrace has developed the purpose-built data lakehouse, Grail , eliminating the need for separate management of indexes and storage. All data is readily accessible without storage tiers, such as costly solid-state drives (SSDs). No storage tiers, no archiving or retrieval from archives, and no indexing or reindexing.

Retail 237

Retail Storage Best Practices Architecture 237

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 201

Infrastructure Storage Virtualization Network 201

All Things Distributed

MAY 19, 2017

We decided to use our inventive powers to design and build a new database engine that would give database systems such as MySQL and PostgreSQL reliability and performance at scale, at a level that could serve even the most demanding OLTP applications.

Database 117

Database Design Cloud Storage 117

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

DZone

JANUARY 23, 2022

Hardware - servers/storage hardware/software faults such as disk failure, disk full, other hardware failures, servers running out of allocated resources, server software behaving abnormally, intra DC network connectivity issues, etc. . – this is addressed through monitoring and redundancy. Redundancy by building additional data centers.

Hardware 243

Hardware DevOps Network Storage 243

Dynatrace

APRIL 28, 2023

This means compromising between keeping data available as long as possible for analysis while juggling the costs and overhead of storage, archiving, and retrieval. This allows you to design data management and retention policies based on individual requirements, starting from days of retention up to a decade.

Analytics 213

Analytics Storage Infrastructure Metrics 213

DZone

OCTOBER 11, 2022

Simpler UI Testing with CasperJS ( Architects Zone – Architectural Design Patterns & Best Practices). Using MongoDB as a cache store ( Architects Zone – Architectural Design Patterns & Best Practices). Why haven’t cash-strapped American schools embraced open source? Hacker News). Thoughts, Insights and Further Pointers.

Java 244

Java Best Practices Google Analytics 244

Dynatrace

AUGUST 1, 2022

Kubernetes was initially designed with a strong focus on stateless workloads, meaning these workloads do not need to store any persistent data. You quickly realize that it will take ages to fill up the overprovisioned database storage. Two days later, your database runs out of storage in the middle of the night. Dynatrace news.

Storage 187

Storage Database Network Tuning 187

All Things Distributed

MARCH 23, 2021

Storage was one of our biggest pain points, and the traditional systems we used just weren’t fitting the needs of the Amazon.com retail business. When we took a hard look at our storage for the Amazon ecommerce web site in 2005, we realized that the majority of our data needed an object (or key-value) store.

Ecommerce 216

Ecommerce Retail Storage Scalability 216

Dynatrace

OCTOBER 4, 2022

Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. Unlike data warehouses, however, data is not transformed before landing in storage. A data lakehouse provides a cost-effective storage layer for both structured and unstructured data. Data management.

Artificial Intelligence 208

Artificial Intelligence Analytics Storage Government 208

Dynatrace

OCTOBER 4, 2022

A data lakehouse addresses these limitations and introduces an entirely new architectural design. This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). Grail is built for such analytics, not storage.

Artificial Intelligence 269

Artificial Intelligence Analytics Storage Architecture 269

Percona

MARCH 11, 2024

Volume Expansion, a feature that became generally available since Kubernetes version 1.24, allows users to increase the capacity of their Persistent Volumes and underlying storage within Kubernetes. In our Operators, we use Stateful Sets – a Kubernetes resource that was designed for […]

Storage 92

Storage Cloud Design Availability 92

DZone

JANUARY 25, 2022

A basic example could be listening to an AWS SQS queue and saving a file in Google Cloud Storage Bucket for each message that has been received. A book that defines a series of patterns and blueprints to help us to better design large component-based applications.

AWS 185

AWS Storage Google Design 185

All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? With the launch of the AWS Storage Gateway our customers can now integrate their on-premises IT environment with AWSâ??s

Storage 115

Storage AWS Virtualization Cloud 115

Scalegrid

MARCH 8, 2024

Use the ‘–log-bin’ option to designate a base name for the binary log files. Following creation, best practices dictate carefully transferring the secure backup file and its associated binary log SQL files to your designated recovery host. To enable this feature, you must: Turn the ‘log_bin’ variable to ‘ON’.

Database 162

Database Strategy Servers Best Practices 162

DZone

JANUARY 20, 2023

Do Not Be Misled Designing and implementing a scalable graph database system has never been a trivial task. It has been a norm to perceive that distributed databases use the method of adding cheap PC(s) to achieve scalability (storage and computing) and attempt to store data once and for all on demand.

Scalability 201

Scalability Big Data Hardware Internet 201

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â??

Storage 122

Storage Cloud AWS Media 122

Strategic Tech

MAY 24, 2019

Over the past few years I have focused much of my learning and work choices around learning about the design of sociotechnical systems?—?how how to design software architectures and organise teams around them. As part of good design, we should consider the effects of changing all of these variables.

Design 45

Design Software Architecture Strategy Technology 45

Dynatrace

OCTOBER 6, 2023

Grail: Enterprise-ready data lakehouse Grail, the Dynatrace causational data lakehouse, was explicitly designed for observability and security data, with artificial intelligence integrated into its foundation. Buckets are similar to folders, a physical storage location. There is a default bucket for each table.

Artificial Intelligence 262

Artificial Intelligence Analytics Metrics Storage 262