DZone

JUNE 25, 2019

It’s really scary knowing that such corruptions are happening in the memory of our computers and servers – that is before they even reach the network and storage portions of the stack. That data must then be safely transported over a network to the storage system where it is written to disk. Well, if you’re using one of the storage protocols that lack end-to-end checksums (e.g. performance storage database checksum data corruption data safety

Storage 130

Storage Systems Transportation Network 130

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Storage 70

Storage Engineering Database Servers 70

DZone

NOVEMBER 13, 2019

performance storage spark caching in-memory spark caching spark memoryImprove your Spark memory. In the previous tutorial , we demonstrated how to get started with Spark and Alluxio. To share more thoughts and experiments on how Alluxio enhances Spark workloads, this article focuses on how Alluxio helps to optimize the memory utilization of Spark applications.

Storage 109

Storage Cache Performance 109

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. One day I faced the problem with downloading a relatively large binary data file from PostgreSQL. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 116

Storage Efficiency Database Testing 116

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceThere are several AI/ML focused use cases to highlight.

Storage 100

Storage FinTech Artificial Intelligence Analytics 100

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. In contrast, there are generally available NVMe solutions that can scale from 100TB to 1PB of shared NVMe storage at the performance of local NVMe SSDs, providing the opportunity to significantly increase the depth of the training for neural networks. big data performance data storage ssd nvme gpu ai ml

Storage 100

Storage Network Scalability Big Data 100

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Analysed from the perspective of cloud-native design this presents a number of issues: CPU, memory, storage, and bandwidth resources are all aggregated at each node, and can’t be scaled independently, making it hard to fit a workload efficiently across multiple dimensions. But the ephemeral storage service for intermediate data is not based on S3.

Storage 82

Storage Engineering Cache Serverless 82

DZone

MAY 22, 2019

However, Hive cannot access a single table directly using a single query with the data of this Hive table across different mediums of storage and different clusters. This becomes a need when the data volume grows too large to fit a single medium of storage or cluster, and also when the users need to take into account the following considerations: Storage cost, where some partitions are less important than others and can be stored on cheaper storage tiers.

Storage 100

Storage Systems Virtualization Cloud 100

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Summary tl;dr There are two ways to ensure that IO goes directly to the backend storage (direct attach disk, SAN or HCI datastore) Use a “raw” or “physical” device (#<diskID) Open files on the filesystem with … The post Microsoft diskspd. Part 2, Accurately testing storage devices.

Storage 40

Storage Performance Testing Performance Testing 40

n0derunner

JUNE 28, 2019

The post View from Nutanix storage during Postgres DB benchmark appeared first on n0derunner. A quick look at how the workload is seen from the Nutanix CVM. In this example from prior post. The Linux VM running postgres has two virtual disks – one taking transaction log writes. The other is doing reads and writes from the main datafiles.

Storage 63

Storage Benchmarking Virtualization Cache 63

SQL Shack

JULY 14, 2020

This article explores the configuration of Windows failover clusters, storage controllers and quorum configurations for SQL Server Always On Availability Groups. Prerequisites In this series of articles, we will configure the SQL Server always on availability groups from end to end for your learning purpose. We covered the following topics in the previous articles. A […]. AlwaysOn Availability Groups Clustering High Availability

Storage 52

Storage Availability Servers 52

DZone

MAY 30, 2019

With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. This has given rise to a completely new set of computing workloads for Machine Learning which drives Artificial Intelligence applications. AI/ML can be applied across a broad spectrum of applications and industries.

Storage 100

Storage FinTech Social Media Artificial Intelligence 100

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " From an operator perspective it makes it harder to follow the classic cloud-native design in which a global storage layer is separate to compute. In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Storage 64

Storage Serverless Latency Cloud 64

High Scalability

FEBRUARY 19, 2019

Redis Cluster is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. At ScaleGrid, we recently added support for Redis Clusters on our platform through our fully managed Redis hosting plans.

Storage 202

Storage Cache Scalability Open Source 202

All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? With the launch of the AWS Storage Gateway our customers can now integrate their on-premises IT environment with AWSâ??s s storage infrastructure. The AWS Storage Gateway is a service connecting an on-premises software appliance with cloud-based storage.

Storage 63

Storage AWS Virtualization Infrastructure 63

The Netflix TechBlog

JULY 8, 2020

cloud-storage data data-infrastructure aws netflixBy Torio Risianto, Bhargavi Reddy, Tanvi Sahni, Andrew Park Continue reading on Netflix TechBlog ».

Infrastructure 145

Infrastructure Storage AWS Cloud 145

Scalegrid

MAY 13, 2020

High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Greenplum also introduced many features which are unavailable within PostgreSQL, such as parallel data loading, resource management, storage enhancements and advanced query optimization, making it an attractive offering when you are comparing the two. Polymorphic Data Storage.

Big Data 186

Big Data Database Artificial Intelligence Open Source 186

High Scalability

APRIL 3, 2019

PostgreSQL is an open source object-relational database system that has soared in popularity over the past 30 years from its active, loyal, and growing community. For the 2nd year in a row, PostgreSQL has kept the title of #1 fastest growing database in the world according to the DBMS of the Year report by the experts at DB-Engines. So what makes PostgreSQL so special, and how is it being used today?

Database 188

Database Cloud Open Source Storage 188

All Things Distributed

MAY 18, 2010

Expanding the Cloud - Amazon S3 Reduced Redundancy Storage. Today a new storage option for Amazon S3 has been launched: Amazon S3 Reduced Redundancy Storage (RRS). This new storage option enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy. This has been an option that customers have been asking us about for some time so we are really pleased to be able to offer this alternative storage option now.

Storage 62

Storage AWS Cloud Scalability 62

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. That file is stored in our object storage service, which splits and encrypts the file into separate chunks, storing the chunks in Amazon S3. Our object storage service splits objects into many parts and stores them in S3.

Media 185

Media Storage Processing Cache 185

Scalegrid

JULY 17, 2020

JSONB storage has some drawbacks vs. traditional columns: PostreSQL does not store column statistics for JSONB columns. JSONB storage results in a larger storage footprint. JSONB storage does not deduplicate the key names in the JSON. This can result in considerably larger storage footprint compared to MongoDB BSON on WiredTiger or traditional column storage. If that doesn’t work, the data is moved to out-of-line storage.

Storage 126

Storage Efficiency Database Open Source 126

Scalegrid

JANUARY 20, 2020

Note: ScaleGrid implements follower clusters using storage snapshots. And since the entire import is performed using storage snapshots, rather than a logical dump, the process is nearly instantaneous. Follower clusters are a ScaleGrid feature that allows you to keep two independent database systems (of the same type) in sync. Unlike cloning or replication, this allows you to maintain an active, point-in-time copy of your production data.

Analytics 142

Analytics Database Systems Storage 142

Particular Software

MAY 11, 2017

If you're using Azure Storage Persistence and haven't upgraded to NServiceBus 6 yet, get ready for a tremendous performance boost for your application when you do especially if you make use of sagas. In the previous version of Azure Storage Persistence, looking up a saga by a correlation property was not as fast as looking it up by SagaId. Azure Table Storage, where saga data is stored, is limited to indexing on two columns: the Partition Key and the Row Key.

Storage 40

Storage Azure Performance Systems 40

All Things Distributed

SEPTEMBER 20, 2013

Not everybody agreed that the "N-ary Storage Model" (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives. Combined with the rise of data warehouse workloads, where there is often significant redundancy in the values stored in columns, and database models based on column oriented storage took off. A Decomposition Storage Model , George P.

Storage 62

Storage Database Hardware Cache 62

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. Building and managing archive storage that needs to remain operational for decades if not centuries is a major challenge for most organizations. A Complete Storage Solution. storage that is directly accessible. All Things Distributed.

Storage 69

Storage AWS Cloud Media 69

Scalegrid

NOVEMBER 12, 2019

Note: The community has already started work on the zheap storage engine that overcomes this limitation. The past several years have seen increasing adoption for PostgreSQL. PostgreSQL is an amazing relational database. Feature-wise, it is up there with the best, if not the best. There are many things I love about it – PL/ PG SQL, smart defaults, replication (that actually works out of the box), and an active and vibrant open source community.

Database 137

Database Open Source Storage Virtualization 137

All Things Distributed

FEBRUARY 6, 2012

Driving Storage Costs Down for AWS Customers. As we showed last week one of the services that is growing rapidly is the Amazon Simple Storage Service (S3). AWS today announced a substantial price drop per February 1, 2012 for Amazon S3 standard storage to help customers drive their storage cost down. Other storage tiers may see even greater cost savings. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems.

Storage 62

Storage AWS Scalability Infrastructure 62

All Things Distributed

FEBRUARY 20, 2015

RAID: High-Performance, Reliable Secondary Storage Peter Chen, Edward Lee, Garth Gibson, Randy Katz and David Patterson, ACM Computing Surveys, Vol 26, No. Disk arrays, which organize multiple, independent disks into a large, high-performance logical disk, were a natural solution to dealing with constraints on performance and reliability of single disk drives. The term "RAID" was invented by David Patterson, Garth A. Gibson, and Randy Katz at the University of California, Berkeley in 1987.

Storage 69

Storage Development Performance 69

High Scalability

APRIL 16, 2019

In this three-part blog series, we introduced a High Availability (HA) Framework for MySQL hosting in Part I, and discussed the details of MySQL semisynchronous replication in Part II. Now in Part III, we review how the framework handles some of the important MySQL failure scenarios and recovers to ensure high availability. MySQL Failover Scenarios. Scenario 1 – Master MySQL Goes Down. The Corosync and Pacemaker framework detects that the master MySQL is no longer available.

Availability 173

Availability Open Source Network Database 173

n0derunner

MARCH 9, 2020

storage fio samsung ssdStart by identifying the exact SSD type by using lsscsi. lsscsi [1:0:0:0] cd/dvd QEMU QEMU DVD-ROM 2.5+ /dev/sr0 [2:0:0:0] disk ATA SAMSUNG MZ7LM1T9 404Q /dev/sda [2:0:1:0] disk ATA SAMSUNG MZ7LM1T9 404Q /dev/sdb [2:0:2:0] disk ATA SAMSUNG MZ7LM1T9 404Q /dev/sdc [2:0:3:0] disk ATA SAMSUNG MZ7LM1T9 404Q /dev/. Device Specs. The spec sheet for this ssd claims the following performance characteristics.

Servers 63

Servers Performance Storage 63

n0derunner

JULY 15, 2020

Storage Hardware and Devices nvme optaneThe easiest way to identify NVME drives backed by either NAND flash or Optane is to use $ lspci -v The output will look like this for NVME/NAND 00:0d.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express]) Subsystem: Intel Corporation NVMe Datacenter SSD [3DNAND] SE 2.5" The post Identifying Optane drives in Linux appeared first on n0derunner.

Storage 40

Storage Hardware 40

Uber Engineering

DECEMBER 19, 2019

Architecture General Engineering CPU Infrastructure Observability Productivity Reliability Search Infrastructure Storage Uber Eats Velocity

Infrastructure 67

Infrastructure Transportation Network Storage 67

n0derunner

MARCH 30, 2020

diskspd benchmarking storage windowsInstalling Disk-Speed (diskspd). Get diskpd binary from Microsft : [link] Manual is here: [link] Agree to License Extract Zip file (DocumentsDiskspd-2.0.21a) open Terminal/Command Prompt cd to the Extracted directory cd to AMD64 Overview diskspd operates on windows filesystems, and will read / write to one or more files concurrently. The NULL byte problem By default, … The post Microsoft diskspd. Part 1 Preparing to test.

Speed 40

Speed Testing Storage Benchmarking 40

n0derunner

MARCH 30, 2020

diskspd benchmarking storage windowsInstalling Disk-Speed (diskspd). Get diskpd binary from Microsft : [link] Manual is here: [link] Agree to License Extract Zip file open Terminal/Command Prompt cd to the Extracted directory cd to AMD64 Overview diskspd operates on windows filesystems, and will read / write to one or more files concurrently. The NULL byte problem By default, when … The post Microsoft diskpd. Part 1 Preparing to test. appeared first on n0derunner.

Speed 40

Speed Testing Storage Benchmarking 40

DZone

MAY 8, 2019

performance monitoring apm log management log efficient log management and monitoring log management best practices log storageWhen managing cloud-native applications, it's essential to have end-to-end visibility into what's happening at any given time. This is especially true because of the distributed and dynamic nature of cloud-native apps, which are often deployed using ephemeral technologies like containers and serverless functions.

Best Practices 100

Best Practices Efficiency Serverless Monitoring 100

n0derunner

MARCH 13, 2020

To achieve the maximum throughput on a storage device, we will usually use a large IO size to maximize the amount of data is transferred per IO request. For historical reasons, many storage testers will use a 1MB IO size for sequential testing. SSD Performance benchmarking kernel linux ssd storageFirst things First. CDC 9762 SMD disk drive from 1974. Why do we tend to use 1MB IO sizes for throughput benchmarking?

Storage 43

Storage Benchmarking Testing Performance 43

Uber Engineering

OCTOBER 17, 2018

Architecture Uber Data Apache Apache Hadoop Apache Parquet Apache Spark Big Data Data Modeling Data Warehouse Docker Engineering Hadoop Hoodie Hudi JSON Latency MySQL PostgresSQL Storage Uber EngUber is committed to delivering safer and more reliable transportation across our global markets.

Big Data 69

Big Data Latency Transportation Traffic 69

Uber Engineering

AUGUST 3, 2018

Architecture Uber Data Cassandra Data Management Data Storage Data Warehouse Databook Dropwizard Gradle HDFS HIVE Infrastructure Kafka Metadata MySQL Postgres Quartz Queryparser RESTful API Uber Uber Data Knowledge Uber Engineering VerticaFrom driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data.

Big Data 70

Big Data Transportation Storage Infrastructure 70

Dynatrace

OCTOBER 20, 2020

To free disk space, log files related to metric and configuration storage, in addition to Elasticsearch log files, are now automatically deleted after two months. The Cluster nodes overview page now displays Dynatrace version and transaction storage size details for each node. Previous metric storage and configuration data backups are now removed once the new backups are completed. Dynatrace news. New features and enhancements. General.

Storage 124

Storage Metrics Systems 124

The Netflix TechBlog

OCTOBER 19, 2020

Our distributed tracing infrastructure is grouped into three sections: tracer library instrumentation, stream processing, and storage. A second job taps the data feed from the first job, does tail sampling of data and writes traces to the storage system. Storage: don’t break the bank!

Infrastructure 160

Infrastructure Storage Transportation Open Source 160

SQL Shack

AUGUST 17, 2020

The storage subsystem is one of the significant performance factors for the databases. In this article, we will discuss how to resolve I/O problems that is a very important point for the SQL Server troubleshooting.

Storage 76

Storage Database Servers Monitoring 76

All Things Distributed

AUGUST 24, 2012

The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 The Five-Minute Rule Ten Years Later, and Other Computer Storage Rules of Thumb , Jim Gray and Goetz Graefe, ACM SIGMOD Record 26 (4): 63â??68, In 2007 Goetz revisited the results, this time also including SSD storage into the mix, which not surprisingly did change everything. All Things Distributed.

Hardware 62

Hardware Data Engineering Scalability Engineering 62

All Things Distributed

DECEMBER 27, 2012

S3 is not only a highly reliable and available storage service but also one of the most powerful web serving engines that exists today. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. My Best Christmas Present â?? Root Domain Support for Amazon S3 Website Hosting. By Werner Vogels on 27 December 2012 12:00 PM. Permalink. Comments ().

Traffic 71

Traffic Scalability Infrastructure Engineering 71