Uber Engineering

FEBRUARY 22, 2018

As our … The post Code Migration in Production: Rewriting the Sharding Layer of Uber’s Schemaless Datastore appeared first on Uber Engineering Blog. For context, we deployed more than 40 Schemaless instances and many thousands of storage nodes in 2016 alone.

Code 94

Code Storage Scalability Engineering 94

DZone

JANUARY 2, 2024

What Is Sharding? Sharding, a database architecture pattern, involves partitioning a database into smaller, faster, more manageable parts called shards. Each shard is a distinct database, and collectively, these shards make up the entire database.

Architecture 260

Architecture Database Scalability Servers 260

Percona

FEBRUARY 21, 2024

Backups are crucial for every database system, and having a reliable, fast, and hot backup is the demand for next-generation database systems. Percona Backup for MongoDB (PBM) is a backup management tool that enhances the existing backup capability of MongoDB by providing various layers of backups such as physical, logical, incremental, PITR, etc.In

Database 97

Database Systems Software Software 97

Baron Schwartz

FEBRUARY 14, 2019

I wrote yesterday about Vitess , a scale-out sharding solution for MySQL. Another similar product is Citus , which is a scale-out sharding solution for PostgreSQL. Citus solves the following problems for users: Sharding. Citus handles all of the sharding, so applications do not need to be shard-aware.

Analytics 40

Analytics Open Source Database Innovation 40

Percona

SEPTEMBER 28, 2023

To replace a live node from shards/replica set Let’s say it’s an EBS snapshot. In this case, follow the below steps: Add newly launched node to the shard from the existing primary node. rs.remove("old-shard-host:27018") rs.conf() Shut down the mongo service from the old node. Copy the hot backups to it. priority = 0.5

Open Source 85

Open Source Cloud Servers Testing 85

The Netflix TechBlog

MARCH 10, 2023

Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries. Hence, we used the asset types and time buckets based on asset creation date for data sharding across the Cassandra nodes.

Media 237

Media Traffic Processing Design 237

Percona

NOVEMBER 2, 2023

x: Live resharding of databases for uninterrupted sharded key changes. Sharded time-series collections for improved scalability and performance. The upgrade process : Whether you’re dealing with a standalone system, a replica set, or a sharded cluster, we outline the steps to take, ensuring a smooth MongoDB upgrade experience.

Best Practices 92

Best Practices Hardware Tuning Scalability 92

Percona

JANUARY 19, 2023

Sharding: Sharding is the concept of splitting data horizontally, i.e. by distributing data into multiple servers (shards), meaning that the different portions of data for a given table, may be stored on many different servers. It supports native sharding being transparent for the application. It supports native sharding.

Open Source 93

Open Source Storage Database Big Data 93

Percona

APRIL 24, 2023

It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time. Download Percona XtraDB Cluster 8.0.32-24

Open Source 81

Open Source Storage Servers Database 81

Percona

OCTOBER 2, 2023

It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time. Download Percona Distribution for PostgreSQL 16.0

Open Source 84

Open Source Scalability Database Servers 84

Percona

DECEMBER 4, 2023

It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time. Download Percona XtraBackup 8.0.35-30

Open Source 84

Open Source Azure Servers Programming 84

Percona

AUGUST 7, 2023

It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time. Percona Backup for MongoDB 2.2.1

Open Source 88

Open Source Scalability Servers Database 88

The Netflix TechBlog

SEPTEMBER 29, 2022

All data structures related to a queue — from the messages it contains to the in-memory secondary indexes needed to support dequeue-by-filter — are placed in the same Redis shard. As noted above (see “System of record” section), a queue and its contents can only currently occupy one Redis shard. Hot queues however can grow big, esp.

Latency 212

Latency Systems Media Serverless 212

Percona

NOVEMBER 20, 2023

PBM will notify when the data will be ready for copying with the prompt saying from exactly which node on each shard it should be done (backup needs to be done only from one node on each replica set). PBM will look into the actual data during the restore and define the most recent common cluster time across all shards.

Storage 87

Storage Database Strategy Speed 87

Percona

MARCH 13, 2023

It includes Percona Server for MongoDB, a fully compatible source-available, drop-in replacement for MongoDB, and Percona Backup for MongoDB, a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets. This release is based on Percona Server for MongoDB 4.2.24-24

Servers 83

Servers Open Source Database Software 83

Percona

FEBRUARY 27, 2023

It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time. Percona Backup for MongoDB 2.0.4 was released.

Monitoring 72

Monitoring Open Source Database Software 72

Percona

JULY 24, 2023

Percona Backup for MongoDB 2.2.0 , a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets, was released on July 13, 2023. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set) and for restoring those backups to a specific point in time.

Open Source 76

Open Source Scalability Database Monitoring 76

The Netflix TechBlog

MARCH 7, 2024

We use metaflow.Table to resolve all input shards which are distributed to Metaflow tasks which are responsible for processing terabytes of data collectively. Each task loads the data using metaflow.MetaflowDataFrame, performs matching using Pandas, and populates a corresponding shard in an output Table.

Systems 226

Systems Media Cache Open Source 226

Percona

FEBRUARY 10, 2023

Assuming a simple 3-node Percona Server for MongoDB cluster without sharding and nothing else in your current namespace, you should be able to run the following commands to allow for automatic storage resizing of all the PVCs. In action You first need to add the following annotation to your Storage Class: resize.topolvm.io/enabled=true

Storage 114

Storage Blockchain AWS Cloud 114

High Scalability

SEPTEMBER 10, 2018

You know this because in a browser, no matter where you are in the world, you navigate to facebook.com, netflix.com, or google.com, you never enter the URL for independent shards, yet all your data and services magically follow you around like a hyperactive puppy. That's the world we've come to expect.

Cloud 144

Cloud Internet Internet Google 144

Percona

DECEMBER 27, 2023

And with multi-cloud in mind, we consistently incorporate new sharding functionalities into our roadmaps, such as the upcoming Shard-per-location support in the Percona Operator for MongoDB.

Cloud 84

Cloud Open Source Strategy Best Practices 84

DZone

MARCH 14, 2023

Shuffle-sharding is a valuable technique to achieve this. By dividing resources into equal segments and periodically shuffling them, shuffle-sharding can distribute resources evenly and prevent any client or task from relying on a specific segment for too long.

Systems 130

Systems Availability Performance Development 130

Percona Community

APRIL 18, 2019

The topic I’m presenting addresses a growing and unfulfilled need: the ability to run stateful workloads in Kubernetes. Running stateless applications is now considered a solved problem. However, it’s currently not practical to put databases like MySQL in containers, give them to Kubernetes, and expect it to manage their life cycles.

Database 52

Database 52

Percona Community

OCTOBER 2, 2020

Abstract Kunlun Distributed Database Cluster is a distributed DBMS that aims to combine the best of both MySQL and PostgreSQL for a highly performant, highly available, highly scalable and fault-tolerant, easy to use and manage database system that requires minimal human maintenance.

Storage 52

Storage Scalability Database Availability 52

Percona

OCTOBER 10, 2023

Our Operators Team is also working on improvements especially important for sharded clusters, and the Percona Monitoring and Management team is planning to look into more scalability and management-enhancing options. We also want to focus more on the security aspects not available outside of MongoDB Enterprise.

Availability 105

Availability Servers Database Open Source 105

Percona

MARCH 10, 2023

A capped collection can’t be sharded. Capped collection is useful in storing the log information as it keeps the data ordered by the events. Disadvantage: There are some restrictions of the capped collection. A capped collection can’t have TTL indexes.

Open Source 109

Open Source Storage Speed Servers 109

Percona Community

OCTOBER 9, 2020

Sharded environment poses an additional burden, as any logical migration must be applied multiple times, once for each shard.

Servers 52

Servers 52

Percona Community

JULY 16, 2019

In one of my past blogs I explained the contribution of MongoDB® views in organization security. In this blog, I will take it one step further and I will try to approach the concept of a materialized view in MongoDB. In computing, a materialized view is a database object that contains the results of a query (definition taken from Wikipedia).

Database 40

Database 40

Scalegrid

OCTOBER 2, 2019

ScaleGrid’s hosting for Redis™ service allows these organizations to automate all of their time-consuming management tasks, such as backups, upgrades, scaling, replication, sharding, monitoring, alerts, log rotations, and OS patching, so their DBAs, developers, and DevOps teams can focus on new product development and optimizing performance.

Azure 100

Azure Availability Open Source Virtualization 100

Percona Community

DECEMBER 4, 2020

Overview In this article, I look at how poorly designed sharding systems and replication systems in PostgreSQL affect query performance in high volume situations, and how to ensure data consistency across many servers.

Performance 52

Performance Cache Traffic Database 52

Percona Community

OCTOBER 12, 2020

Percona Live Online Agenda Slot: Wed 21 Oct • New York 7:00 a.m. London 12:00 noon • New Delhi 4:30 p.m. Singapore 7:00 p.m. Why is your talk exciting? I’m not sure if my talk is exciting, but I’m quite positive the subject is!

40

40

Percona

SEPTEMBER 1, 2023

In this blog post, we will talk about MongoDB sharding and walk through the main reasons why you should start a cluster (independent of the approach you have chosen). What is sharding in MongoDB? Sharding in MongoDB is a technique used to distribute a database horizontally across multiple nodes or servers, known as “shards.”

Best Practices 74

Best Practices Traffic Database Servers 74

Percona Community

APRIL 1, 2020

However, an application’s life doesn’t end with deployment, so today we will update our application and figure out how to manage topology, sharding, and authorization, and change the role configuration. We have recently talked about how to deploy a Tarantool Cartridge application. Feeling interested? Where did we leave off?

52

52

Percona Community

SEPTEMBER 18, 2019

released like a month ago (still newborn) and I am going to cover what’s new in three different areas: sharding, indexing, and the aggregation framework. It’s all about MongoDB® 4.2 I can promise you this, there are a lot of new features and improvements in MongoDB 4.2 and I am thrilled to present those to you.

52

52

Scalegrid

FEBRUARY 7, 2019

Redis Cluster is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. Sharding with Redis Cluster. This makes adding and removing shards, retiring, or scaling individual nodes easy.

Java 138

Java Database Availability AWS 138

Scalegrid

JANUARY 5, 2024

These can be mitigated through the implementation of: efficient query optimization caching of database queries utilization of database indexes implementation of session storage employing database read replication and sharding.

Efficiency 147

Efficiency Storage Database Scalability 147

Scalegrid

DECEMBER 14, 2023

To ensure optimal performance, load balancing techniques such as sharding can be used which spread out client requests over various MongoDB instances. Timeouts may occur due to poor network connections or if the primary node fails to reply within an allocated period.

Java 130

Java Network Servers Database 130