All Things Distributed

MARCH 23, 2021

Back when S3 launched on March 14, 2006 (also known as “Pi Day” to some), iPhones didn’t exist, neither did tweets or likes, Facebook was still only used by a few colleges and universities, and you couldn’t hail a ride or order lunch with an app. 15 years is a long time in the world of technology.

Ecommerce 216

Ecommerce Retail Storage Scalability 216

Dynatrace

MARCH 27, 2024

AWS AppFabric ingests and normalizes audit logs from SaaS applications and delivers them to an Amazon Simple storage service (Amazon S3) bucket in an organization’s AWS account. Follow the instructions here to deploy the Dynatrace AWS S3 Log Forwarder on your AWS Account. Note that AWS AppFabric is available in the US East (N.

AWS 245

AWS Analytics Government Monitoring 245

Percona

FEBRUARY 22, 2024

In our last post, we looked into the lifecycle of applications in Kubernetes. We see that Kubernetes doesn’t handle database backups itself. This is where Kubernetes Operators come into action.

AWS 85

AWS Database Cloud Software 85

DZone

SEPTEMBER 25, 2023

The implemented storage is based on MinIO as an AWS S3-compatible object storage service. This article focuses on the model file storage using an approach also applicable and used for machine learning models. Furthermore, the article gives an overview of alternative storage solutions and outlines the benefits of object-based storage.

Storage 258

Storage AWS Architecture Cloud 258

All Things Distributed

DECEMBER 27, 2012

Root Domain Support for Amazon S3 Website Hosting. I have been a big fan of the Amazon S3 Static Website Hosting feature since its launch and this blog happily is being served from it. S3 is not only a highly reliable and available storage service but also one of the most powerful web serving engines that exists today.

Website 127

Website Traffic Storage Scalability 127

The Polyglot Developer

OCTOBER 17, 2018

If you’re using Amazon S3 for a CDN, it’s possible to serve compressed, gzipped files from an Amazon S3 bucket, though there are a few extra steps beyond the standard process of serving compressed files from your own web server.

Servers 52

Servers Processing Development 52

Scalegrid

FEBRUARY 8, 2024

Amazon S3 and Microsoft Azure Blob Storage leverage distributed storage solutions. Within Amazon S3’s offerings are features like metadata tagging, different classes of data movement and storage options, configuring control over access permissions, and ensuring safety against disasters through data replication mechanisms.

Storage 130

Storage Systems Big Data Azure 130

Smashing Magazine

NOVEMBER 8, 2018

Sharing Data Among Multiple Servers Through AWS S3. Sharing Data Among Multiple Servers Through AWS S3. But since this article is about connecting an application running on the server with AWS S3, we don’t consider this solution. Note 3: There are 3rd party plugins for uploading files from WordPress to S3.

AWS 44

AWS Servers Storage Cloud 44

All Things Distributed

AUGUST 17, 2011

No Server Required - Jekyll & Amazon S3. As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. Just dropping your website in an S3 bucket brings all that power to you. Amazon S3 FTW! Comments ().

Servers 126

Servers Social Media AWS Website 126

The Polyglot Developer

APRIL 18, 2016

I’m actually hosting these podcasts using Amazon’s S3 service because it is cheap and incredibly easy to get set up with. The post Host A Podcast For Cheap On Amazon’s S3 Service appeared first on The Polyglot Developer. This is a great question!

Database 59

Database Development 59

Percona

MARCH 13, 2024

Now, here we will see some advanced stuff and other backup/restore options available with PBM.Let’s discuss each one.Taking backups on remote storage (AWS S3/Google […]

AWS 88

AWS Storage Google Availability 88

Percona

OCTOBER 18, 2023

tasks: - name: daily-s3-physical enabled: true keep: 3 type: physical storageName: s3-us-west Now, you can enable PITR. With our Operator, users can leverage integration with AWS KMS or just encrypt/decrypt backups with AES-256 encryption algorithm with any S3-compatible storage. It can be done either through setting tasks.[].type

Best Practices 90

Best Practices Storage AWS Big Data 90

Percona

FEBRUARY 10, 2023

This IAM role can be queried using EC2 instance metadata to access an S3 bucket. 24 , xbcloud can read instance metadata and fetch credentials from an instance profile, utilizing it to authenticate against an S3 bucket. Xbcloud is a tool part of Percona XtraBackup and allows you to upload and download backups to Amazon S3 storage.

Storage 91

Storage Open Source AWS Servers 91

Percona

MAY 15, 2023

The interesting part is that we can have multiple kinds of backup solutions, like: On-demand Scheduled Full Point in time recovery with log streaming Automation will allow us to set a schedule as simple as this: schedule: - name: "sat-night-backup" schedule: "0 0 * * 6" keep: 3 storageName: s3-eu-west - name: "daily-backup" schedule: "0 3 * * *" keep: (..)

Traffic 122

Traffic Scalability Database Servers 122

Dynatrace

JULY 25, 2022

This new service enhances the user visibility of network details with direct delivery of Flow Logs for Transit Gateway to your desired endpoint via Amazon Simple Storage Service (S3) bucket or Amazon CloudWatch Logs. What is AWS Transit Gateway? What can you expect from VPC Flow Logs for Transit Gateway.

AWS 217

AWS Transportation Network Traffic 217

Percona

MARCH 28, 2023

A few important options for repo2, available in the AWS S3:repo-type is s3 indicating the AWS S3, and it could be azure for Azure Cloud, GCS for Google Cloud,repo2-s3-bucket, repo2-s3-endpoint, repo2-s3-key-secret, and repo2-s3-region attributes varies from cloud to cloud.

Servers 81

Servers Azure AWS Cloud 81

All Things Distributed

AUGUST 17, 2011

As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable.

Servers 60

Servers Storage Website Scalability 60

Dynatrace

OCTOBER 7, 2021

Amazon S3: The Simple Storage Service stores and retrieves data from anywhere with scalability, data availability, security, performance, and a high degree of durability. AWS AppSync: AppSync offers a fully managed approach to developing APIs with GraphQL — connecting to AWS DynamoLB or Lambda along with adding caches and client-side data.

Serverless 280

Serverless AWS Lambda Storage 280

Dynatrace

APRIL 8, 2021

Amazon S3 and CloudFront provide a perfect solution for hosting and serving such static assets. AWS Lambda as well as S3 are mostly billed based on consumption. Serving single-page applications with S3 and Lambda. Yet, the HTML and JavaScript of a SPA must still be hosted somewhere.

Lambda 221

Lambda AWS Serverless Monitoring 221

All Things Distributed

NOVEMBER 9, 2009

At the end of Q3 2009 we counted over 82 billion objects in Amazon S3. Congrats to the team for providing such a rock solid service! When looking at the graph keep in mind that the first 4 markers are a year apart, but the last one only 6 months.

78

78

All Things Distributed

MARCH 18, 2008

A few days ago, on March 14, Amazon S3 quietly celebrated its 2nd birthday. That S3 is growing up fast is obvious from the number of objects the customers trust us with. I think congratulations are in order and I certainly wish the service "many happy returns of this day".

78

78

DZone

JANUARY 29, 2020

For some internal reasons well known Amazon S3 bucket was chosen for this purpose. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). To resolve the problem it was suggested to find more suitable data storage. The choice affected the project's unit test base.

Storage 124

Storage Efficiency Testing Database 124

All Things Distributed

FEBRUARY 23, 2011

Free at Last - A Fully Self-Sustained Blog Running in Amazon S3. With these last two dependencies gone this blog is now 100% self-sustained in Amazon S3 and does no longer depend on any server I have to maintain. No Server Required - Jekyll & Amazon S3. Free at Last - A Fully Self-Sustained Blog Running in Amazon S3.

AWS 79

AWS Storage Big Data Servers 79

Dynatrace

APRIL 8, 2021

Amazon S3 and CloudFront provide a perfect solution for hosting and serving such static assets. AWS Lambda as well as S3 are mostly billed based on consumption. Serving single-page applications with S3 and Lambda. Yet, the HTML and JavaScript of a SPA must still be hosted somewhere.

Lambda 182

Lambda AWS Serverless Monitoring 182

The Netflix TechBlog

MARCH 7, 2024

Data: Fast Data Our main data lake is hosted on S3, organized as Apache Iceberg tables. Once we have discovered the Parquet files to be processed, MetaflowDataFrame takes over: it downloads data using Metaflow’s high-throughput S3 client directly to the process’ memory, which often outperforms reading of local files.

Systems 226

Systems Media Cache Open Source 226

The Netflix TechBlog

DECEMBER 8, 2021

Step Function DAG for S3 Public Access Block Remediation Diagram from a remediation to enable S3’s public access block on a non-compliant bucket. We’re still exploring new ways we might be able to use this, and are excited for how we might evolve our approach in the near future.

AWS 196

AWS Cloud Infrastructure Scalability 196

Percona

APRIL 26, 2023

Results PBM supports different storage types, but for this implementation, we decided to start with the most widely used – S3 compatible. 3xlarge (12vCPU,9 6GB RAM) 5Tb ~2280 min * Tests were made on AWS i3en instances with the S3 storage in the same region. ** We didn’t wait for 5Tb restore on v2.0.3

Speed 78

Speed Storage AWS Azure 78

Dynatrace

NOVEMBER 20, 2023

Accelerate your cloud journey with Dynatrace observability for AWS S3 logs – blog The AWS S3 storage location can now be easily integrated with Dynatrace Grail™ data lakehouse, so you can answer any question at any time using log data and avoid the complexity of storage management.

AWS 208

AWS Artificial Intelligence Analytics Cloud 208

All Things Distributed

NOVEMBER 5, 2007

Expo in Berlin where we will announce the ability for Amazon S3 customers to store their objects in Amazon’s European storage cloud. I am about to go on stage at Web 2.0 This has been a frequently requested feature by our European customers for various reasons, better latency control being the most important one.

Latency 60

Latency Storage Cloud 60

Dynatrace

SEPTEMBER 23, 2021

For this, the development team builds a web application and successfully deploys it in AWS, using S3 for the front end, AWS API Gateway, AWS Lambda for Python, and DynamoDB in the back end. Let’s say that to promote a new offering, a telecom company organizes a contest with prizes for its customers.

AWS 223

AWS Lambda Serverless Metrics 223

DZone

JULY 10, 2019

The Apache Spark + Alluxio stack is getting quite popular particularly for the unification of data access across S3 and HDFS. In addition, compute and storage are increasingly being separated causing larger latencies for queries. Alluxio is leveraged as compute-side virtual storage to improve performance.

Best Practices 133

Best Practices Latency Storage Tuning 133

Dynatrace

SEPTEMBER 7, 2022

With Amazon Web Services, the main sources from which to ingest logs—Simple Storage Service, or S3, and CloudWatch —come with an additional cost. The Dynatrace platform can ingest and evaluate a large amount of data from different sources.

Traffic 215

Traffic AWS Network Cloud 215

The Morning Paper

MARCH 8, 2020

Customer data is persisted in S3 (or the equivalent services when running on Azure or GCP), and compute is handled in EC2 instances. If all data was read from S3 every time, performance would suffer, so of course Snowflake has a caching layer – a distributed ephemeral storage service shared by all the nodes in a warehouse.

Storage 112

Storage Engineering Cache Serverless 112

DZone

JUNE 25, 2019

And do you remember the time when Amazon had to globally reboot their entire S3 service due to a single bit error? Google’s in-depth study of bit-level DRAM errors showed that such uncorrectable errors are a fact of life. The Error-Prone Data Trail.

Storage 145

Storage Systems Transportation Network 145

Percona

FEBRUARY 13, 2023

An IAM instance profile does not need the --s3-secret-key nor the --s3-access-key but the profile has a six-hour lifespan. Offered free as an open source solution, Percona XtraBackup drives down backup costs while providing unique features for MySQL backups. With this version, qpress / QuickLZ are deprecated.

Open Source 82

Open Source Database Servers Scalability 82

Adrian Cockcroft

MAY 6, 2023

Then they tried to scale it to cope with high traffic and discovered that some of the state transitions in their step functions were too frequent, and they had some overly chatty calls between AWS lambda functions and S3.

Serverless 145

Serverless Lambda Best Practices Traffic 145

All Things Distributed

OCTOBER 4, 2006

Jeremy Zawodny has some interesting calculations on the absolute cost advantage of using S3 for your home backup server. For me however the most compelling argument is later in his posting when he mentions the administrative cost of running a highly available home server.

Servers 60

Servers Availability 60