INDUSTRY INSIGHTS YOUR PEERS ARE READING
Previously, I wrote about Amazon QuickSight , a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost. MORE
WirelessCar is also using a host of AWS security services to ensure that the data it collects for its customers is secure, and also that the services the company offers are reliable as well. MORE
With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. MORE
Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. All Things Distributed. MORE
Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. big data performance data storage ssd nvme gpu ai ml MORE
Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions. MORE
Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. We’ve seen similar high marshalling overheads in big data systems too.) MORE
Seamless ingestion of large volumes of sensed data. AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift. Advanced problem solving that connects big data with machine learning. MORE
The council has deployed IoT Weather Stations in Schools across the City and is using the sensor information collated in a Data Lake to gain insights on whether the weather or pollution plays a part in learning outcomes. MORE
big-data processing, machine learning, quantum computing, and so on). Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. For those of us who pursued computer architecture as a career, this is well understood. MORE
Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. Design a flexible data model ? —?Represent MORE
It’s awesome for discovering how grid systems, CSS animation, Big Data, etc all play roles in real-world web design. It includes tutorials, links to data-visualization tools, design resources and articles that cite real-world business experiments. MORE
We believe that with the launch of the Seoul Region, AWS will enable many more enterprise customers in Korea to reduce the cost of their IT operations and innovate faster in critical new areas such as big data analysis, Internet of Things, and more. MORE
The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. MORE
Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. MORE
AWS data centers in Canada will draw from a regional electricity grid that is 99 percent powered by hydropower. Having the ability to provide these services locally enables Box to better serve Canadian enterprises looking for cloud solutions while ensuring their data is stored inside Canada. MORE
Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse. MORE
The new European region, coupled with the existing AWS Regions in Dublin and Frankfurt, and a future one in London, will provide customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, Big Data analysis, Internet of Things applications, and more. MORE
Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., For the services under study, Seer has a sweet spot when trained with around 100GB of data and a 100ms sampling interval for measuring queue depths. MORE
NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases. MORE
All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? introducing the Asia Pacific (Sydney) Region. By Werner Vogels on 12 November 2012 05:00 AM. Permalink. Comments (). MORE
Many of us are lucky enough to be on mobile plans which allow several gigabytes of data transfer per month. Failing that, we are usually able to connect to home or public WiFi networks that are on fast broadband connections and have effectively unlimited data. The Cost Of Mobile Data. MORE
Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Alternatively, a number of organizations have created their own internal home-grown systems for managing and distilling web performance and monitoring data. MORE
Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Obviously, containerized data workloads have their own challenges. MORE
The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. MORE
The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden. MORE
Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. Applications: ETL, Data Analysis. MORE
This region will provide even lower latency and strong data sovereignty to local users. Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. MORE
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN MORE
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture. Common in-memory data interfaces. MORE
We live in a world where massive volumes of data are generated from websites, connected devices and mobile apps. However, the data infrastructure to collect, store and process data is geared toward developers (e.g., Big data challenges. MORE
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements). MORE
From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog. MORE
At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. MORE
To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets. MORE
How companies can use ideas from mass production to create business with data. In this way, designers are part of an ecosystem in which the functionalities of simulations, data and people come together, enabling them to develop better products faster. Value creation through data. MORE
The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data. MORE
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. MORE
Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. MORE
Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley. USENIX’s LISA conference is the premier event for topics in production system engineering. MORE
In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch. MORE
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster MORE
how many messages was send for some time period and how much it cost) and a typical key/value queries like: “return 1 message by the message id” Using a columnar analytical database can be a big challenge here. In my case, I’m using this data as a simulation of text messages, and will show how we can use ClickHouse as a backend for an API. Loading the JSON data to Clickhouse. Updating / deleting data in ClickHouse. MORE
Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods. MORE
Big Data Related Topics Highly Scalable
AUGUST 20, 2013
The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.
Rigor
MAY 22, 2019
Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Alternatively, a number of organizations have created their own internal home-grown systems for managing and distilling web performance and monitoring data.
Abhishek Tiwari
DECEMBER 27, 2017
Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Obviously, containerized data workloads have their own challenges.
Uber Engineering
OCTOBER 17, 2018
To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.
The Morning Paper
MAY 14, 2019
Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., For the services under study, Seer has a sweet spot when trained with around 100GB of data and a 100ms sampling interval for measuring queue depths.
Uber Engineering
AUGUST 3, 2018
From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.
DZone
JULY 29, 2019
In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.
DZone
JUNE 4, 2019
NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.
DZone
JUNE 3, 2019
Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. big data performance data storage ssd nvme gpu ai ml
Uber Engineering
AUGUST 14, 2019
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.
Highly Scalable
MAY 1, 2012
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).
Uber Engineering
APRIL 5, 2018
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN
All Things Distributed
AUGUST 18, 2011
Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. All Things Distributed.
Highly Scalable
MARCH 10, 2015
Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods.
Highly Scalable
MARCH 1, 2012
At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques.
Uber Engineering
OCTOBER 30, 2018
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster
Highly Scalable
JANUARY 31, 2012
Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. Applications: ETL, Data Analysis.
Highly Scalable
JUNE 5, 2012
Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures.
Abhishek Tiwari
DECEMBER 27, 2017
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture. Common in-memory data interfaces.
DZone
MAY 21, 2019
The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations.
DZone
MAY 30, 2019
With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous.
The Netflix TechBlog
APRIL 29, 2019
Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions.
ACM Sigarch
APRIL 8, 2019
big-data processing, machine learning, quantum computing, and so on). Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. For those of us who pursued computer architecture as a career, this is well understood.
The Morning Paper
JUNE 23, 2019
Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. We’ve seen similar high marshalling overheads in big data systems too.)
Abhishek Tiwari
DECEMBER 22, 2017
Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse.
All Things Distributed
NOVEMBER 5, 2015
This region will provide even lower latency and strong data sovereignty to local users. Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific.
Brendan Gregg
APRIL 30, 2018
Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley. USENIX’s LISA conference is the premier event for topics in production system engineering.
Smashing Magazine
JULY 29, 2019
Many of us are lucky enough to be on mobile plans which allow several gigabytes of data transfer per month. Failing that, we are usually able to connect to home or public WiFi networks that are on fast broadband connections and have effectively unlimited data. The Cost Of Mobile Data.
All Things Distributed
DECEMBER 13, 2016
The council has deployed IoT Weather Stations in Schools across the City and is using the sensor information collated in a Data Lake to gain insights on whether the weather or pollution plays a part in learning outcomes.
All Things Distributed
SEPTEMBER 5, 2013
Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data.
All Things Distributed
JUNE 26, 2016
Seamless ingestion of large volumes of sensed data. AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift. Advanced problem solving that connects big data with machine learning.
KeyCDN
OCTOBER 2, 2018
It’s awesome for discovering how grid systems, CSS animation, Big Data, etc all play roles in real-world web design. It includes tutorials, links to data-visualization tools, design resources and articles that cite real-world business experiments.
All Things Distributed
OCTOBER 7, 2015
We live in a world where massive volumes of data are generated from websites, connected devices and mobile apps. However, the data infrastructure to collect, store and process data is geared toward developers (e.g., Big data challenges.
All Things Distributed
DECEMBER 12, 2018
WirelessCar is also using a host of AWS security services to ensure that the data it collects for its customers is secure, and also that the services the company offers are reliable as well.
All Things Distributed
JANUARY 6, 2016
We believe that with the launch of the Seoul Region, AWS will enable many more enterprise customers in Korea to reduce the cost of their IT operations and innovate faster in critical new areas such as big data analysis, Internet of Things, and more.
All Things Distributed
NOVEMBER 15, 2016
Previously, I wrote about Amazon QuickSight , a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost.
All Things Distributed
JUNE 20, 2017
The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong.
All Things Distributed
APRIL 4, 2017
The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden.
All Things Distributed
NOVEMBER 12, 2012
All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud â?? introducing the Asia Pacific (Sydney) Region. By Werner Vogels on 12 November 2012 05:00 AM. Permalink. Comments ().
Percona
JANUARY 14, 2019
how many messages was send for some time period and how much it cost) and a typical key/value queries like: “return 1 message by the message id” Using a columnar analytical database can be a big challenge here. In my case, I’m using this data as a simulation of text messages, and will show how we can use ClickHouse as a backend for an API. Loading the JSON data to Clickhouse. Updating / deleting data in ClickHouse.
All Things Distributed
SEPTEMBER 29, 2016
The new European region, coupled with the existing AWS Regions in Dublin and Frankfurt, and a future one in London, will provide customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, Big Data analysis, Internet of Things applications, and more.
All Things Distributed
DECEMBER 20, 2017
How companies can use ideas from mass production to create business with data. In this way, designers are part of an ecosystem in which the functionalities of simulations, data and people come together, enabling them to develop better products faster. Value creation through data.
All Things Distributed
DECEMBER 8, 2016
AWS data centers in Canada will draw from a regional electricity grid that is 99 percent powered by hydropower. Having the ability to provide these services locally enables Box to better serve Canadian enterprises looking for cloud solutions while ensuring their data is stored inside Canada.
88 
Let's personalize your content