INDUSTRY INSIGHTS YOUR PEERS ARE READING
One topic that always gets me excited is how to take computer science research and implement it in production systems. real systems do not fail by stopping in a nice and clean way). This weekend I am travelling to Australia for the first AWS Summit of 2017. MORE
There is a high cost associated with work that leaves your team… team boundaries and software boundaries should be isomorphic” — James Lewis, Thoughtworks I’ve written and spoken a lot about architecting sociotechnical systems and how to find boundaries. MORE
Based on our experience running business systems in production, we know we need to monitor our theme park to make sure it's working properly. How many CPU cycles is a system using? Infrastructure monitoring tools generally treat systems as "black boxes" that consume resources. MORE
In the past I've shared similar methodologies applied to other operating systems, and finished porting them to BSD for this talk. For my first trip to Paris I gave the closing keynote at [EuroBSDcon 2017] on performance methodologies, using FreeBSD 11.1 as an analysis target. MORE
Performance monitoring is a must to do the task for a DBA. You should ensure that the database performance is optimal all the time without any impact on the databases. Performance issues act like an open stage, and you need to look at every aspect such as CPU, RAM, server performance, database performance, indexes, blocking, […]. Deadlocks MORE
At the height of the Cold War, the United States passed the Federal Aid Highway Act of 1956, giving birth to the Interstate Highway System. They can be observed in our software systems as well. What if the system was built to allow junior developers to actively participate? MORE
Thus, whenever a master MySQL goes down (whether due to a MySQL crash, OS crash, system reboot, etc.), This ensures that the system continues to be available to the applications. MORE
For small deployments, it isn’t a problem however when scaling up to hundred or even thousands of systems things can become complicated. Even when all the systems are mapped correctly by Dynatrace, identifying these systems is a real challenge. S _ for the system. MORE
Let’s assume for a moment that your data survives its many passes through a system’s DRAM and emerges intact. That data must then be safely transported over a network to the storage system where it is written to disk. Random bit flips are far more common than most people, even IT professionals, think. Surprisingly, the problem isn’t widely discussed, even though it is silently causing data corruption that can directly impact our jobs, our businesses, and our security. MORE
This is the first post in a series of posts on different approaches to systems security especially as they apply to hardware and architectural security. In this post, we will consider the use of mathematics/cryptography as an approach to improving systems security. MORE
Gitlab had a very public outage last month. Most companies provide some kind of explanation when their services are interrupted. Those are usually sanitized (or seem sanitized) to make things seem better than they actually are. MORE
This week, the developers of Amazon Aurora have won the 2019 Association for Computing Machinery's (ACM) Special Interest Group on Management of Data (SIGMOD) Systems Award. MORE
Computer systems, from the Internet-of-Things devices to datacenters, are complex and optimizing them can enhance capability and save money. Developing simulators, however, is time-consuming and requires a great deal of infrastructure development regarding a prospective system. MORE
Get the latest insights on MySQL , MongoDB , PostgreSQL , Redis , and many others to see which database management systems are most favored this year. Based on our findings, SQL still holds 60% with rising demand for systems such as PostgreSQL: SQL Database Use: 60.48%. MORE
In reality, in any non-trivial installation, there are multiple tools collecting, storing and displaying overlapping sets of metrics from many types of systems and different levels of abstraction. What if your monitoring systems fail? “Quis custodiet ipsos custodes?”?—?Juvenal MORE
PyTorch-BigGraph: a large-scale graph embedding system Lerer et al., SysML’19. We looked at graph neural networks earlier this year, which operate directly over a graph structure. MORE
The M/M/1 queue also assumes that the arrival rate is not affected by the unbounded number of tasks in the queue (called an “open system”). With two blog posts, we argue for more use of simple models beyond Amdahl’s Law. MORE
Ginseng: keeping secrets in registers when you distrust the operating system Yun & Zhong et al., Suppose you did go to the extreme length of establishing an unconditional root of trust for your system, even then, unless every subsequent piece of code you load is also fully trusted (e.g., MORE
PostgreSQL is an open source object-relational database system that has soared in popularity over the past 30 years from its active, loyal, and growing community. For the 2nd year in a row, PostgreSQL has kept the title of #1 fastest growing database in the world according to the DBMS of the Year report by the experts at DB-Engines. So what makes PostgreSQL so special, and how is it being used today? MORE
” This paper describes a “far memory” system that has been in production deployment at Google since 2016. The objective is to find the lowest cold age threshold that still allows the system to satisfy its performance constraints. MORE
Teaching rigorous distributed systems with efficient model checking Michael et al., It describes the labs environment, DSLabs , developed at the University of Washington to accompany a course in distributed systems. A visual debugger/system explorer. MORE
Machine learning systems are stuck in a rut Barham & Isard, HotOS’19. In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability. MORE
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The Andrew File System. I am bringing with me a paper with one of first distributed systems that had actually see wide-spread commercial deployment. All Things Distributed. MORE
Towards federated learning at scale: system design Bonawitz et al., This is a high level paper describing Google’s production system for federated learning. The FL system overall comprises a set of devices (e.g., SysML 2019. MORE
External Payload Storage External payload storage was implemented to prevent the usage of Conductor as a data persistence system and to reduce the pressure on its backend datastore. The workflow status listener provides hooks to connect to any notification system of your choice. MORE
The reason is in How Complex Systems Fail. How Complex Systems Fail The Westworld season finale made an interesting claim: humans are so simple and predictable they can be encoded by a 10,247-line algorithm. Small enough to fit in the pages of a thin virtual book. MORE
This is where Alluxio comes in and interfaces with applications like Hive as a distributed virtual file system to create tables with multiple partitionings in a different storage system. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system. MORE
Redis Cluster is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. At ScaleGrid, we recently added support for Redis Clusters on our platform through our fully managed Redis hosting plans. MORE
In this three-part blog series, we’ll share the following three common problem scenarios that you can easily solve by building an automated self-healing system with Dynatrace and xMatters Flow Designer: Process crash. Depending on the type of Dynatrace issue, xMatters prompts on-call resources with response option buttons that launch workflows across your systems to start the automated self-healing process—and to keep stakeholders and customers updated. Dynatrace news. MORE
This analysis powers our services and enables the delivery of more seamless and reliable user … The post Scaling Uber’s Apache Hadoop Distributed File System for Growth appeared first on Uber Engineering Blog. Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. MORE
BPF (eBPF) tracing is a superpower that can analyze everything, and I'll show you how in my upcoming book BPF Performance Tools: Linux System and Application Observability , coming soon from Addison Wesley. A time where you can pose arbitrary questions of the system, and it can answer them. MORE
An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems Gan et al., In this paper we explore the implications microservices have across the cloud system stack. Operating system and network implications. MORE
The latest adaptation by the SQL Server team is moving the system tables (metadata) for tempdb to In-Memory OLTP (aka memory-optimized). SQL Performance System Configuration tempdb Hekaton in-memory OLTP SQL Server 2019 MORE
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Virtualizing Operating Systems. This weekends back-to-basics reading is on operating system virtualization. There are two papers that deserve the "classic" tag as they both form the basis for operating system virtualization that is in production today. All Things Distributed. By Werner Vogels on 20 July 2012 12:00 PM. Permalink. Comments (). MORE
Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. Raw content data along with templates are version controlled using Git or similar versioning systems. MORE
Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. MezzFS?—?Mounting MORE
The interfaces to our digital system have been dictated by the capabilities of our computer systems—keyboards, mice, graphical interfaces, remotes, and touch screens. As a result, they fail to deliver a truly seamless and customer-centric experience that integrates our digital systems into our analog lives. All of these benefits make voice a game changer for interacting with all kinds of digital systems. MORE
Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 " Hints for computer system design ". MORE
[T]he aim of such systems is ultimately to produce themselves: their own organization and identity is their most important product. -- Gareth Morgan, Images of Organization , p. This is in contrast to allopoietic systems, which use components (raw materials such as silicon and plastic) to generate something (mobile phones and computers) which are distinct from the thing that created it (the factory where they are made). The system thus organizes its environment as part of itself. MORE
As we scaled up to our present level of support for 14 million trips per day, the car in that … The post Migrating Functionality Between Large-scale Production Systems Seamlessly appeared first on Uber Engineering Blog. MORE
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Hints for Computer Systems Design. I find that going back to the basics of system, network and language design forces a good appreciation for keeping designs simple and focus on those fundamentals that matter most to users. Last weeks paper was the classic End-To-End Arguments in System Design , by J. All Things Distributed. MORE
performance mysql memory operating system bug memory management memory allocatorsWhen users experience memory usage issues with any software, including MySQL, their first response is to think that it’s a symptom of a memory leak. As this story will show, this is not always the case. This story is about a bug. MORE
An Overview of System Testing and End-to-end testing: End-to-end testing and System testing always go hand in hand, but even an experienced test professional can get confused about the vast. Read more System Testing Vs End-To-End Testing: Which One is Better to Opt? The post System Testing Vs End-To-End Testing: Which One is Better to Opt? MORE
I remember the first time I deployed a system into production. Once the system was deployed, I wanted to see if everything was working properly, so I ran through a simple checklist: Is my database up? Yes/No) If the answers to these questions were all yes, then the system was working correctly. If the answer to any of those questions was no, then the system wasn't working correctly and I needed to take action to correct it. MORE
Systems Related Topics Dynatrace
APRIL 24, 2019
For small deployments, it isn’t a problem however when scaling up to hundred or even thousands of systems things can become complicated. Even when all the systems are mapped correctly by Dynatrace, identifying these systems is a real challenge. S _ for the system.
Dynatrace
JULY 25, 2019
In this three-part blog series, we’ll share the following three common problem scenarios that you can easily solve by building an automated self-healing system with Dynatrace and xMatters Flow Designer: Process crash. Depending on the type of Dynatrace issue, xMatters prompts on-call resources with response option buttons that launch workflows across your systems to start the automated self-healing process—and to keep stakeholders and customers updated. Dynatrace news.
Brendan Gregg
JULY 15, 2019
BPF (eBPF) tracing is a superpower that can analyze everything, and I'll show you how in my upcoming book BPF Performance Tools: Linux System and Application Observability , coming soon from Addison Wesley. A time where you can pose arbitrary questions of the system, and it can answer them.
DZone
MAY 7, 2019
performance mysql memory operating system bug memory management memory allocatorsWhen users experience memory usage issues with any software, including MySQL, their first response is to think that it’s a symptom of a memory leak. As this story will show, this is not always the case. This story is about a bug.
The Morning Paper
JUNE 27, 2019
Machine learning systems are stuck in a rut Barham & Isard, HotOS’19. In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability.
The Morning Paper
JUNE 6, 2019
Towards federated learning at scale: system design Bonawitz et al., This is a high level paper describing Google’s production system for federated learning. The FL system overall comprises a set of devices (e.g., SysML 2019.
The Morning Paper
APRIL 16, 2019
Teaching rigorous distributed systems with efficient model checking Michael et al., It describes the labs environment, DSLabs , developed at the University of Washington to accompany a course in distributed systems. A visual debugger/system explorer.
The Morning Paper
MAY 12, 2019
An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems Gan et al., In this paper we explore the implications microservices have across the cloud system stack. Operating system and network implications.
High Scalability
JULY 2, 2018
The reason is in How Complex Systems Fail. How Complex Systems Fail The Westworld season finale made an interesting claim: humans are so simple and predictable they can be encoded by a 10,247-line algorithm. Small enough to fit in the pages of a thin virtual book.
Professor Beekums
MARCH 5, 2017
Gitlab had a very public outage last month. Most companies provide some kind of explanation when their services are interrupted. Those are usually sanitized (or seem sanitized) to make things seem better than they actually are.
DZone
MAY 22, 2019
This is where Alluxio comes in and interfaces with applications like Hive as a distributed virtual file system to create tables with multiple partitionings in a different storage system. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system.
ACM Sigarch
MARCH 18, 2019
Computer systems, from the Internet-of-Things devices to datacenters, are complex and optimizing them can enhance capability and save money. Developing simulators, however, is time-consuming and requires a great deal of infrastructure development regarding a prospective system.
Brendan Gregg
OCTOBER 28, 2017
In the past I've shared similar methodologies applied to other operating systems, and finished porting them to BSD for this talk. For my first trip to Paris I gave the closing keynote at [EuroBSDcon 2017] on performance methodologies, using FreeBSD 11.1 as an analysis target.
Uber Engineering
AUGUST 7, 2019
As we scaled up to our present level of support for 14 million trips per day, the car in that … The post Migrating Functionality Between Large-scale Production Systems Seamlessly appeared first on Uber Engineering Blog.
The Morning Paper
JUNE 9, 2019
PyTorch-BigGraph: a large-scale graph embedding system Lerer et al., SysML’19. We looked at graph neural networks earlier this year, which operate directly over a graph structure.
Uber Engineering
APRIL 5, 2018
This analysis powers our services and enables the delivery of more seamless and reliable user … The post Scaling Uber’s Apache Hadoop Distributed File System for Growth appeared first on Uber Engineering Blog. Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.
SQL Shack
AUGUST 12, 2019
Performance monitoring is a must to do the task for a DBA. You should ensure that the database performance is optimal all the time without any impact on the databases. Performance issues act like an open stage, and you need to look at every aspect such as CPU, RAM, server performance, database performance, indexes, blocking, […]. Deadlocks
ACM Sigarch
MARCH 25, 2019
The M/M/1 queue also assumes that the arrival rate is not affected by the unbounded number of tasks in the queue (called an “open system”). With two blog posts, we argue for more use of simple models beyond Amdahl’s Law.
ACM Sigarch
OCTOBER 4, 2018
This is the first post in a series of posts on different approaches to systems security especially as they apply to hardware and architectural security. In this post, we will consider the use of mathematics/cryptography as an approach to improving systems security.
All Things Distributed
MARCH 12, 2018
The interfaces to our digital system have been dictated by the capabilities of our computer systems—keyboards, mice, graphical interfaces, remotes, and touch screens. As a result, they fail to deliver a truly seamless and customer-centric experience that integrates our digital systems into our analog lives. All of these benefits make voice a game changer for interacting with all kinds of digital systems.
Scalegrid
MARCH 4, 2019
Get the latest insights on MySQL , MongoDB , PostgreSQL , Redis , and many others to see which database management systems are most favored this year. Based on our findings, SQL still holds 60% with rising demand for systems such as PostgreSQL: SQL Database Use: 60.48%.
Adrian Cockcroft
APRIL 18, 2018
In reality, in any non-trivial installation, there are multiple tools collecting, storing and displaying overlapping sets of metrics from many types of systems and different levels of abstraction. What if your monitoring systems fail? “Quis custodiet ipsos custodes?”?—?Juvenal
All Things Distributed
JULY 4, 2019
This week, the developers of Amazon Aurora have won the 2019 Association for Computing Machinery's (ACM) Special Interest Group on Management of Data (SIGMOD) Systems Award.
Particular Software
NOVEMBER 7, 2018
At the height of the Cold War, the United States passed the Federal Aid Highway Act of 1956, giving birth to the Interstate Highway System. They can be observed in our software systems as well. What if the system was built to allow junior developers to actively participate?
The Morning Paper
APRIL 4, 2019
Ginseng: keeping secrets in registers when you distrust the operating system Yun & Zhong et al., Suppose you did go to the extreme length of establishing an unconditional root of trust for your system, even then, unless every subsequent piece of code you load is also fully trusted (e.g.,
High Scalability
FEBRUARY 19, 2019
Redis Cluster is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. At ScaleGrid, we recently added support for Redis Clusters on our platform through our fully managed Redis hosting plans.
Particular Software
JULY 18, 2017
I remember the first time I deployed a system into production. Once the system was deployed, I wanted to see if everything was working properly, so I ran through a simple checklist: Is my database up? Yes/No) If the answers to these questions were all yes, then the system was working correctly. If the answer to any of those questions was no, then the system wasn't working correctly and I needed to take action to correct it.
The Netflix TechBlog
JULY 30, 2019
External Payload Storage External payload storage was implemented to prevent the usage of Conductor as a data persistence system and to reduce the pressure on its backend datastore. The workflow status listener provides hooks to connect to any notification system of your choice.
Strategic Tech
JUNE 23, 2018
There is a high cost associated with work that leaves your team… team boundaries and software boundaries should be isomorphic” — James Lewis, Thoughtworks I’ve written and spoken a lot about architecting sociotechnical systems and how to find boundaries.
All Things Distributed
DECEMBER 23, 2013
Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 " Hints for computer system design ".
The Agile Manager
NOVEMBER 30, 2015
[T]he aim of such systems is ultimately to produce themselves: their own organization and identity is their most important product. -- Gareth Morgan, Images of Organization , p. This is in contrast to allopoietic systems, which use components (raw materials such as silicon and plastic) to generate something (mobile phones and computers) which are distinct from the thing that created it (the factory where they are made). The system thus organizes its environment as part of itself.
Particular Software
APRIL 9, 2018
Based on our experience running business systems in production, we know we need to monitor our theme park to make sure it's working properly. How many CPU cycles is a system using? Infrastructure monitoring tools generally treat systems as "black boxes" that consume resources.
The Netflix TechBlog
MARCH 6, 2019
Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. MezzFS?—?Mounting
High Scalability
APRIL 16, 2019
Thus, whenever a master MySQL goes down (whether due to a MySQL crash, OS crash, system reboot, etc.), This ensures that the system continues to be available to the applications.
The Morning Paper
MAY 21, 2019
” This paper describes a “far memory” system that has been in production deployment at Google since 2016. The objective is to find the lowest cold age threshold that still allows the system to satisfy its performance constraints.
High Scalability
APRIL 3, 2019
PostgreSQL is an open source object-relational database system that has soared in popularity over the past 30 years from its active, loyal, and growing community. For the 2nd year in a row, PostgreSQL has kept the title of #1 fastest growing database in the world according to the DBMS of the Year report by the experts at DB-Engines. So what makes PostgreSQL so special, and how is it being used today?
All Things Distributed
MARCH 30, 2017
One topic that always gets me excited is how to take computer science research and implement it in production systems. real systems do not fail by stopping in a nice and clean way). This weekend I am travelling to Australia for the first AWS Summit of 2017.
All Things Distributed
JULY 20, 2012
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Virtualizing Operating Systems. This weekends back-to-basics reading is on operating system virtualization. There are two papers that deserve the "classic" tag as they both form the basis for operating system virtualization that is in production today. All Things Distributed. By Werner Vogels on 20 July 2012 12:00 PM. Permalink. Comments ().
Abhishek Tiwari
NOVEMBER 3, 2018
Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. Raw content data along with templates are version controlled using Git or similar versioning systems.
All Things Distributed
NOVEMBER 17, 2012
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - The Andrew File System. I am bringing with me a paper with one of first distributed systems that had actually see wide-spread commercial deployment. All Things Distributed.
Software Testing Help
MARCH 11, 2018
An Overview of System Testing and End-to-end testing: End-to-end testing and System testing always go hand in hand, but even an experienced test professional can get confused about the vast. Read more System Testing Vs End-To-End Testing: Which One is Better to Opt? The post System Testing Vs End-To-End Testing: Which One is Better to Opt?
All Things Distributed
JULY 13, 2012
Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Hints for Computer Systems Design. I find that going back to the basics of system, network and language design forces a good appreciation for keeping designs simple and focus on those fundamentals that matter most to users. Last weeks paper was the classic End-To-End Arguments in System Design , by J. All Things Distributed.
SQL Performance
AUGUST 2, 2019
The latest adaptation by the SQL Server team is moving the system tables (metadata) for tempdb to In-Memory OLTP (aka memory-optimized). SQL Performance System Configuration tempdb Hekaton in-memory OLTP SQL Server 2019
82 
Let's personalize your content