DZone

SEPTEMBER 14, 2023

Apache Spark , a robust open-source data processing framework, has emerged as a game-changer in this domain. In today's data-driven world, efficient data processing plays a pivotal role in the success of any project.

Big Data 269

Big Data Processing Games Open Source 269

DZone

APRIL 4, 2023

Apache Kafka is a battle-tested distributed stream-processing platform popular in the financial industry to handle mission-critical transactional workloads. Kafka’s ability to handle large volumes of real-time market data makes it a core infrastructure component for trading, risk management, and fraud detection.

Processing 262

Processing Database Infrastructure Testing 262

DZone

AUGUST 4, 2022

Spark-Radiant is Apache Spark Performance and Cost Optimizer. Spark-Radiant will help optimize performance and cost considering catalyst optimizer rules, enhance auto-scaling in Spark, collect important metrics related to a Spark job, Bloom filter index in Spark, etc. Spark-Radiant is now available and ready to use.

Performance 246

Performance Metrics Availability Big Data 246

DZone

JANUARY 25, 2022

Apache Camel is an integration framework, which basically means that it is a set of tools that will help you connect one system to another. A basic example could be listening to an AWS SQS queue and saving a file in Google Cloud Storage Bucket for each message that has been received.

AWS 185

AWS Storage Google Design 185

DZone

JULY 17, 2023

Apache Spark is a leading platform in the field of big data processing, known for its speed, versatility, and ease of use. This article delves into various techniques that can be employed to optimize your Apache Spark jobs for maximum performance. However, getting the most out of Spark often involves fine-tuning and optimization.

Big Data 130

Big Data Performance Open Source Tuning 130

Dynatrace

OCTOBER 18, 2022

The vulnerability affects the Apache Commons Text library. But current analysis indicates that the Apache Commons Text vulnerability is not as widespread and much harder to exploit. of the library, and it has already been patched in Apache Common Text version 1.10.0. The vulnerability is resolved in Apache Commons Text 1.10.0,

Java 217

Java Open Source Virtualization Code 217

DZone

JANUARY 20, 2023

As a bird’s eye view, Apache ZooKeeper has been leveraged to get coordination services for managing distributed applications. To consider as an example, Apache Kafka uses ZooKeeper for choosing their leader node for the topic partitions.

Servers 152

Servers Systems 152

DZone

DECEMBER 27, 2023

Welcome to the first post in our exciting series on mastering offline data pipeline's best practices, focusing on the potent combination of Apache Airflow and data processing engines like Hive and Spark. In this kickoff post, we delve into the intricacies of Apache Airflow and AWS EMR, a managed cluster platform for big data processing.

Best Practices 130

Best Practices Data Engineering Big Data Games 130

DZone

DECEMBER 21, 2020

Today, the Apache Flink project developed by the Apache Software Foundation, one of the leaders in the Big Data software market, was chosen as a test subject for the analyzer. So, what is Apache Flink? Today, Apache Flink can be used in projects written using Java, Scala, Python, and even SQL.

Code 150

Code Java Big Data Open Source 150

Uber Engineering

JULY 22, 2021

As Uber’s business grew, we scaled our Apache Hadoop (referred to as ‘Hadoop’ in this article) deployment to 21000+ hosts in 5 years, to support the various analytical and machine learning use cases. We built a team with varied … The post Containerizing Apache Hadoop Infrastructure at Uber appeared first on Uber Engineering Blog.

Infrastructure 145

Infrastructure Analytics Engineering Architecture 145

DZone

DECEMBER 20, 2023

Observability platforms are akin to the immune system. Just like immune cells are everywhere in human bodies, an observability platform patrols every corner of your devices, components, and architectures, identifying any potential threats and proactively mitigating them.

Speed 265

Speed Architecture Systems Processing 265

DZone

OCTOBER 10, 2023

In enterprises, SREs, DevOps, and cloud architects often discuss which platform to choose for observability for faster troubleshooting of issues and understanding about performance of their production systems.

Open Source 262

Open Source DevOps Infrastructure Metrics 262

DZone

FEBRUARY 15, 2024

When it comes to Java web servers, Apache Tomcat remains a strong favorite. Red Hat JBoss Web Server (JWS) combines the servlet engine (Apache Tomcat) with the web server (Apache HTTPD), and modules for load balancing (mod_jk and mod_cluster).

Servers 241

Servers Java Virtualization Engineering 241

DZone

DECEMBER 4, 2023

According to several sources we queried, more than 33 percent of the world's web servers are running Apache Tomcat, while other sources show that it's 48 percent of application servers. Some of these instances have been containerized over the years, but many still run in the traditional setup of a virtual machine with Linux.

Servers 262

Servers Virtualization Engineering 262

Uber Engineering

MARCH 16, 2023

Uber’s Global Data Warehouse team leveraged Apache Hudi to drastically improve performance of traditional batch ETL pipelines by going incremental, improving business-critical data’s freshness, quality, and completeness.

Performance 98

Performance 98

Uber Engineering

JUNE 29, 2023

Experience the power of row-level secondary indexing in Apache Parquet, enabling 3-20X faster upserts and unlocking new possibilities for efficient table ACID operations in today’s Lakehouse architecture.

Architecture 77

Architecture Efficiency 77

DZone

FEBRUARY 21, 2023

In this hands-on lab from ScyllaDB University, you will learn how to use the ScyllaDB CDC source connector to push the row-level changes events in the tables of a ScyllaDB cluster to a Kafka server. What Is ScyllaDB CDC?

Open Source 306

Open Source Servers 306

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Processing 139

Processing Engineering Systems Architecture 139

Uber Engineering

JANUARY 15, 2020

As Uber’s operations became more complex and we offered additional features and … The post Engineering SQL Support on Apache Pinot at Uber appeared first on Uber Engineering Blog.

Engineering 115

Engineering Analytics Big Data Database 115

DZone

FEBRUARY 1, 2024

I also created a demo around it, which featured the following components: The Apache APISIX API Gateway A Kotlin/Spring Boot service A Python/Flask service And a Rust/Axum service I've recently improved the demo to deepen my understanding and want to share my learning.

259

259

The Netflix TechBlog

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 259

Data Engineering Engineering Software Engineering Best Practices 259

SQL Shack

OCTOBER 7, 2020

In this article, we will learn how to use Apache JMeter to perform a load test on SQL Server. This test type enables us to measure the application behaviors under specific conditions so that it enables us to observe a variety of resource consumptions (CPU, memory, latency, response times, etc) and it also helps to […].

Servers 115

Servers Testing Performance Latency 115

DZone

AUGUST 22, 2023

As a major part of a company's data asset, logs bring value to businesses in three aspects: system observability, cyber security, and data analysis. They are your first resort for troubleshooting, your reference for improving system security, and your data mine, where you can extract information that points to business growth.

Systems 100

Systems 100

DZone

JUNE 16, 2016

In this article I am going to show you two approaches to load balance JAXRS web services without an Apache or a hardware load balancer. Below are the two approaches to load balance JAXRS services using CXF and Apache Camel. The full code is available on GitHub.

Hardware 130

Hardware Strategy Code Availability 130

DZone

MARCH 18, 2024

The data silos problem is like arthritis for online businesses because almost everyone gets it as they grow old. Businesses interact with customers via websites, mobile apps, H5 pages, and end devices. For one reason or another, it is tricky to integrate the data from all these sources.

Mobile 136

Mobile Website 136

InfoQ

OCTOBER 20, 2023

Contentsquare needed notification functionality for many use cases within its platform. The company created a generic solution spanning multiple services as part of its microservice architecture. During the implementation, the developers had to improve observability and overcome some scalability challenges. By Rafal Gancarz

Scalability 104

Scalability Architecture Development Design 104

InfoQ

APRIL 26, 2024

Allegro experimented with different performance optimization options to improve Apache Kafka producer tail latency and eventually switched all its clusters to the XFS filesystem. The company used Kafka protocol sniffing, JVM profiling, and eBPF, which proved instrumental in identifying and eliminating performance bottlenecks.

Latency 109

Latency Performance Tuning Design 109

Uber Engineering

APRIL 5, 2018

This analysis powers our services and enables the delivery of more seamless and reliable user … The post Scaling Uber’s Apache Hadoop Distributed File System for Growth appeared first on Uber Engineering Blog.

Systems 109

Systems Big Data Storage Infrastructure 109

DZone

SEPTEMBER 20, 2021

If you are reading this blog post, most probably you are planning or already planned or just getting started with Apache JMeter. Learning any performance testing tool could be challenging. Sometimes the technical documentation looks abstract, and the learning curve could be steep.

Testing Tools 329

Testing Tools Performance Testing Testing Performance 329

Dynatrace

DECEMBER 16, 2021

Log4Shell is a software vulnerability in Apache Log4j 2 , a popular Java library for logging error messages in applications. Apache issued a patch for CVE-2021-44228, version 2.15, on December 6. What is Log4Shell? Attackers can exploit the vulnerability using text messages to control a computer remotely.

Internet 258

Internet Internet Games Java 258

DZone

OCTOBER 6, 2023

Apache JMeter is an open-source, Java-based tool used for load and performance testing of various services, particularly web applications. It supports multiple protocols like HTTP, HTTPS, FTP, and more. JMeter can simulate heavy loads on a server to analyze performance under different conditions.

Java 255

Java Government Open Source Performance Testing 255

VoltDB

MARCH 23, 2023

The post Top 5 Apache Kafka Use Cases: When to Use Kafka appeared first on Volt Active Data.

52

52

DZone

NOVEMBER 3, 2020

Several words about Apache Kafka. Several Words About Apache Kafka. Several Words About Apache Kafka. Kafka is an open-source stream ing software platform developed by the Apache Software Foundation, written in Scala and Java. How to start Kafka performance testing with JMeter + Pepper-Box plugin ? Pros and cons.

Performance Testing 214

Performance Testing Testing Performance Open Source 214

DZone

AUGUST 31, 2023

In the demo, I show how you can see the traces of a simple distributed system consisting of the Apache APISIX API Gateway, a Kotlin app with Spring Boot, a Python app with Flask, and a Rust app with Axum. One of my current talks focuses on Observability in general and Distributed Tracing in particular, with an OpenTelemetry implementation.

Metrics 239

Metrics Systems 239

DZone

SEPTEMBER 25, 2023

The model is deployed as a REST service using Python Flask and Apache web server, packaged in a docker container. The high-level architecture of the application is depicted in the following picture:

Storage 258

Storage AWS Architecture Cloud 258

DZone

SEPTEMBER 23, 2020

This time I would like to show my experience creating an End2End test for a Camel integration application that connects these layers: Apache Kafka. Apache ActiveMQ Artemis. Apache Camel rest endpoints. (<- the app). PostgreSQL. Localstack ( AWS similar ). Red Hat Ingress upload service. Minio ( AWS similar ).

Testing 269

Testing AWS Servers Engineering 269

Uber Engineering

MARCH 12, 2017

With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala , the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies … The post Hudi: Uber Engineering’s Incremental Processing Framework on Apache Hadoop appeared (..)

Processing 98

Processing Latency Storage Engineering 98