Accelerate your cloud journey with Dynatrace observability for AWS S3 logs

Dynatrace accelerates enterprise observability, troubleshooting, security, and automation use cases that rely on log data from Amazon Web Services S3. AWS S3 is a popular storage location for AWS services and third party solutions, which can now be more easily integrated with Dynatrace Grail™ data lakehouse. This allows you to answer any question at any time using log data while avoiding the complexity of storage management.

Logs complement metrics and enable automation

Cloud practitioners agree that observability, security, and automation go hand in hand. The increasing complexity of cloud service architectures requires a rock-solid understanding of the activity, health status, and security of cloud services. Logs complement out-of-the-box metrics and enable automated actions for responding to availability, security, and other service events.

Many AWS services and third party solutions use AWS S3 for log storage. We hear from our customers how important it is to have a centralized, quick, and powerful access point to analyze these logs; hence we’re making it easier to ingest AWS S3 logs and leverage Dynatrace Log Management and Analytics powered by Grail.

Centralized log management for scalable ingestion into Grail

As AWS S3 proves to be the preferred way of storing cloud logs, enterprise customers face mounting challenges in putting S3 log data to use. Dynatrace Log Management and Analytics powered by Grail enables you to get answers from logs with any query at any time. However, as a first step, logs from S3 need to be ingested into Dynatrace Grail.

To date, some customers have used open source or community-backed components to forward logs from S3 to Dynatrace. A Dynatrace S3 log forwarder has been available for some time to early adopters, with community support only. While these are great examples of innovation and the power of the community, enterprises often require the type of continuous support and maintenance that only comes with official software.

Another painful blocker is the need for more support for multiple S3 accounts and AWS regions. Some enterprise customers use over a thousand accounts for cloud services, which dramatically increases complexity and overhead within production environments. If an organization operates in multiple geographies and AWS regions, they can centralize logs in a regional S3 bucket, as AWS services send log data to S3 buckets in the same region where they run.

Because data context is missing for logs, it’s slow or even impossible to build causal relationships in your observability data. Most importantly, it’s impossible to establish relationships between infrastructure and application events, business impact, and real user events.

Without the ability to connect logs in S3 with Dynatrace, more expensive or cumbersome alternatives are often used, which slow down troubleshooting and have a lasting business impact.

Ingest logs from AWS S3 with one forwarder

Dynatrace now has a solution for forwarding logs from AWS S3 to its industry-leading log analysis platform, providing enterprise-level support. This makes S3 logs a robust and reliable way of collecting logs, forwarding them to Dynatrace, analyzing log data via DQL or in apps, and thus putting log data to use in solving your observability and security use cases.

The AWS S3 log forwarder can ingest any text- or JSON-formatted logs, which unlocks not only log data from AWS services but also from common third-party logs. Service providers such as Akamai, Fastly, Netlify, open source tools like Apache Airflow, and many others have built-in log delivery integration with S3. The Dynatrace AWS S3 log forwarder is designed to be extensible, so you can customize ingestion settings to fit your specific use cases as well as add custom attributes to your logs.

We designed the Dynatrace AWS S3 log forwarder to scale up to meet the demands of our largest customers, some of whom operate thousands of AWS accounts. The model of deploying one forwarder per AWS region and AWS account doesn’t scale well, so the Dynatrace AWS S3 log forwarder has built-in support for log ingestion from multiple AWS Accounts and AWS regions with a single log forwarder deployment.

The S3 log forwarder also keeps the metadata of log messages, so the originating AWS account and region as well as service-specific attributes like resource ID are preserved. This makes it possible to tie log messages back to the apps, infrastructure, and cloud services where they originated, and enables the unified observability of the Dynatrace platform.

As the cloud footprint of a company grows, so does its log data volume. This is why Dynatrace AWS S3 log forwarder throughput is aligned with the ingest volume of Grail data lakehouse to support your growing data needs.

After you ingest logs into Grail, you can put the data to use with exploratory analytics in Notebooks or transform complex data into easy-to-understand visualizations using Dashboards.

Set up log forwarding on AWS S3

Dynatrace Amazon S3 log forwarder is an AWS Lambda function that supports out-of-the-box parsing and forwarding of logs for the following AWS Services:

• AWS Elastic Load Balancing access logs (ALB, NLB, and Classic ELB)
• Amazon CloudFront access logs
• AWS CloudTrail logs
• AWS Global Accelerator flow logs
• Amazon Managed Streaming for Kafka logs
• AWS Network Firewall alert and flow logs
• Amazon Redshift audit logs
• Amazon S3 access logs
• Amazon VPC DNS query logs
• Amazon VPC Flow logs (default logs)
• AWS WAF logs

Additional context for these and other AWS services can be covered thanks to a built-in parsing mechanism. This is achieved either by Dynatrace AWS S3 forwarder or log processing mechanisms in Dynatrace.

The log forwarder sends the data to the generic log ingest API in your Dynatrace SaaS tenant for Grail analysis. This is another use case where S3 log ingestion can be used to address a wide range of use cases.

Get started today

All the information you need to get started is listed below:

• Discover Dynatrace AWS S3 Log forwarder in Github and ingest logs either to Log Management and Analytics (powered by Grail) or Log Monitoring Classic
• Read the documentation for cloud log forwarding and generic ingest API
• Test out the integration with a free trial of Dynatrace SaaS
• Share your feedback about your cloud journey with logs in the Dynatrace Community

Do you work extensively with AWS S3 logs?

If so, stay tuned for more news about direct AWS Kinesis Data Firehose configuration in AWS console. Or explore the recently introduced support for AWS Lambda logs.