AWS and Dynatrace automation hit the jackpot at Perform Las Vegas

Have you ever had to provision 2,500 virtual host instances and 120 Kubernetes clusters in two days?

Well, that’s exactly what the Dynatrace University team did to support Dynatrace’s hands-on training (HoT) days at Dynatrace’s annual user conference Perform in Las Vegas. The Dynatrace dashboard below shows the thousands of EC2 instances coming up and then being removed at the close of the training. Quite impressive!

Using environment automation from both AWS and Dynatrace and supported by the AWS Infrastructure Event Management program, Dynatrace University successfully delivered the required environments – three times more than the conference the year before!   Let review how they did this, the results and how you too can use automation in your projects.

Dynatrace University

Dynatrace University is the team within Dynatrace that provides certification, self-paced micro-learning courses, and interactive instructor-led training. At Perform 2020, Dynatrace University helped deliver 24 instructor-led HoT sessions across two full days and provided on-site proctoring for those who wanted to obtain their Dynatrace Associate Certification.

Across the two days, there were sixteen new sessions – delivered for the first time – based on content derived from lab exercises developed and delivered by working with Dynatrace experts and our Partner communicate to showcase Dynatrace’s newest features.

For the sessions, each student needed to have their own Dynatrace SaaS tenant to monitor and perform the hands-on exercises. This meant that over 2,000 individual Dynatrace SaaS tenants needed to be set up along with the installation of the Dynatrace OneAgent and the Dynatrace OneAgent Operator for the session applications.

AWS Infrastructure Event Management program

Throughout the planning phase, Dynatrace took part in the AWS Infrastructure Event Management program. During this program, we worked closely with our Technical Account Manager from AWS to co-ordinate AWS resource limit increases and to help identify and mitigate risks that may have occurred during the event. This was helpful and allowed us to have a seamless rollout leading up to the event.

Automation

This year we invested a lot in automation at Perform. In the past, setting up all the hosts, clusters, and demo applications was a manual process that was very time consuming and error-prone. And with three times the amount of infrastructure and growing complexity of environments for year’s Perform compared to last year, manual setup wasn’t an option.

All the infrastructure to run the applications used for the sessions were created using CloudFormation and Lambda. Below are the three main Lambda functions that handled the full life cycle of Perform environments:

1. Create CloudFormation stacks requested by the instructors like automating the OneAgent installation or setting up Route53 records.
2. Export environment details like URLs and token in a format needed for the print shop that created individualized student cards.
3. Remove all Perform infrastructure quickly to save on costs after the event.

During the build-out and testing, one of the instructors developed an automation script using the Dynatrace OneAgent on a host API and shared it on the HoT day chat channel. The team loved the idea and this script got incorporated into the Dynatrace University environment automation scripts. True DevOps culture in action!

To create the Dynatrace SaaS tenants for each student, the Dynatrace’s Autonomous Cloud Enablement (ACE) team used their orchestration scripts to set up all the environments the same way they do in our production SaaS environments. Then using another automated script that calls Dynatrace APIs like the Token API, all the student user accounts and tokens were generated for each student and assigned to the student tenants.  You can see a similar automation process on this GitHub repo.

Monitoring

Throughout this entire process, we used Dynatrace cloud monitoring for AWS to gain insight into our entire environment automation process.

To help everyone supporting HoT day sessions and the certification room, both remotely and onsite, a simple Dynatrace dashboard was set up with quick links to registration, environment guides, and support. Real-time data fed charting for registrations, AWS infrastructure utilization, and network availability fed by AWS CloudWatch metrics.

The results

By leveraging the Dynatrace API, AWS CloudFormation, AWS Lambda, and the AWS Infrastructure Event Management program, we provisioned and terminated over 1,500 CloudFormation stacks, 2,500 EC2 instances, and 120 Kubernetes clusters in two days. During the two days of HoT sessions, we had zero issues come up with the underlying infrastructure at completion.

We’re already planning how we can take this a step further for next year. The nice thing, with all the time invested for Perform, is that everything directly relates to our University instructor-led HoT sessions so we can continue to innovate and provide value across the board without duplicated effort.

Everything we did, whether it was through automation or increasing AWS limits, has been replicated in all of the regions we work in globally. So for every single Dynatrace event, we are ready!

Takeaways you can use for your own automation

Whether you have the exact requirements with Dynatrace or not, you can use the below for planning, provisioning and managing AWS resources in an automated and secure manner at scale.

1. Full-stack AWS resource monitoring for a single view across your entire ecosystem.
2. Dynatrace AWS monitoring with CloudWatch metrics and metadata.
3. Dynatrace API to automate your monitoring tasks and export different types of data.
4. Dynatrace Autonomous Enablement Services to help you create an autonomous cloud strategic vision and get you started through rapid, meaningful automation services engagements.
5. AWS Infrastructure Event Management for large-scale events such as product or application launches, infrastructure migrations, and marketing events.