article thumbnail

What is IT automation?

Dynatrace

Expect to spend time fine-tuning automation scripts as you find the right balance between automated and manual processing. This requires significant data engineering efforts, as well as work to build machine-learning models. By tuning workflows, you can increase their efficiency and effectiveness.

article thumbnail

3. Psyberg: Automated end to end catch up

The Netflix TechBlog

This helps overwrite data only when required and minimizes unnecessary reprocessing. As seen above, by chaining these Psyberg workflows, we could automate the catchup for late-arriving data from hours 2 and 6. The Data Engineer does not need to perform any manual intervention in this case and can thus focus on more important things!

Tuning 244
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” Please share your experience by adding your comments below and stay tuned for more on data lineage at Netflix in the follow up blog posts. .

article thumbnail

Organise your engineering teams around the work by reteaming

Abhishek Tiwari

I also have a strong feeling that long-lived teams are not good for innovation and disruption. Contrarian view What I am proposing here is some key principals to change how you deploy your engineers to do their best work in a fast-paced environment. Secondly, fine-tune team composition based on work.