Building Data Pipelines in Scala

Building Data Pipelines in Scala

47 Degrees, in partnership with Trainologic, will be hosting an in-depth, three-day training class called Building Data Pipelines in Scala at The Pioneer Collective in Seattle, Washington.

In this course, students will learn how to build robust, performant, and highly-available data pipelines using Scala and related technologies.

During the course, we will discuss many technical aspects regarding design & implementation considerations around the suggested technology stack. The course will cover Kafka, Flink, Spark, and Akka. The course will focus on the streaming aspect of these technologies.

The course includes hands-on exercises which are crucial to the understanding of the topics.

Audience

  • Scala Developers

Prerequisites

Students taking this course should have:

  • Experience with Scala

Syllabus

Introduction to Streaming (2 hours)

  • Streaming Design Considerations
  • Challenges in Streaming Applications
  • Use Cases
  • Streaming in Microservices Architecture

Akka Streaming (3 hours)

  • Introduction to Akka
  • The Actor Model
  • Akka Streaming

Spark (6 hours)

  • Introduction to Spark
  • Resilient-Distributed-Datasets (RDD)
  • Overview on Datasets
  • Spark Streaming (DStream)
  • Structured Streaming
  • Introduction to Flink
  • DataSet and DataStream
  • Transformations

Apache Kafka

  • Introduction to Apache Kafka
  • Kafka as a Spark DataSource
  • Kafka as a Flink Source
  • Kafka as a Sink

Real-world Streaming (5 hours)

  • Event time (timestamps) and Window Aggregations
  • Checkpoints and Robustness
  • Monitoring and Logging
  • Handling State
  • Scheduling
  • Serialization Considerations
  • Buffering
  • Memory & GC
  • Performance Best Practices

Duration: 3 days

About the Instructor

Shimi Bandiel is an expert in Scala, Big Data and JVM technologies. With more than 18 years of development, consulting & training, Shimi has accompanied many development teams during their development phases and help them overcome technical challenges. Shimi is an experienced trainer who delivered more than 400 courses for the last 13 years varying from beginner to expert levels with great feedbacks.

In recent years, Shimi has focused on Scala and Big Data frameworks (Spark & Flink) and helped several organizations develop and tune their solutions.

Want to train your entire team? Contact us to see about a group discount or customized training for your organization.

blog comments powered by Disqus

Ensure the success of your project

47 Degrees can work with you to help manage the risks of technology evolution, develop a team of top-tier engaged developers, improve productivity, lower maintenance cost, increase hardware utilization, and improve product quality; all while using the best technologies.