Stream Processing and Analytics

What Is Stream Processing? Examples of Stream Processing- Scaling Up Data Processing- Distributed Stream Processing- Introducing Apache Spark. Stream-Processing Model: Sources and Sinks- Immutable Streams Defined from One Another- Transformations and Aggregations- Window Aggregations – Stateless and Stateful Processing- The Effect of Time. Practical Component: a. Installing and configuring Apache Spark b. Installing and configuring the Scala IDE c. Installing and configuring JDK

Components of a Data Platform- Architectural Models- The Use of a Batch-Processing Component in a Streaming Application- Referential Streaming Architectures- Streaming Versus Batch Algorithms. Apache Spark as a Stream-Processing Engine: Spark’s Memory Usage- Understanding Latency- Throughput- Oriented Processing- Fast Implementation of Data Analysis. Practical Component: a. Write your own Spark Streaming program, to count the number of words in text data received from data server listening on a TCP socket b. Write a simple Spark Streaming program that prints a sample of the tweets it receives from Twitter every second.

Running Apache Spark with a Cluster Manager- Spark’s Own Cluster Manager – Resilience and Fault Tolerance in a Distributed System- Data Delivery Semantics- Microbatching and One-Element-at-a-Time – Bringing Microbatch and One-Record-at a- Time Closer Together- Dynamic Batch Interval- Structured Streaming Processing Model. Spark’s Resilience Model: Resilient Distributed Datasets in Spark – Spark Components – Spark’s Fault- Tolerance Guarantees. Practical Component: a. Create Spark RDD using parallelize with sparkContext.parallelize() method and using Spark shell b. Write a scripts in Spark to Read all text files from a directory into a single RDD c. Write a spark program to load a CSV file into Spark RDD using a Scala d. Write a Spark Streaming program for adding 1 to the stream of integers in a reliable, fault tolerant manner, and then visualize them.

Introducing Structured Streaming- The Structured Streaming Programming Model – Structured Streaming in Action – Structured Streaming Sources – Structured Streaming Sinks – Event Time– Based Stream Processing. Practical Component: a. Develop a streaming application by- Connecting to a Stream, Preparing the Data in the Stream, Performing Operations on Streaming Dataset, creating a Query, Starting the Stream Processing and Exploring the data. b. Create a Structured streaming job by Initializing Spark, acquiring streaming data from sources, declaring the operations we want to apply to the streaming data and outputting the resulting data using Sinks. c. Create a small but complete Internet of Things (IoT)-inspired streaming program. d. Define the schema in Structured Streaming to handle the data at different levels. e. Create custom sinks to write data to systems not supported by the default implementations

Introducing Spark Streaming – The Spark Streaming Programming Model – The Spark Streaming Execution Model – Spark Streaming Sources – Spark Streaming Sinks – Time-Based Stream Processing- Working with Spark SQL – Checkpointing – Monitoring Spark Streaming- Performance Tuning. Practical Component: (i) Develop any Spark Streaming application and do the following: a) Create a Spark Streaming Context, b) Define one or several Dstreams from data sources or other Dstreams c) Define one or more output operations to materialize the results of these