> For the complete documentation index, see [llms.txt](https://bigdata-2.gitbook.io/bd201notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://bigdata-2.gitbook.io/bd201notes/topics-for-writiing-task/spark.md).

# Spark

* Spark VS map Reduce (in memory computation).
* what is rdd, what it consist of?
* actions and transformations
* wide and narrow transformations
* what is shuffle and why does it bad?
* Deploy modes. (Local, Client and Cluster)
* run modes (local\[1] vs local\[\*] vs remote \[with yarn only or not?])
* application workflow (jobs, stages, tasks)
* why using collect is bad practice?
* reduceByKey vs groupByKey vs combineByKey (rdd)
* Coalesce VS Repartition
* RDD VS Dataframe VS Dataset
* Catalyst, CBO, Tungsten
* Spark optimization (spark.serializer, spark.driver.maxResultSize, spark.sql.shuffle.partitions Vs spark.default.parallelism)
* Cache vs persist method in Spark
* window operations. What is it in spark.
* metastore
* Speculation execution
* DStream. Nature and anatomy.
* Window calculations in Streaming.
* Checkpointing in Spark Streaming. WAL.
* Spark Streaming Recievers
* Spark Streaming application workflow
* foreach partition VS foreach RDD
* Monitoring Streaming Queries (Reporting Metrics, Dropwizard )
* Spark Structured streaming: basic concept; Watermarking
* Spark Structured streaming Output modes. (Append; Complete; Update) 
* Spark Kafka integration