# Spark

* Spark VS map Reduce (in memory computation).
* what is rdd, what it consist of?
* actions and transformations
* wide and narrow transformations
* what is shuffle and why does it bad?
* Deploy modes. (Local, Client and Cluster)
* run modes (local\[1] vs local\[\*] vs remote \[with yarn only or not?])
* application workflow (jobs, stages, tasks)
* why using collect is bad practice?
* reduceByKey vs groupByKey vs combineByKey (rdd)
* Coalesce VS Repartition
* RDD VS Dataframe VS Dataset
* Catalyst, CBO, Tungsten
* Spark optimization (spark.serializer, spark.driver.maxResultSize, spark.sql.shuffle.partitions Vs spark.default.parallelism)
* Cache vs persist method in Spark
* window operations. What is it in spark.
* metastore
* Speculation execution
* DStream. Nature and anatomy.
* Window calculations in Streaming.
* Checkpointing in Spark Streaming. WAL.
* Spark Streaming Recievers
* Spark Streaming application workflow
* foreach partition VS foreach RDD
* Monitoring Streaming Queries (Reporting Metrics, Dropwizard )
* Spark Structured streaming: basic concept; Watermarking
* Spark Structured streaming Output modes. (Append; Complete; Update) 
* Spark Kafka integration