YARN: Exec Modes

How to execute tasks?

• Local mode

  • Job is submitted on the (master) node of the Resource Manager

• Remote

  • Hadoop is installed on a remote machine

  • Hadoop client jars installed on the local computer

  • Hadoop configuration files define the remote Hadoop cluster for the local Hadoop client

  • Does not work for few cases

• SCP + remote

  • SCP + SSH + run locally

WRITING CUSTOM APPLICATIONS

Grand total of 3 protocols:

• ApplicationClientProtocol

  • Application launching program

  • submitApplication

• ApplicationMasterProtocol

  • Protocol between AM & RM for resource allocation

  • registerApplicationMaster / allocate / finishApplicationMaster

• ContainerManagementProtocol

  • Protocol between AM & NM for container start/stop

  • startContainers / stopContainers