SPARK ARCHITECTURE:
------------------------------------
There are 5 parts to it
->Driver program
->Cluster Manager
->Worker Node
->Executor
->Task

Driver program:
---------------------
Driver Program in the Apache Spark architecture calls the main program of an application and created SparkSession.
A SparkSession consists of all the basic functionalities.
Spark Driver contains various other components such as DAG Scheduler, Task Scheduler, and Backend Scheduler which are responsible for translating the user-written code into jobs that are actually executed on the cluster.
Job is split into multiple smaller tasks which are further distributed to worker nodes and can also be cached there.
SparkDriver and SparkSession collectively watch over the job execution within the cluster.
SparkDriver works with the Cluster Manager to manage various other jobs.

Cluster Manager:
-----------------------
The role of the cluster manager is to allocate resources across applications. The Spark is capable enough of running on a large number of clusters.
It consists of various types of cluster managers such as Hadoop YARN, Apache Mesos, and Standalone Scheduler

Worker Node:
------------------
The worker node is a slave mode
Its role is to run the application code in the cluster.

Executor:
-------------
An executor is a process launched for an application on a worker node.
It runs tasks and keeps data in memory or disk storage across them.
It read and writes data to the external sources
Every application contains its executor.

Task:
-------
A unit of work that will be sent to one executor.

Reply Hide

Posted in: Azure | ID: Q99 |
May 07, 2022, 10:30 AM | 0 Replies