Apache Spark

Installation:
Step 1: Install VMware Player
installing VMware Player v5 on the 64-bit Windows 7 computer to be reliable and causes no problems. VMware Player can also be installed on a Linux computer

 

Architecture:

Apache Spark provides four main submodules, which are SQL, MLlib, GraphX, and Streaming The modules are interoperable, so data can be passed between them. For instance, streamed data can be passed to SQL, and a temporary table can be created.

architecture

Spark Streaming:
spark streaming can be used for processing the real-time streaming data. This is based on micro batch style of computing and processing. It uses the DStream which is basically a series of RDDs, to process the real-time data.

Spark SQL:
Spark SQL provides the capability to expose the Spark datasets over JDBC API and allow running the SQL like queries on Spark data using traditional BI and visualization tools. Spark SQL allows the users to ETL their data from different formats it’s currently in (like JSON, Parquet, a Database), transform it, and expose it for ad-hoc querying.

• Spark MLlib:
MLlib is Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives.

• Spark GraphX:
GraphX is the new (alpha) Spark API for graphs and graph-parallel computation. At a high level, GraphX extends the Spark RDD by introducing the Resilient Distributed Property Graph: a directed multi-graph with properties attached to each vertex and edge. To support graph computation, GraphX exposes a set of fundamental operators (e.g., subgraph, join Vertices, and aggregate Messages). In addition, GraphX includes a growing collection of graph algorithms and builders to simplify graph analytics tasks.

DStream:
A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data. DStreams can either be created from live data (such as, data from TCP sockets, Kafka, Flume, etc.) using StreamingContext or it can be generated by transforming existing DStreams using operations such as map, window and reduceByKeyAndWindow. While a Spark Streaming program is running, each DStream periodically generates a RDD, either from live data or by transforming the RDD generated by a parent DStream.
This class contains the basic operations available on all DStreams, such as map, filter and window. In addition, PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKeyAndWindow and join. These operations are automatically available on any DStream of pairs (e.g., DStream [(Int, Int)] through implicit conversions.
DStreams internally is characterized by a few basic properties: – A list of other DStreams that the DStream depends on – A time interval at which the DStream generates an RDD – A function that is used to generate an RDD after each time interval

dstream

 

Resilient Distributed Data:

RDD is the core concept in Spark framework. Think about RDD as a table in a database. It can hold any type of data. Spark stores data in RDD on different partitions.
They help with rearranging the computations and optimizing the data processing.
They are also fault tolerance because an RDD know how to recreate and recompute the datasets.
RDDs are immutable. You can modify an RDD with a transformation but the transformation returns you a new RDD whereas the original RDD remains the same.
RDD supports two types of operations:
• Transformation
• Action
Transformation: It don’t return a single value, they return a new RDD. Nothing gets evaluated when you call a Transformation function, it just takes an RDD and return a new RDD.
Some of the Transformation functions are map, filter, flatMap, groupByKey, reduceByKey, aggregateByKey, pipe, and coalesce.
Action:
Action operation evaluates and returns a new value. When an Action function is called on a RDD object, all the data processing queries are computed at that time and the result value is returned.
Some of the Action operations are reduce, collect, count, first, take, countByKey, and foreach.

Rating: 5.0. From 1 vote.
Please wait...

You may also like...

2 Responses

  1. heart health says:

    Hello my loved one! I want to say that this post is awesome, nice written and include almost all vital infos. I’d like to peer extra posts like this .

    No votes yet.
    Please wait...
  2. ergfirnolikz says:

    I am perpetually thought about this, thanks for putting up.

    No votes yet.
    Please wait...

Leave a Reply

Your email address will not be published. Required fields are marked *