In this lab we will demonstrate some basic Spark parallelism concepts with CDE. In our first application we will analyze 1b banking transactions (153gb) and perform two simple transformations: select ...

Spark Properties and Spark Application Architecture Let's take a look together at the Spark optimization methods we've used in our projects. First up is setting Spark properties. Before diving into ...

This report focuses on how to tune a Spark application to run on a cluster of instances. We define the concepts for the cluster/Spark parameters, and explain how to configure them given a specific set ...

Abstract: Configuration tuning is vital to optimize the performance of big data analysis platforms like Spark. Existing methods (e.g. auto-tuning relational databases) are not effective for tuning ...

Hadoop and MapReduce, the parallel programming paradigm and API originally behind Hadoop, used to be synonymous. Nowadays when we talk about Hadoop, we mostly talk about an ecosystem of tools built ...

Abstract: Tuning configurations of Spark jobs is not a trivial task. State-of-the-art auto-tuning systems are based on iteratively running workloads with different configurations. During the ...