Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
This question does not appear to be about
a specific programming problem, a software algorithm, or software tools primarily used by programmers
. If you believe the question would be on-topic on
another Stack Exchange site
, you can leave a comment to explain where the question may be able to be answered.
Closed
7 years ago
.
Magpie (
https://github.com/llnl/magpie
) is a package of shell scripts used for running Big Data applications (such as Spark) on top of traditional HPC clusters (such as those managed by SLURM). It manages much of the complexity of queueing jobs for you, but you may still need to modify the submission scripts to suit your own Spark application - see the file
submission-scripts/script-sbatch/magpie.sbatch-spark
for more information.
–
Just use Spark's standalone mode side-by-side with whatever Slurm
does. This means allocating a static subset of resources on the
target machines to each system.
Implement Slurm as a cluster option
for Spark. This is best for true management and sharing of
resources, but much harder because of you have to figure out Spark's
clustering abstractions and implement them correctly. You could
start by looking at how YARN or Mesos support is implemented.
