Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I am using the Spark's
collectAsMap
function [
Spark CollectAsMap
to obtain a Map. In this map, when I do the
put
operation I am getting the following exception:
ERROR ApplicationMaster: User class threw exception: java.lang.UnsupportedOperationException
java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209)
Is the map obtained from collectAsMap unmodifiable?
I assume you are working in Java. You are getting this error because in java spark, collectAsMap returns a java wrapper around a scala map. In Spark 2.2, this wrapper is a custom class defined in this source file. As you can see, it does not define the put method, hence your error.
A workaround could be to simply copy the map into a java HashMap as follows
List<Tuple2<Integer, Integer>> list = new ArrayList<>();
list.add(new Tuple2<>(1,2));
list.add(new Tuple2<>(3,4));
Map<Integer, Integer> map = sc.parallelize(list)
.mapToPair( x -> x )
.collectAsMap();
Map<Integer, Integer> newMap = new HashMap<>(map);
newMap.put(7, 8);
System.out.println(newMap);
That yields the expected {1=2, 3=4, 7=8}.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
