Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am new to PySpark and I have an AskReddit json file which I got from this link . I am trying to create an RDD which I then hope to perform operation such as map and flatmap. I was advised to get the json in a jsonlines format but despite using pip to install jsonlines, I am unable to import the package in the PySpark notebook. Below is what I have tried for reading in the json.

In [10]: import json
         data = json.load(open("AskReddit.json", "r"))
         jsonrdd = sc.parallelize(data)
         jsonrdd.take(5) 
Out[11]: [u'kind', u'data']
I also tried to do the following which gives me the whole contents of the json file after doing jsonrdd.take(1).
In [6]: jsonrdd = sc.wholeTextFiles("*.json")
        jsonrdd.take(1)
However, I would like to get each json object as one line in the rdd. How would I go about this?
You can use SparkSQL's read.json to read the file like - 
jdf = spark.read.json("path/to/AskReddit.json")
and perform all kind of sql type of operation, even rdd type of operations on it.But json struture is really really nested with no fixed columns that can be derived with something like explode
You are better off with using read.json and using dataframe like- 

.withColumn('after',jdf.data.after)\
.withColumn('before',jdf.data.before)\
.withColumn('modhash',jdf.data.modhash)\
.withColumn('NestedKind',explode(jdf.data.children.kind))\
.withColumn('subreddit',explode(jdf.data.children.data.subreddit))\
.withColumn('clicked',explode(jdf.data.children.data.clicked))\
.show()
+--------------------+-------+---------+------+--------------------+----------+---------+-------+
|                data|   kind|    after|before|             modhash|NestedKind|subreddit|clicked|
+--------------------+-------+---------+------+--------------------+----------+---------+-------+
|[t3_66qv3r,null,W...|Listing|t3_66qv3r|  null|3r7ao0m7qiadae13d...|        t3|AskReddit|  false|
|[t3_66qv3r,null,W...|Listing|t3_66qv3r|  null|3r7ao0m7qiadae13d...|        t3|AskReddit|  false|
Use SparkContext library and read file as text and then map it to json.
from pyspark import SparkContext
sc = SparkContext("local","task").getOrCreate()
rddfile = sc.textFile(filename).map(lambda x: json.loads(x))
Assuming you are using spark 2.0+ you can do the following:
df = spark.read.json(filename).rdd
Check out the documentation for pyspark.sql.DataFrameReader.json for more details. Note this method expects a JSON lines format or a new-lines delimited JSON as I believe you mention you have.
                Actually I need to get it in JSON lines format. I used pip install jsonlines. However, when I try to import the package in the jupyter notebook running spark it says no module found even though I can import when running python in the terminal.
– dreamin
                Apr 22, 2017 at 1:13
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.