读取MongoDB数据到Pandas中

我们可以通过pymongo包连接mongodb进行数据处理,然后将数据存储到pandas的dataframe里面。

例子中的student表的数据结构如下:

	{'_id': ObjectId('5c7138f4e0411eb39749fdff'), 'name': 'student1', 'id_no': 1, 'scores': {'math': 63, 'art': 72, 'music': 93}}
	{'_id': ObjectId('5c7138f4e0411eb39749fe00'), 'name': 'student2', 'id_no': 2, 'scores': {'math': 58, 'art': 70, 'music': 67}}
	{'_id': ObjectId('5c7138f4e0411eb39749fe01'), 'name': 'student3', 'id_no': 3, 'scores': {'math': 66, 'art': 80, 'music': 81}}

不含嵌套数据

如果我们读取mongodb的数据不含嵌套数据,我们可以直接将其载入到Pandas的dataframe中:

import pymongo as pm
import pandas as pd
import numpy as np
import datetime as dt
client = pm.MongoClient('mongodb://user1:user1@127.0.0.1:27017')
db = client['my_db']
proection = {'name':1,'id_no':1, }
mongo_data = list(db['students'].find({}, proection))
df = pd.DataFrame(mongo_date)

如果我们读取mongodb的数据含有嵌套数据,我们需要先将所有嵌套的数据通过json_util工具先解析出来,具体代码如下:

import pymongo as pm
import pandas as pd
import numpy as np
import datetime as dt
from bson import json_util
from pandas.io.json import json_normalize
import json
client = pm.MongoClient('mongodb://user1:user1@127.0.0.1:27017')
db = client['my_db']
mongo_data = list(db['students'].find({}))
sanitized = json.loads(json_util.dumps(mongo_data))
normalized = json_normalize(sanitized)
df = pd.DataFrame(normalized)
print(df)

结果如下: