Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am trying to remove keys with nan values from a dictionary formed from pandas using python. Is there a way I can achieve this. Here is a sample of my dictionary:

{'id': 1, 'internal_id': '1904', 'first_scraping_time': '2020-04-17 12:44:59.0', 'first_scraping_date': '2020-04-17', 'last_scraping_time': '2020-06-20 03:08:47.0', 'last_scraping_date': '2020-06-20', 'is_active': 1,'flags': nan, 'phone': nan,'size': 60.0, 'available': '20-06-2020', 'timeframe': nan, 'teaser': nan, 'remarks': nan, 'rent': 4984.0, 'rooms': '3', 'downpayment': nan, 'deposit': '14952', 'expenses': 600.0, 'expenses_tv': nan, 'expenses_improvements': nan, 'expenses_misc': nan, 'prepaid_rent': '4984', 'pets': nan, 'furnished': nan, 'residence_duty': nan, 'precision': nan, 'nearby_cities': nan,'type_dwelling': nan, 'type_tenants': nan, 'task_id': '614b8fc2-409c-403a-9650-05939e8a89c7'}
Thank you!
                {k : v for k,v  in d.items() if v is not nan} ? where d is your dictionary then re-assign to a new variable. set nan to np.nan i.e nan = np.nan
– Umar.H
                Jun 23, 2020 at 12:31
                @L3viathan if its coming from a pandas function won't it be equal to numpy.nan rather than math.isnan regardless very helpful post thank you
– Umar.H
                Jun 23, 2020 at 12:37
nan is a tricky object to work with because it doesn't equal (or even necessarily share object identity) with anything, including itself.
You can use math.isnan to test for it:
import math
new = {key: value for (key, value) in old.items() if not math.isnan(value)}
                pd.isna can be used instead of math.isnan, posting this comment 'cause pandas is tagged. pd.isna(math.nan)-> True, pd.isna(np.nan)->True. +1
– Ch3steR
                Jun 23, 2020 at 12:40
                I noticed that both pandas and numpy have such functions. Is there any benefit, apart from being able to apply them to arrays/matrices?
– L3viathan
                Jun 23, 2020 at 12:50
                pd.isna can handle None too, while math.isnan and np.isnan can't. And like you mentioned pd.isna and np.isnan can applied be on arrays.
– Ch3steR
                Jun 23, 2020 at 12:56
                In that case, I think math.isnan is a better answer. nan is not the same as None (which OP may not want to filter out), and it's being used on a single element here.
– L3viathan
                Jun 23, 2020 at 13:50
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.