The Python Oracle

Why does fillna with median on dataframe still leaves Na/NaN in pandas?

Become part of the top 3% of the developers by applying to Toptal https://topt.al/25cXVn

--

Music by Eric Matyas
https://www.soundimage.org
Track title: Cosmic Puzzle

--

Chapters
00:00 Question
02:15 Accepted answer (Score 4)
03:00 Thank you

--

Full question
https://stackoverflow.com/questions/5025...

Question links:
[this]: https://stackoverflow.com/questions/3402...
[this]: http://this
[null count tables, before and after]: https://i.stack.imgur.com/3E5BI.png
[NaN examples]: https://i.stack.imgur.com/2By6v.png

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #pandas #dataframe #series #imputation

#avk47



ACCEPTED ANSWER

Score 5


The problem is with this line:

TT_df = TT_df.fillna(TT_df.median())

Your dataframe has strings and you are attempting to calculate medians on strings. This doesn't work.

Here's a minimal example:

import pandas as pd, numpy as np

df = pd.DataFrame({'A': ['A', 'B', np.nan, 'B']})

df = df.fillna(df.median())

print(df)

     A
0    A
1    B
2  NaN
3    B

What you should do is fillna with median only for numeric columns:

for col in df.select_dtypes(include=np.number):
    df[col] = df[col].fillna(df[col].median())