Why does fillna with median on dataframe still leaves Na/NaN in pandas?
Become part of the top 3% of the developers by applying to Toptal https://topt.al/25cXVn
--
Music by Eric Matyas
https://www.soundimage.org
Track title: Cosmic Puzzle
--
Chapters
00:00 Question
02:15 Accepted answer (Score 4)
03:00 Thank you
--
Full question
https://stackoverflow.com/questions/5025...
Question links:
[this]: https://stackoverflow.com/questions/3402...
[this]: http://this
[null count tables, before and after]: https://i.stack.imgur.com/3E5BI.png
[NaN examples]: https://i.stack.imgur.com/2By6v.png
--
Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...
--
Tags
#python #pandas #dataframe #series #imputation
#avk47
--
Music by Eric Matyas
https://www.soundimage.org
Track title: Cosmic Puzzle
--
Chapters
00:00 Question
02:15 Accepted answer (Score 4)
03:00 Thank you
--
Full question
https://stackoverflow.com/questions/5025...
Question links:
[this]: https://stackoverflow.com/questions/3402...
[this]: http://this
[null count tables, before and after]: https://i.stack.imgur.com/3E5BI.png
[NaN examples]: https://i.stack.imgur.com/2By6v.png
--
Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...
--
Tags
#python #pandas #dataframe #series #imputation
#avk47
ACCEPTED ANSWER
Score 5
The problem is with this line:
TT_df = TT_df.fillna(TT_df.median())
Your dataframe has strings and you are attempting to calculate medians on strings. This doesn't work.
Here's a minimal example:
import pandas as pd, numpy as np
df = pd.DataFrame({'A': ['A', 'B', np.nan, 'B']})
df = df.fillna(df.median())
print(df)
A
0 A
1 B
2 NaN
3 B
What you should do is fillna with median only for numeric columns:
for col in df.select_dtypes(include=np.number):
df[col] = df[col].fillna(df[col].median())