The Python Oracle

How can I use pickle to save a dict (or any other Python object)?

--------------------------------------------------
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Over a Mysterious Island Looping

--

Chapters
00:00 How Can I Use Pickle To Save A Dict (Or Any Other Python Object)?
00:19 Accepted Answer Score 1327
01:02 Answer 2 Score 178
02:20 Answer 3 Score 47
02:37 Answer 4 Score 17
02:51 Answer 5 Score 16
03:58 Thank you

--

Full question
https://stackoverflow.com/questions/1121...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #dictionary #pickle

#avk47



ACCEPTED ANSWER

Score 1338


Try this:

import pickle

a = {'hello': 'world'}

with open('filename.pickle', 'wb') as handle:
    pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)

with open('filename.pickle', 'rb') as handle:
    b = pickle.load(handle)

print(a == b)

There's nothing about the above solution that is specific to a dict object. This same approach will will work for many Python objects, including instances of arbitrary classes and arbitrarily complex nestings of data structures. For example, replacing the second line with these lines:

import datetime
today = datetime.datetime.now()
a = [{'hello': 'world'}, 1, 2.3333, 4, True, "x", 
     ("y", [[["z"], "y"], "x"]), {'today', today}]

will produce a result of True as well.

Some objects can't be pickled due to their very nature. For example, it doesn't make sense to pickle a structure containing a handle to an open file.




ANSWER 2

Score 179


Use:

import pickle

your_data = {'foo': 'bar'}

# Store data (serialize)
with open('filename.pickle', 'wb') as handle:
    pickle.dump(your_data, handle, protocol=pickle.HIGHEST_PROTOCOL)

# Load data (deserialize)
with open('filename.pickle', 'rb') as handle:
    unserialized_data = pickle.load(handle)

print(your_data == unserialized_data)

The advantage of HIGHEST_PROTOCOL is that files get smaller. This makes unpickling sometimes much faster.

Important notice: The answer was written in 2015 (Python 3.4!). Back then, the maximum file size of pickle was about 2 GB.

Alternative way

import mpu
your_data = {'foo': 'bar'}
mpu.io.write('filename.pickle', data)
unserialized_data = mpu.io.read('filename.pickle')

Alternative Formats

For your application, the following might be important:

  • Support by other programming languages
  • Reading / writing performance
  • Compactness (file size)

See also: Comparison of data serialization formats

In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python




ANSWER 3

Score 48


Save a dictionary into a pickle file.

import pickle

favorite_color = {"lion": "yellow", "kitty": "red"}  # create a dictionary
pickle.dump(favorite_color, open("save.p", "wb"))  # save it into a file named save.p

# -------------------------------------------------------------
# Load the dictionary back from the pickle file.
import pickle

favorite_color = pickle.load(open("save.p", "rb"))
# favorite_color is now {"lion": "yellow", "kitty": "red"}



ANSWER 4

Score 17


In general, pickling a dict will fail unless you have only simple objects in it, like strings and integers.

Python 2.7.9 (default, Dec 11 2014, 01:21:43)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type(globals())
<type 'dict'>
>>> import pickle
>>> pik = pickle.dumps(globals())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>>

Even a really simple dict will often fail. It just depends on the contents.

>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <function <lambda> at 0x102178668>: it's not found as __main__.<lambda>

However, if you use a better serializer like dill or cloudpickle, then most dictionaries can be pickled:

>>> import dill
>>> pik = dill.dumps(d)

Or if you want to save your dict to a file...

>>> with open('save.pik', 'w') as f:
...   dill.dump(globals(), f)
...

The latter example is identical to any of the other good answers posted here (which aside from neglecting the picklability of the contents of the dict are good).