Python 3.6 Pickling Custom Procedure

I have some objects of class A which has its own method to be pickled, call it custom_module.customPickle(A) which takes an instance of A and returns a serialization string. I also

Solution 1:

You can use the copyreg module to register custom functions for pickling and unpickling; the function you register acts like a __reduce__ method on the class.

If you return a tuple of (unpickle_function, state), then the registered unpickle_function callable will be called to unpickle it again, with state as the argument, so you can use your music21.converter.thawStr() function there:

import copyreg
import music21.converter
import music21.stream


defpickle_music21_stream(stream_obj):
    return music21.converter.thawStr, (music21.converter.freezeStr(stream_obj),)

copyreg.pickle(music21.stream.Stream, pickle_music21_stream)

(the constructor argument to copyreg is ignored in recent Python versions)

This registers a global handler for those objects. You can also use a dispatch table per pickler, see [*Dispatch Tables on how you'd register one.

Now, when pickling, when encountering any instances of Stream the handle_stream() function is used to produce a serialisation, and the thawStr() function will be used to unpickle that data again.

However, the music21.converter functions use pickle themselves. They effectively pack and clean up the stream, and then pickle the resulting Stream instance. This will then call the custom handler, and you have an infinite loop.

The work-around is to use a custom dispatch table to handle pickling and unpickling. Avoid using copyreg in this case, as it sets a global hook that'll be called recursively each time a Stream object is being pickled.

Your own pickle infrastructure needs to use a custom pickler:

import copyreg
import io
import pickle
import music21.converter
import music21.stream


defpickle_music21_stream(stream_obj):
    return music21.converter.thawStr, (music21.converter.freezeStr(stream_obj),)


defdumps(obj):
    f = io.BytesIO()
    p = pickle.Pickler(f)
    p.dispatch_table = copyreg.dispatch_table.copy()
    p.dispatch_table[music21.stream.Stream] = pickle_music21_stream
    p.dump(obj)
    return f.getvalue()


defloads(data):
    return pickle.loads(data)  # hook is registered in the pickle data

Here the custom function is only called when a Stream instance is found in your own data structure. The music21 routines use the global pickle.dumps() and pickle.loads() functions and won't use the same hook.