You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
RFC: deprecate write_bytes option in unpackers #197
I feel write_bytes option in Unpacker makes implementation complicated.
I forgot why I added it. I'll check history and consider another API for the usecase.
msgpack-python has too many options other msgpack implementation doesn't have.
I doubt some people uses this library for usecases which is not target of msgpack.
At least reading #23, target is not "avoid full unpacking", but "avoid repacking".
Since skip() and read_map_header() can be used to avoid full unpacking.
And I think write_bytes is not optimal way to achieve the target.
I'm not entirely convinced something like this is necessary at all.
Assuming it is, I would rather it be realized through a more useful API that allows access to the internal buffer of the Unpacker.
Tell unpack not to consume used buffer space. You can now unpack as many messages as necessary. This may also be useful when unpacking a large number of tiny messages.
Get a memoryview of the used buffer. This method can also be used to give access to the available space so it can be used for socket.read_into or similar.
I'm not entirely convinced something like this is necessary at all.
Yes. Removing the API should be first option, unless ruby's msgpack implementation doesn't have same API. (Since ruby's msgpack is maintained the author of msgpack.) http://ruby.msgpack.org/MessagePack/Unpacker.html
Since @jnothman has not responded yet, can we assume this feature can be dropped for 1.0?
I can see use cases where one might want to read the next packed message from a raw data stream, e.g., when reading messages from a file. This operation is semantically similar to skip (as it looks at only parts of the data until it finds the end of the message) but would return the skipped data. That data can then still be unpacked if necessary.
Since skip has no return value right now, maybe it could do this?
Might I also add that the current write_bytes implementation causes a lot of Python method calls to write tiny pieces of data one-by-one, whereas skip translates to nicely optimized C-code?
Removing functionality should always be the last option. It breaks code. However, semi-broken APIs have to be removed at some point. IMO nobody should use msgpack like this.
Hi, I didn't see the ping in May, sorry. I think this was a context where the length of or exact message encoded was used, as well as the decoded object. I suspect getting the raw msgpack on skip would be a sufficient alternative. I agree that it is not a feature worth slowing down normal use, and I suspect you're going to break little code out there. (The case where I had used it it seems to be no longer used.)
I don't think it's slowing down normal use (this should be easy pickings for branch prediction).
It's more about #191, i.e., if 1.0 is going to break code we might as well clean up the API in other places.
I just stumbled across this thread and wanted to make sure you're aware of the following:
write_bytes was previously suggested as a solution for indexing msgpack files (see here). If write_bytes becomes deprecated, what is the recommended approach for finding byte offsets of msgpack items within larger msgpack files?
Has write_bytes deprecation already been finalized?
IMO this is a classic case of "you're doing it wrong". messagepack is not a wire protocol or container format. Sending raw JSON messages over a socket or dumping them in a file one after the other is almost always a terrible idea. If you need more functionality than just reading one message after the other you need some form of protocol around it.
If you can't fix your protocol and desperately need to index a packed stream, I suggest using read_size instead. Look at this gist: https://gist.github.com/jfolz/74b130649e9134c35b5fcf200a1ddbe7
It tells the Unpacker to read from the stream one byte at a time. Sounds very slow, so we wrap that stream in a BufferedReader.
Timings (py3.5):
It's too close to really call one implementation faster than the other, but read_size needs just one extra line for the BufferedReader vs a callback and try-except block for write_bytes.
Couldn't stand these slow solutions. #227 adds tell() to Unpacker. No real reason to have a write_bytes-type API really. If you need the data you can just read it from a seekable stream after unpacking.
Is this change the reason why I can no longer provide a byte buffer parameter to the skip method ? My code is using the example code on how to deserialize a map and fails to execute because skip no longer accepts the parameter :
`
nelems = unpacker.read_map_header()
for i in range(nelems):
key = unpacker.unpack()
bytestream = BytesIO()
unpacker.skip(bytestream.write) <-- fails to compile
Activity
methane commentedon May 20, 2016
msgpack-python has too many options other msgpack implementation doesn't have.
I doubt some people uses this library for usecases which is not target of msgpack.
jfolz commentedon May 20, 2016
This? #23
Seems like a convoluted way to do framing.
methane commentedon May 21, 2016
cc: @jnothman
At least reading #23, target is not "avoid full unpacking", but "avoid repacking".
Since
skip()andread_map_header()can be used to avoid full unpacking.And I think
write_bytesis not optimal way to achieve the target.methane commentedon May 22, 2016
#198 makes clear what write_bytes does.
https://github.com/msgpack/msgpack-python/pull/198/files#diff-0924a5fc6aaa4b057364baf18aa2c267R546
In Packer, we have similar option
autoreset. But I feelautoresetis not good, too.I'll look more later when I have time.
methane commentedon May 22, 2016
Current API is bad because
write_bytesis just a additional return value semantically.Current API:
is same to:
I dislike callback only for return value. There are some options for new API.
1. Add
captureoptionPros: simple
Cons: Cant add typehint to
.unpack()method.2. Add
.unpack_and_capture()methodPros: clean
Cons: What should I do when adding more options?
.unpack_and_capture_and_XXX()is horrible.3. Add
.start_capture()and.stop_capture()Pros: User can capture serialized data for multiple
.unpack()call efficiently.Cons: Make unpacker more stateful
Since recent Python has type hinting, I dislike option 1. I'll choose from option 2 or 3.
I don't know is there more better name than
capture, since my English vocabulary is too bad.Any ideas?
jfolz commentedon May 22, 2016
I'm not entirely convinced something like this is necessary at all.
Assuming it is, I would rather it be realized through a more useful API that allows access to the internal buffer of the Unpacker.
socket.read_intoor similar.methane commentedon May 23, 2016
Yes. Removing the API should be first option, unless ruby's msgpack implementation doesn't have same API. (Since ruby's msgpack is maintained the author of msgpack.)
http://ruby.msgpack.org/MessagePack/Unpacker.html
jfolz commentedon Sep 10, 2016
Since @jnothman has not responded yet, can we assume this feature can be dropped for 1.0?
I can see use cases where one might want to read the next packed message from a raw data stream, e.g., when reading messages from a file. This operation is semantically similar to
skip(as it looks at only parts of the data until it finds the end of the message) but would return the skipped data. That data can then still be unpacked if necessary.Since
skiphas no return value right now, maybe it could do this?jfolz commentedon Sep 10, 2016
Might I also add that the current
write_bytesimplementation causes a lot of Python method calls to write tiny pieces of data one-by-one, whereasskiptranslates to nicely optimized C-code?Removing functionality should always be the last option. It breaks code. However, semi-broken APIs have to be removed at some point. IMO nobody should use msgpack like this.
jnothman commentedon Sep 10, 2016
Hi, I didn't see the ping in May, sorry. I think this was a context where the length of or exact message encoded was used, as well as the decoded object. I suspect getting the raw msgpack on skip would be a sufficient alternative. I agree that it is not a feature worth slowing down normal use, and I suspect you're going to break little code out there. (The case where I had used it it seems to be no longer used.)
jfolz commentedon Sep 10, 2016
I don't think it's slowing down normal use (this should be easy pickings for branch prediction).
It's more about #191, i.e., if 1.0 is going to break code we might as well clean up the API in other places.
methane commentedon Sep 11, 2016
I found relating question on Stackoverflow
http://stackoverflow.com/a/38726237/3430118
As I answered there, double-packing is idiom used by msgpack owner.
acorver commentedon Apr 25, 2017
I just stumbled across this thread and wanted to make sure you're aware of the following:
write_bytes was previously suggested as a solution for indexing msgpack files (see here). If write_bytes becomes deprecated, what is the recommended approach for finding byte offsets of msgpack items within larger msgpack files?
Has write_bytes deprecation already been finalized?
methane commentedon Apr 25, 2017
Other major language's msgpack libraries support it?
If no, why only Python should support it?
I want to know realistic and typical use case.
I don't want to make msgpack-python complicated to support niche use cases.
jfolz commentedon Apr 25, 2017
IMO this is a classic case of "you're doing it wrong". messagepack is not a wire protocol or container format. Sending raw JSON messages over a socket or dumping them in a file one after the other is almost always a terrible idea. If you need more functionality than just reading one message after the other you need some form of protocol around it.
If you can't fix your protocol and desperately need to index a packed stream, I suggest using
read_sizeinstead. Look at this gist:https://gist.github.com/jfolz/74b130649e9134c35b5fcf200a1ddbe7
It tells the
Unpackerto read from the stream one byte at a time. Sounds very slow, so we wrap that stream in aBufferedReader.Timings (py3.5):
It's too close to really call one implementation faster than the other, but
read_sizeneeds just one extra line for theBufferedReadervs a callback and try-except block forwrite_bytes.jfolz commentedon Apr 29, 2017
Couldn't stand these slow solutions. #227 adds
tell()toUnpacker. No real reason to have awrite_bytes-type API really. If you need the data you can just read it from a seekable stream after unpacking.methane commentedon Jul 24, 2017
Thanks. I'll mark
write_bytesdeprecated, and remove it by 1.0.Deprecate write_bytes option in Unpacker. (#262)
ChristThePhone commentedon Feb 4, 2019
Is this change the reason why I can no longer provide a byte buffer parameter to the skip method ? My code is using the example code on how to deserialize a map and fails to execute because skip no longer accepts the parameter :
`
nelems = unpacker.read_map_header()
`
What's the recommended way of doing this now ?
methane commentedon Feb 4, 2019
Why you did it?
Without more higher goal, there is no recommendation.