Backward incompatible API change toward 1.0

DRAFT
This issue is for write down my ideas.

Changing default behavior to use new spec.

See also: https://github.com/msgpack/msgpack/blob/master/spec.md#upgrading-messagepack-specification.

v1.0

Drop encoding and unicode_errors option in Unpacker. Add raw=False option instead. When raw=True is passed, raw data in msgpack is deserialized into bytes object.

Drop encoding and unicode_errors option in Packer. Unicode is encoded with UTF-8 always.
'surrogatepass' is not allowed to keep generated msgpack clean. use_bin_type=False is the only way to create dirty (raw contains non UTF-8 data) msgpack.

v0.5.x

Add raw=True option to unpacker.

v0.6

0.6 is version for warnings.

Packer warns when encoding or unicode_errors is specified.

Unpacker warns when encoding or unicode_errors is specified. ("Use raw=False instead").

PyPI package name

In past, easy_install crawls msgpack website and finds msgpack-x.y.z.tar.gz. But it was for C.
That's why I moved from msgpack to msgpack-python

Sadly, pip doesn't support transitional package (empty, but depends on new name package).
So I release msgpack as both of msgpack-python and msgpack for a while until 1.0.

As 1.0, I release msgpack only.

msgpack could be the v1.0 release on PyPI, and v0.5 is kept as msgpack-python for compatibility.

Dropping/renaming parameters should be avoided as much as possible. If the only downside of use_bin_type is that it isn't as descriptive as it could be, that's not good enough a reason to break code. Deprecating it and printing warnings is OK. numpy usually keeps parameters around a couple of versions even if their semantics are broken.

I have a question about

But use_bin_type=False doesn't use raw8 type too.

This is about the descriptiveness, not the behavior of the parameter, correct? compat_mode=True would also disable raw8, since it is not part of the original spec.

Also, what would the exact effect of raw_as_byte=True be? As I understand it, it would disable the decoding of str* values. If that is what it does, I think it should be decode_str or simply decode.

Dropping of the encoding parameter may seem harsh, but the spec says

String extending Raw type represents a UTF-8 string

so there really is no argument. I'm a firm believer in utf8everywhere, so I fully support this.

In conclusion, it's inevitable to break some code somewhere.
The spec says the new types should be used by default for good reason.
The majority of projects (most likely) use the default parameters. These people will - under all circumstances - suddenly receive str (PY2: unicode) objects in instead of bytes (PY2: str) and run into all sorts of funny Attribute/TypeErrors somewhere in their code.

If msgpack-python stays at v0.5 on PyPI, existing projects work and receive deprecation warnings, which is good.
If msgpack is v1.0+ on PyPI it can have a cleaner API without breaking existing projects, since upgrading will not happen implicitly.
Of course this has some downsides as well. What happens if both msgpack-python and msgpack are installed? They live in the same namespace.

If the only downside of use_bin_type is that it isn't as descriptive as it could be, that's not good enough a reason to break code. Deprecating it and printing warnings is OK. numpy usually keeps parameters around a couple of versions even if their semantics are broken.

Yes. It's why 0.5 will be released before 1.0. 1.0 will be released at least 6 months later than 0.5. (I think one year later).
And why I want to rename is the msgpack spec says it as "compatibility mode".

This is about the descriptiveness, not the behavior of the parameter, correct?

Yes. Some people claims me "msgpack-python deosn't use raw8! It's wrong!".
use_bin_type is too wrong name for them.

Also, what would the exact effect of raw_as_byte=True be? As I understand it, it would disable the decoding of str* values.

Yes. It should be utf-8 in most case.

If that is what it does, I think it should be decode_str or simply decode.

Hmm, I'm considering decode_str=True.

Of course this has some downsides as well. What happens if both msgpack-python and msgpack are installed? They live in the same namespace.

It must be avoided. I think msgpack-python should be transitional package at last.
Another option is making msgpack2 package. But it would be too nervous.
Even requests package did backward incompatible major version up.

Yes. It's why 0.5 will be released before 1.0. 1.0 will be released at least 6 months later than 0.5. (I think one year later).
And why I want to rename is the msgpack spec says it as "compatibility mode".

That should be plenty of time for actively developed projects to make the switch.

It must be avoided. I think msgpack-python should be transitional package at last.
Another option is making msgpack2 package. But it would be too nervous.
Even requests package did backward incompatible major version up.

Thinking some more about conflicts. IMO having two incompatible versions live in the same namespace is way worse than breaking some projects that didn't specify the required version correctly (msgpack-python instead of msgpack-python<1.0.0).

Imagine msgpack 1.0.0 is already installed. A package that depends on msgpack-python would cause pip to implicitly overwrite this with 0.5.0, correct? Yet everything is fine - both dist-info dirs are there.

I'm no PyPI-expert, but is it possible to have msgpack-python be an alias to msgpack? I.e., if a project depends on msgpack-python, msgpack is installed instead?

Recent example about transitional package is uritemplate.py

It is empty package, and depends to uritemplate.

I see. uritemplate.py does nothing, it simply requires uritemplate. That would be a good way to do it.

I released 0.5 without implementing some new options / deprecations.
I'll implement them in 0.6 instead. I don't know how many 0.x before 1.0 for now.

It is unclear to me what the impact the following will be:

Unpacker warns when decode_str is not specified. ("Future version of msgpack will remove encoding, unicode_errors options and decode str types with utf-8. If you want to unpack data contains invalid utf-8 sequence in raw, pass decode_str=False. Otherwise, decode_str=True to suppress this warning")

What would decode_str=false actually do?

I handle data that I get from various sources that are potentially icky and not well formed; and typically re-pack data into msgpack for storage. One use case is re-serializing data from JSON to msgpack format. Python3 actually does the unicode decoding with surrogatepass set to handle some invalid str data which may have been legitimately shoveled into json. Being able to set the unicode decode error handling allows me to handle these potentially icky data sources without any issue.

What would decode_str=false actually do?

I missed to replace it to raw_as_bytes in 1.0.
When raw_as_bytes=True, msgpack doesn't decode raw as bytes.

New msgpack should use raw for canonical UTF-8 data (no surrogatepass).
But old msgpack may contain non UTF-8 data.

In such case, you can get raw as bytes and anything you want. (e.g. .decode('latin1'), or .decode('utf-8', 'surrogateescape').

And you can create old msgpack by packb(b"foo", use_bin_type=False)

do you mean the "raw" Unpacker parameter, not "raw_as_bytes"?

do you mean the "raw" Unpacker parameter, not "raw_as_bytes"?

Sorry, I can't understand your question means. What "what do you mean" refers?
Please quote it.

In misc. comments posted above, you are talking about raw_as_bytes.

I didn't find that parameter, but there is raw. Did you mean that?

I searched and found #276.
Yes, it's called raw now.

does that msgpack wrapper look good? borgbackup/borg#3943

btw, if you release a beta or rc first of this on pypi (instead of going directly to a release), one could better test it before potential bigger damage is happening.

@ThomasWaldmann

@ThomasWaldmann I released 1.0.0rc1 now. Please try it.
https://pypi.org/project/msgpack/1.0.0rc1/

I will add some tests and document before v1.0.
I will release v1.0 in next year, after Python 2 EOL.

Did a first linux-based test run with borg 1.1-maint branch and msgpack 1.0.0rc1 and did have same test results as with older msgpack releases. \o/

Is there bsd, solaris or macOS specific code in msgpack so it is worthwhile running the tests there?

Is there bsd, solaris or macOS specific code in msgpack so it is worthwhile running the tests there?

I don't think there is any platform-specific code.
But our CI doesn't run test on architectures other than x86 and amd64.
I don't know there is a free CI service for OSS which provides big-endian architecture.

I'll do some tests in a powerpc qemu VM (big endian) with debian.
It's slow, but works.

Hmm, I can't test msgpack with the borg tests on debian jessie powerpc (dependency issues, jessie is too old for borg master). I'll try running the msgpack tests...

Even testing just msgpack was a bit of a pain:

py27: tests all worked, but it seems it does not test _cmsgpack on 2.7, so there wasn't much point to it.
py34: tests all worked (but I could not start that via tox as py34 is already unsupported, so i just started pytest and made sure from msgpack import _cmsgpack worked)
could not test py35+ as the powerpc debian VM only has 3.4 and the whole architecture was dropped for newer debian releases.

BTW, i researched for alternative BE architectures that are still supported: maybe mips on qemu is an option IF one does not need a lot of memory. So maybe an option for msgpack, but maybe not for borg.

Navigation Menu

Repository navigation

Backward incompatible API change toward 1.0 #191

Changing default behavior to use new spec.

v1.0

v0.5.x

v0.6

PyPI package name

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Backward incompatible API change toward 1.0 #191

Description

Changing default behavior to use new spec.

v1.0

v0.5.x

v0.6

PyPI package name

Activity

jfolz commented on May 5, 2016

jfolz commented on Sep 11, 2016

methane commented on Oct 3, 2016

jfolz commented on Oct 3, 2016

methane commented on Oct 3, 2016

jfolz commented on Oct 3, 2016

methane commented on Jan 7, 2018

vEpiphyte commented on Jan 9, 2018

methane commented on Jan 10, 2018

methane commented on Jan 10, 2018

ThomasWaldmann commented on Jun 30, 2018

methane commented on Jul 1, 2018

ThomasWaldmann commented on Jul 1, 2018

methane commented on Jul 2, 2018

ThomasWaldmann commented on Jul 7, 2018

ThomasWaldmann commented on Dec 5, 2019

methane commented on Dec 12, 2019

ThomasWaldmann commented on Dec 13, 2019

methane commented on Dec 13, 2019

ThomasWaldmann commented on Dec 14, 2019

ThomasWaldmann commented on Dec 15, 2019

ThomasWaldmann commented on Dec 15, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions