How to increase max_array_len #338

Activity

1. methane commented on Dec 13, 2018

   

   methane

   on Dec 13, 2018

   Member

   More actions

   Why don't you search before asking here?
   It is written in docstring and document.
   https://msgpack-python.readthedocs.io/en/latest/api.html#msgpack.Unpacker
2. methane

   closed this as completedon Dec 13, 2018
3. methane commented on Dec 13, 2018

   

   methane

   on Dec 13, 2018

   Member

   More actions

   See also https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst

   unpacker: Default value of input limits are smaller than before to avoid DoS attack. If you need to handle large data, you need to specify limits manually. (#319)
5. fake-name commented on Jan 28, 2019

   

   fake-name

   on Jan 28, 2019

   Last edited by fake-name

   More actions

   Do continue on this issue, how do you disable the limits?

   The documentation states:

   max_array_len (int) – Limits max length of array. (default: max_buffer_size or 128*1024)

   So, is it max_array_len or is it 128*1024? Additionally, the function signature for Unpacker has it specified to defaulting to max_array_len=-1, which isn't either of the previous values. Does -1 mean 128*1024, or does it mean INT_MAX - 1 (like I'd expect it to).

   In general, default arguments should be the default value, or None (for cases of mutable values, like list), in which case it is explicitly documented that passing none/-1/whatnot is internally translated to <default value>.

   Basically, the verbiage in the documentation isn't clear how all the options interact, and what values mean what.

   Additionally, it doesn't really make sense to me to have max_array_len/max_map_len etc.... do anything if max_buffer_size is set to 0 (INT_MAX), and having to specify a pile of options just to turn off safety checking for contexts where I'm shipping LARGE chunks of data back and forth between endpoints I control is annoying. I have a context where I pretty regularly have 100+ MByte messages.

   ---

   Really, if you're going to have checking in place, add another class like UntrustedUnpacker that does this stuff.
6. fake-name

   added 2 commits that reference this issue on Jan 28, 2019

   Revert msgpack-python because of ridiculous defaults in newer versions.

   ...

   f46074b

   Revert msgpack-python because of ridiculous defaults in newer versions.

   ...

   3076de8
7. methane commented on Jan 28, 2019

   

   methane

   on Jan 28, 2019

   Member

   More actions

   Do continue on this issue, how do you disable the limits?

   There are no way. msgpack has it's upper bound (2**64). So max_xxx_len = 2**64 is enough.

   So, is it max_array_len or is it 128*1024?

   Do you mean "is it max_buffer_len or is it 128*1024?

   If max_buffer_len >0, it's max_buffer_len.
   If max_buffer_len == 0 (or other falthy values), it's 128*1024.
   This usage of or is very common Python idiom.

   Does -1 mean 128*1024, or does it mean INT_MAX - 1 (like I'd expect it to).

   -1 is default value, and semantics for default value is described already.

   Additionally, it doesn't really make sense to me to have [snip]. I have a context where I pretty regularly have 100+ MByte messages.

   You don't have to "specify a pile of options". (BTW, I don't think 5 is not "a pile of". You can count it by one hand.)
   You can just use max_buffer_len=10*1024**3 (10GB) in such case. If 10GB is not enough, you can use 100GB (if your machine have 100GB RAM).

   Really, if you're going to have checking in place, add another class like UntrustedUnpacker that does this stuff.

   If I added it, people keep using Unpacker for untrusted source. Security hole is keep living.
   Anyway, it's too late. Making weaker than released version is not possible option.
8. fake-name commented on Jan 28, 2019

   

   fake-name

   on Jan 28, 2019

   Last edited by fake-name

   More actions

   -1 is default value, and semantics for default value is described already.

   My point was that no, it's not. And you still haven't specified if -1 means INT_MAX-1 or 128*1024. Reading the docs page https://msgpack-python.readthedocs.io/en/latest/api.html#msgpack.Unpacker, I can interpret it either way.

   This usage of or is very common Python idiom.

   I mean, if you're writing code, yeah, but it doesn't help in the docs. And the general default value for "use defaults" is None. Using -1 seems like you're writing C code, not python.

   You don't have to "specify a pile of options". (BTW, I don't think 5 is not "a pile of". You can count it by one hand.)

   If I needed to not specify anything previously, and the functionality is (at least in my application, and probably mode uses of the library) pointless, yes, it is a pile of options.

   If max_buffer_len >0, it's max_buffer_len.
   If max_buffer_len == 0 (or other falthy values), it's 128*1024.

   So 0 means 128*1024, except for max_buffer_size, where it means INT_MAX.

   It'd be nice if 0 meant the same thing when passed to max_buffer_size, max_array_len, max_map_len and max_ext_len. Having them all act differently is a great recipe for confusion (and really, use None for "this will be overridden by a externally specified default").

   If I added it, people keep using Unpacker for untrusted source. Security hole is keep living.

   I don't agree this is a problem in the first place. You might as well just make unpackb() return None in all cases, for any input, otherwise someone will figure out how to break something.

   Libraries are the art of balancing unusual use-cases against the common use cases. Is anyone exposing a messagepack API unauthenticated to the public? Adding features that make that work better at the cost of normal usage is not helpful or good design.
9. methane commented on Jan 28, 2019

   

   methane

   on Jan 28, 2019

   Member

   More actions

   Libraries are the art of balancing unusual use-cases against the common use cases. Is anyone exposing a messagepack API unauthenticated to the public?

   msgpack is JSON-like format. Msgpack can be used anywhere JSON is used.
   It is used very widely, more than you think.
   And authenticated or not is not matter.

   Adding features that make that work better at the cost of normal usage is not helpful or good design.

   That's why I released 0.6.1.
10. msgpack

    locked as too heated and limited conversation to collaborators on Jan 28, 2019