socket.recv() and recvfrom() with MSG_TRUNC raise ValueError and lose the datagram since 3.15.1 #160622

Bug report

Bug description:

Since 3.15.1, socket.recv(bufsize, MSG_TRUNC) and socket.recvfrom(bufsize, MSG_TRUNC) raise ValueError: size larger than allocated size when the datagram is longer than bufsize. Without MSG_PEEK, the datagram is already consumed by the time the exception is raised, so the data is lost:

import socket
a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
a.send(b"abcdefgh")
a.send(b"second")
try:
    print(b.recv(2, socket.MSG_TRUNC))
except ValueError as e:
    print(e)
try:
    print(b.recvfrom(2, socket.MSG_TRUNC))
except ValueError as e:
    print(e)
b.setblocking(False)
b.recv(100)  # BlockingIOError: both datagrams are gone

3.15.1:

size larger than allocated size
size larger than allocated size

3.15.0 and 3.14 (the long-standing behavior, see gh-69121):

b'ab\x00\x00\x00\x00\x00\x00'
(b'se\x00\x00\x00\x00', None)

With MSG_TRUNC, Linux recv(2) returns the real length of the datagram even if it was longer than the buffer. sock_recv() and sock_recvfrom() pass that length to PyBytesWriter_FinishWithSize(), which, after c38a347 (backport of gh-157226 to 3.15), rejects a size larger than the allocated one. The same code is in main.

The old behavior was a bug too (gh-69121): the returned object was grown to the real datagram length, with only the first bufsize bytes being valid. Still, it is a behavior change in a bugfix release that breaks working code. For example, CRIU's pycriu used len(s.recv(1, socket.MSG_TRUNC | socket.MSG_PEEK)) to find out the message size (fixed in checkpoint-restore/criu#3268 by using recv_into(), which returns the value from recv(2) as is).

Raising after the data is consumed seems to be the worst option. Returning the min(outlen, bufsize) bytes actually received (as suggested in gh-69121, and similar to what recvmsg() does) would avoid both the data loss and the garbage bytes.

CPython versions tested on:

3.14, 3.15

Operating systems tested on:

Linux

Linked PRs

• gh-160622: Fix socket recv() and recvfrom() with MSG_TRUNC #160677