Queue with maxsize can lead to deadlocks

[tomMoral]

BPO	33078
Nosy	@pitrou, @ned-deily, @applio, @tomMoral
PRs	bpo-33078 - FIX queue size on pickling error #6119 [3.7] bpo-33078 - Fix queue size on pickling error (GH-6119) #6178 bpo-33078 FIX failure on OSX sem_getvalue #6180

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-03-21.19:01:57.839>
created_at = <Date 2018-03-15.06:36:54.012>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'Queue with maxsize can lead to deadlocks'
updated_at = <Date 2018-03-21.21:27:09.127>
user = 'https://github.com/tomMoral'

bugs.python.org fields:

activity = <Date 2018-03-21.21:27:09.127>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-21.19:01:57.839>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2018-03-15.06:36:54.012>
creator = 'tomMoral'
dependencies = []
files = []
hgrepos = []
issue_num = 33078
keywords = ['patch']
message_count = 7.0
messages = ['313855', '314208', '314210', '314211', '314219', '314221', '314224']
nosy_count = 4.0
nosy_names = ['pitrou', 'ned.deily', 'davin', 'tomMoral']
pr_nums = ['6119', '6178', '6180']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33078'
versions = ['Python 3.7', 'Python 3.8']

[tomMoral]

The fix for the Queue._feeder does not properly handle the size of the Queue. This can lead to a situation where the Queue is considered as Full when it is empty. Here is a reproducing script:

import multiprocessing as mp

q = mp.Queue(1)


class FailPickle():
    def __reduce__(self):
        raise ValueError()

q.put(FailPickle())
print("Queue is full:", q.full())
q.put(0)
print(f"Got result: {q.get()}")

[pitrou]

New changeset e2f33ad by Antoine Pitrou (Thomas Moreau) in branch 'master':
bpo-33078 - Fix queue size on pickling error (GH-6119)
e2f33ad

[pitrou]

New changeset bb5b529 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
bpo-33078 - Fix queue size on pickling error (GH-6119) (GH-6178)
bb5b529

[ned-deily]

Buildbots are failing on macOS:

http://buildbot.python.org/all/#/builders/14/builds/804/steps/4/logs/stdio
http://buildbot.python.org/all/#/builders/93/builds/459/steps/4/logs/stdio

[ned-deily]

PR 6180 appears to have fixed the 3.x failures, thanks!

[pitrou]

Good, I think we can close again now!

[ned-deily]

(Fot the record, PR 6181 was the 3.7 backport of PR 6180.)