Skip to content

Make test.script_helper more comprehensive, and use it in the test suite #53726

Closed
@pitrou

Description

@pitrou
BPO 9517
Nosy @ncoghlan, @pitrou, @ezio-melotti, @bitdancer, @voidspace, @berkerpeksag, @serhiy-storchaka
Dependencies
  • bpo-23981: Update test_unicodedata.py to use script_helpers
  • bpo-24033: Update _test_multiprocessing.py to use script helpers
  • bpo-24279: Update test_base64 to use test.support.script_helper
  • bpo-24398: Update test_capi to use test.support.script_helper
  • Files
  • script_helper_del_refcount.patch
  • iss9517_move_script_helpers_py.patch
  • iss9517_move_script_helpers_py.patch
  • iss9517_move_script_helpers.patch
  • issue9517.diff
  • iss9517_move_script_helpers_review1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2010-08-04.22:50:46.386>
    labels = ['type-feature', 'tests']
    title = 'Make test.script_helper more comprehensive, and use it in the test suite'
    updated_at = <Date 2016-05-20.18:20:34.405>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2016-05-20.18:20:34.405>
    actor = 'BreamoreBoy'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Tests']
    creation = <Date 2010-08-04.22:50:46.386>
    creator = 'pitrou'
    dependencies = ['23981', '24033', '24279', '24398']
    files = ['19975', '39003', '39038', '39176', '39177', '39189']
    hgrepos = []
    issue_num = 9517
    keywords = ['patch']
    message_count = 55.0
    messages = ['112917', '112969', '113145', '119517', '119741', '123503', '123504', '123516', '123601', '123602', '123633', '123637', '173718', '221971', '222027', '240305', '240524', '240555', '240664', '240801', '240818', '240982', '240990', '241001', '241010', '241069', '241101', '241103', '241112', '241120', '241122', '241165', '241186', '241221', '241238', '241240', '241244', '241295', '241836', '241839', '241842', '241848', '241906', '241913', '242090', '242641', '242690', '242934', '242945', '243000', '243030', '244001', '244924', '245920', '247985']
    nosy_count = 11.0
    nosy_names = ['ncoghlan', 'pitrou', 'ezio.melotti', 'r.david.murray', 'michael.foord', 'Rodrigue.Alcazar', 'python-dev', 'berker.peksag', 'serhiy.storchaka', 'bobcatfish', 'flipmcf']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue9517'
    versions = ['Python 3.5']

    Activity

    pitrou

    pitrou commented on Aug 4, 2010

    @pitrou
    MemberAuthor

    test.script_helper has a couple of dedicated functions to launch a Python interpreter instance in a subprocess. Unfortunately, it is little used and most test modules use their own ad hoc calls to subprocess instead.

    Remedying the situation would require:

    • adding functions to script_helper (currently, the available functions merge stdout and stderr together, which is clearly undesireable)
    • perhaps improve the existing functions (kill_python() does a strange dance instead of calling communicate() on the subprocess.Popen object, is there a reason for that?)
    • convert most uses of subprocess.<some_func>([sys.executable, ...]) in the test suite to use script_helper instead

    This was suggested by Nick in bpo-477863.

    added
    testsTests in the Lib/test dir
    type-featureA feature request or enhancement
    on Aug 4, 2010
    ncoghlan

    ncoghlan commented on Aug 5, 2010

    @ncoghlan
    Contributor

    (The email daemon was not in a happy place, so posting directly)

    On Thu, Aug 5, 2010 at 8:50 AM, Antoine Pitrou <report@bugs.python.org> wrote:

    • perhaps improve the existing functions (kill_python() does a strange dance instead of calling communicate() on the subprocess.Popen object, is there a reason for that?)

    script_helper just factored out the old test_cmd_line approach which
    was in turn based on a minimalistic change from a popen2 based
    implementation (see
    http://svn.python.org/view/python/trunk/Lib/test/test_cmd_line.py?r1=54386&r2=55245).
    Doing something smarter probably isn't a bad idea.

    ncoghlan

    ncoghlan commented on Aug 6, 2010

    @ncoghlan
    Contributor

    One other feature for the new-and-improved helpers: add a flag to allow "-E" to be omitted (as per the comment in test_cmd_line)

    self-assigned this
    on Aug 6, 2010
    ncoghlan

    ncoghlan commented on Oct 24, 2010

    @ncoghlan
    Contributor

    I still think this is a good idea, I'm just not actively working on it. It might make a good project for someone wanting to get to know the process of working on CPython without having to deal with anything that is particularly tricky to understand.

    removed their assignment
    on Oct 24, 2010
    RodrigueAlcazar

    RodrigueAlcazar commented on Oct 27, 2010

    @RodrigueAlcazar
    Mannequin

    someone wanting to get to know the process of working on CPython without having to deal with anything that is particularly tricky to understand.

    That sounds exactly like me :)

    I can have a look at this ticket.

    bitdancer

    bitdancer commented on Dec 6, 2010

    @bitdancer
    Member

    I just tried using script_helper in a new test, so I have a couple of comments.

    I don't see stdout and stderr being conflated, it looks to me like they are returned separately, at least by the assert methods.

    The assert methods return results, which is unlike other assert methods. This is very useful, even essential, and I wouldn't want to give it up. That conflicts with the current unittest conventions, though.

    It would be a big help if 'err' were returned with the refcount line removed if it is there, which would make tests using the methods return the same 'err' regardless of whether they are run under a debug build or not.

    I think the names of the two assert functions should follow the current unit test conventions (assertPythonRunOK and asssertPythonRunNotOK, perhaps?)

    pitrou

    pitrou commented on Dec 6, 2010

    @pitrou
    MemberAuthor

    I just tried using script_helper in a new test, so I have a couple of
    comments.

    I don't see stdout and stderr being conflated, it looks to me like
    they are returned separately, at least by the assert methods.

    That's because I wrote the assert methods since this issue was opened :)

    It would be a big help if 'err' were returned with the refcount line
    removed if it is there, which would make tests using the methods
    return the same 'err' regardless of whether they are run under a debug
    build or not.

    Indeed.

    I think the names of the two assert functions should follow the
    current unit test conventions (assertPythonRunOK and
    asssertPythonRunNotOK, perhaps?)

    Well, they are functions, not methods, so I don't think they have to
    follow the other convention.

    bitdancer

    bitdancer commented on Dec 7, 2010

    @bitdancer
    Member

    OK, fine on the convention, but I'd still like a more memorable name for assert_python_failure. I've been working on this issue off and on today, and I've had to look up that name at least four times. I can remember assert_python_ok, but I can't remember whether its inverse is assert_python_fails, assert_python_bad, or what. For some reason I haven't guessed 'failure' even once so far :) (I know it's not assert_python_not_ok because I remember it isn't parallel...)

    bitdancer

    bitdancer commented on Dec 8, 2010

    @bitdancer
    Member

    Here is a patch that causes _assert_python to remove the refcount lines from stderr.

    bitdancer

    bitdancer commented on Dec 8, 2010

    @bitdancer
    Member

    Hmm. Having posted that it occurs to me that it could be useful to have the _remove_refcount function in test.support as remove_refcount instead.

    49 remaining items

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      testsTests in the Lib/test dirtype-featureA feature request or enhancement

      Projects

      No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

        Make test.script_helper more comprehensive, and use it in the test suite · Issue #53726 · python/cpython

        Follow Lee on X/Twitter - Father, Husband, Serial builder creating AI, crypto, games & web tools. We are friends :) AI Will Come To Life!

        Check out: eBank.nz (Art Generator) | Netwrck.com (AI Tools) | Text-Generator.io (AI API) | BitBank.nz (Crypto AI) | ReadingTime (Kids Reading) | RewordGame | BigMultiplayerChess | WebFiddle | How.nz | Helix AI Assistant