bpo-12486: Document tokenize.generate_tokens() as public API

[takluyver]

I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented generate_tokens API to a public API.

Why should we do this?

• generate_tokens() has been there for several releases already (at least since 3.3.0), so we can start using it without having to wait until a new Python version is widespread.
• Someone (ahem 👼 ) might already be using it - IDE tab completion doesn't know that it's not public, and there's no other publicly documented way to tokenize code that's already unicode.
• Although the naming is not very clear, the interface is a close parallel to tokenize(), so it's easy to understand.
• We can do this without having to design, implement, review and debug any new interface!

https://bugs.python.org/issue12486

[Carreau]

It's also widely used:

https://github.com/search?l=Python&q=generate_tokens&type=Code

[takluyver]

The test failures appear to be unrelated to this pull request (there's a ConnectionRefusedError in the asyncio tests). Could someone prod them to rerun?

[takluyver]

Close/reopen, hopefully that will rerun tests.

[willingc]

Looks good. Thanks @takluyver

[bedevere-bot]

@willingc: Please replace # with GH- in the commit message next time. Thanks!

[Carreau]

Thanks !