Exception _abc_impl is set to a wrong type when using format_exception

[jato-c8y]

Bug report

Bug description:

Exception encountered during attempt to upgrade a C++ wrapper using the C API from Python 3.9 to 3.13.3.

Exception trace:

_abc_impl is set to a wrong type
<FrameSummary file python/3.13.3/lib/python3.13/traceback.py, line 155 in format_exception>
<FrameSummary file python/3.13.3/lib/python3.13/traceback.py, line 1406 in format>
<FrameSummary file python/3.13.3/lib/python3.13/traceback.py, line 993 in emit>
<FrameSummary file python/3.13.3/lib/python3.13/traceback.py, line 1259 in format_exception_only>
<FrameSummary file <frozen abc>, line 119 in __instancecheck__>

Details:
Exception is seen on attempt to use format_exception from the traceback module.
Previously we were using PyErr_Fetch to get type, value and traceback of the Exception.
We note the signature change of format_exception requires an Exception object.
We switched to use PyErr_GetRaisedException and pass this to format_exception.
We have a class that stores the exception reference object from PyErr_GetRaisedException which is then used to rethrow the exception to be caught in our code.
The catch then uses a callable function to the imported traceback module to invoke format_exception with the exception object. We then see the above exception stack. format_exception(py::make_tuple(e.getException())); Where getException returns the Exception object reference stored.

We have been unable to reproduce this in a Python script yet.

Checked the callable and is as expected <function format_exception at 0x7fda88878720>.
Checked the initial caught exception object and can print the 'what' as Exception('Thrown exception from static init').
Checked the initial caught exception is of type <class 'Exception'>.
Confirmed signature of format_exception using the inspect module and parameters confirmed as exc, value, tb, limit, chain, kwargs

We are running in a sub-interpreter with the GIL taken prior to the try, catch that sees this exception.

We are hoping the above is enough to spot a potential issue or anti-pattern.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[picnixz]

Can you give us the C code that triggers this issue? or at least a prototype and not with just words please? it's hard to track down the flow.

cc @ZeroIntensity

[ZeroIntensity]

Yeah, it'd be nice to see some code.

We are running in a sub-interpreter with the GIL taken prior to the try, catch that sees this exception.

It's hard to say without seeing code, but my best theory is that you're taking the GIL of the main interpreter and then trying to raise it in a subinterpreter, or vice versa. Are you using PyGILState_Ensure, by any chance?

[matj-sag]

Hi all, thanks for the advice so far. I would love to have a minimal repro, unfortunately it's being used via a custom C++ shim (whose tests all pass) and then embedded in a much larger program.

That GIL option is definitely plausible - I thought about it myself, but couldn't see anything going wrong in that regard.

We're generally using this code to aquire GILs:

/** Aquire the GIL. Has std::lock_guard semantics. GIL will be released when this class leaves the scope.
 * The GIL must be acquired before calling from a non-python thread into python. This class can't be moved or copied.
 */
class GIL
{
public:
	/** Acquire the GIL */
	GIL(const interpreter &ip): istate(fns::PyThreadState_GetInterpreter(ip.istate)), tstate(fns::PyThreadState_New(istate)), held(true)
	{
		fns::PyEval_RestoreThread(tstate);
	}
	/** Release the GIL */
	~GIL() { release(); }
	void release()
	{
		if (held) {
			held=false;
			fns::PyThreadState_Clear(tstate);
			fns::PyEval_ReleaseThread(tstate);
			fns::PyThreadState_Delete(tstate);
		}
	}
	GIL(const GIL &) = delete;
	GIL(GIL &&) = delete;
	GIL &operator=(const GIL &) = delete;
	GIL &operator=(GIL &&) = delete;
private:
	PyInterpreterState *istate;
	PyThreadState *tstate;
	bool held;
};

This is being used something like:

GIL gil{subinterpreter};
try {
     / python function that throws, which we wrap in a py::exception
} catch (const py::exception &e) {
    / attempt to format_exception the exception, which fails as described
}

Exceptions are being wrapped with:

inline void check_exception()
{
	if (fns::PyErr_Occurred()) {
		python_exception pe(fns::PyErr_GetRaisedException());
		throw pe;
	}
}

One thing which does occur to me, is that we are looking up format_exception in the default interpreter and storing a reference to that in a singleton, which is then used each time we want to format something, even in subinterpreters (to avoid looking it up each time). Could that be an issue? This definitely worked on 3.9, it's become an issue as we've tried to migrate to the stable ABI on 3.13.

[ZeroIntensity]

One thing which does occur to me, is that we are looking up format_exception in the default interpreter and storing a reference to that in a singleton, which is then used each time we want to format something, even in subinterpreters (to avoid looking it up each time).

Yup, that could cause this problem. You can't share objects between subinterpreters, at all. The interpreter will implicitly share some immortal objects for you (e.g., Py_True and Py_False), but don't worry about that. Objects have a lot of assumptions that aren't portable across interpreters. It might incidentally work on 3.9, but it's still wrong there. Interpreter isolation wasn't fully implemented until 3.12 (see PEP-684).

Basically, the exception is because each interpreter has its own copy of _abc, and thus each have their own copy of its types. The reference to format_exception has a type pointing to the main interpreter's copy, so when called in a subinterpreter, the type check fails, because it's expecting a pointer for its own copy of the type.

[matj-sag]

OK, thanks, I'll review all our usage of that. It certainly wasn't clear when we started working on this that wasn't allowed. I think we can move the cache within where we create each interpreter and maintain it per-interpreter instead. We'll give that a try and let you know

[matj-sag]

Right, that's solved the problem. Thanks for the insights all, you can feel free to close the issue