But isn't that really what lets them do this? In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions.

As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste.

Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads.

If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.

The irony is that some C extensions exist because they have better performance than Python.

On the other hand, there are many C extensions which are just interop/wrappers for existing C code. I think no language is naive enough to think they can get away without C interop.

C extensions are actually pretty nice because you can wrap the C code into idiomatic Python. With ctypes, you have to e.g. maintain two structure definitions, which is very problematic for some codebases.

> In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions.

AFAIK, many projects initially struggle to even achieve performance parity with CPython. The GIL isn't evil, it's a simple solution to a hard problem. But at this point, a complex solution to a hard problem is better if it's faster, and some people care more about speed than C interop (and vice versa).

What would be awesome is a Python runtime that could run with fine-grain locking until a C extension is loaded, and then continue with coarse locking. But the problem is still there's no upgrade path for C interop.

The only thing I can think of is using type annotations and something like Cython's cdef to write Python-implementation-independent C interop that doesn't suck as bad as ctypes. Then the Python rumtime could also lock the arguments at a very fine level while the function is being called.