Salesforce does this with a combination of web components, with a patched up ShadowRoot so that code with a reference to the shadow root can't walk into the rest of the document, and a secure evaluator function related to SES (Secure EcmaScript) to limit the globals the untrusted script has access too.
Salesforce sandboxing is too easy to escape. Last time I needed to implement some feature for Salesforce, I've encountered 4 different escapes. It was also horrible dev experience.
You can also check out the discussion for Figma's earlier work on their plugin system, which is what inspired jitl (above) to create quickjs-emscripten. Previously:
The secure evaluator is wild. I think this is the heart of it: https://github.com/Agoric/realms-shim/blob/v1.1.0/src/evalua...
There's also an idea for isolated web components to solve this in the platform: https://github.com/WICG/webcomponents/issues/1002