If I'm being actively targeted that's a completely different threat model to a potential passive mass dragnet using 'home assistants'.

Someone could activate my phone microphone remotely and record me but if I'm up against that level of adversary I'm probably in deep trouble.

How is a home speaker any different? you can monitor your network traffic and see how much data it's sending home right now. It's definitely not a constant audio stream. You'd have to be targeted.