That's exactly what I do and in fact I've had very few problems with Home Assistant in this setup. I just don't like the idea of needing the VM to be functional 100% of the time for my lights to work. Maybe it's just a mental block or me being old fashioned or whatever, I just think things like lighting should be able to work the old fashioned way in additional to cool home automation stuff.