Not a physicist either, but one practical problem is the speed of light. Time synchronization is likely to become very tricky if you have communicate with time sources that are hundreds of light years away.