You did the right thing. Better to leave the interview on good terms and turn the job down than make a scene yourself and risk burning a future bridge (you never know when you might run into these people again).