Is saying you're welcome improper?

I think it is because when you say your welcome, it means that you are telling someone that they are welcome to do something as in you're welcome to visit anytime. But if you say it in response to thank you then what are you welcoming them to? Ive head that your welcoming them ask you another favor but what if your not? What would you say?