If you're always inviting him to the family gatherings, and he never wants to go with you, this is a sign that he's hiding something. How could you claim this to be a relationship if he doesn't understand your background? How will he look into the eyes' of your father, if he's an unemployed brother with no job? What if your father was to ask, "How are you treating my daughter?" What do you think his answer would be? Of course, he's going to lie and give the answer that he thinks is best, but it's far from the truth.
Maybe he's embarassed of his financial problems, and doesn't want to be around your loved ones. This will cause hinder to his self esteem, and he'll begin to feel less of a man. His male ego will get in the way of a healthy communication, it also depends on the type of relationship he has with your parents. Be honest with yourself, and see the picture for what it is. Under circumstances, he may think poorly of your family. It will cause him to not want to engage in any dealings with your people simply because he doesn't love you as much as you believe he should.
To continue to keep living as a lie is easy, but the hard part is facing your reality. Have you ever considered that just maybe you're not the woman he's destined to marry? If you don't hold the title to his heart, it should be obvious. Do you believe you're worthy of having a real connection with someone? If so, why remain questionable of what you're feeling inside? Release yourself, keep it real, don't try to force anything that's not meant to be.