What, you couldn't return unless you had forgiven him?

[No, he doesn't see the issue of still hating or not forgiving your father's guts yet be civil and such with the rest of your family.]

From the way you talk it's as if he's the only one who wronged you anyway. Unless the rest of the family defended his actions I don't see how you could get angry with them as a whole.