Can merit badge counselors coach their own sons or close relatives (for instance, a nephew)?

Yes, but only if the young man is part of a group of Scouts who are all working on the same merit badge. Approved counselors may coach any Boy Scout who contacts them through the proper procedures.