Can You Turn Down a Dinner Invitation From the Boss?

When the boss invites you to a team dinner at her house, is saying no a career-killer?

Source: Inc Magazine