You can always hide someone
Rather than unfollowing or defriending someone, it might be best to avoid slighting people by making groups. Google Plus and Facebook allow you to put people in groups that don't receive all your messages and that you don't need to read all the posts from. Twitter allows you to make lists to read tweets from.
If you do this, you can connect without letting the person know too much about you. It also avoids that moment when people realize you have dropped them.