From the Contact Detail record:

Tap to the contact from the Contact List or use the search feature to find a contact. To add or remove the contact, tap “Edit” and add or remove Orbits on the Orbit ribbon.

From the Contact List

Swipe right to select the contacts to include in the orbit. Tap Add Contacts to Orbit and choose the orbit to which you want to add the contacts.

From the Orbit View

Select an orbit and tap “Edit”. Select or unselect contacts by tapping and tap “Done” to save.

Alternatively, to remove contacts from an Orbit you can right swipe contacts within the Orbit and tap the trashcan icon in the action bar. This will NOT delete the contact. It will simply remove the contact from the Orbit.