LATERAL THINKING

A man lives on the 14th floor of a building. Every day he takes the elevator to go down to the ground floor to go outside. When he returns he takes the elevator to the 10th floor and walks up the remaining 4 floors to reach his apartment on the tenth floor. But if there is someone else with him in the elevator he goes directly to his floor. Why?