I've long suspected that cell phone use impairs driving ability, whether or not the speaker uses a hands-free device. I've also suspected that a phone conversation distracts the brain in a way that passenger conversation does not. Here are some studies that support those beliefs.
The studies suggest that cell phone use interrupts attention, and it has nothing to do with having your hands on the wheel. The study also shows that listening is just as detrimental as talking, and the effect can linger after the conversation ends. Oddly, there is no similar distraction when listening to "books on tape" for instance, or chatting with a passenger.
I've "felt" this attention deficit when driving. I can just "tell" that I'm not paying as much attention when I'm on my phone. It's like some important part of my brain is focusing on "visualizing" the other party who isn't there. This "visualization" detracts from interpreting my real vision. My theory, anyway.