Some believe it's the grammar, but I have to say that the hardest part about Arabic is how different the pronunciation is from English. There are just so any difficult letters that give learners problems. In other languages you might be able to just ignore the strange letters, but in Arabic if you don't get them right then people are going to have a hard time understanding you at the very least. At times they won't understand a thing you are saying.
I knew a guy who spoke Arabic with a southern US accent and he always got really angry at the teachers because he thought they were picking on him, but in reality they (and the rest of the students) just couldn't understand anything he said. I've also known people who can understand just about everything in Arabic, but when they try to speak no one can understand them. Pronunciation is a very very important part of the Arabic language.