No one denies that Muslims have a right to build the Mosque a few blocks away from the former site of the World Trade Center. The reason people object is that they feel it’s insensitive to those who lost loved ones in the attack and, more broadly, to Americans in general.
What I find ironic about this is that it’s almost the exact same debate that people have periodically over the Confederate flag. You have a right to fly it, but it can be construed as an insult to the descendants of slaves. (And, I might add, to those whose ancestors fought for the Union.) Sure, the mosque is a religious symbol; but at bottom the issue is not a religious one so much as it is a cultural one.
The only difference is that the people likely to support the flying the Confederate flag oppose the mosque, and those who oppose that flag support the mosque.