Confederate Nugent

This morning we ran a story on Ted Nugent's appearance at Gov. McDreamy's inaugural ball early this week.
Nugent showed up wearing a cut-off T-Shirt with a Confederate flag on the back.
Gary Bledsoe, president of the National Association for the Advancement of Colored People of Texas, said the Confederate battle flag is never appropriate.
“Whenever someone sports the Confederate battle flag, many Texans will be offended, and rightly so, because of what it symbolizes the enslavement of African-Americans and more recently the symbol of hate groups and terrorists,” Bledsoe said.
Perry's spokesman Robert Black said the governor would never try to squelch anyone's freedom of speech.
Locally, Ellis County Republican Chairman Rusty Ballard said he didn't have a problem with Nugent playing.
“He believes in many of the conservative issues the Republican Party does," Ballard said. "I thought it was a great deal having him play. Nugent is a great supporter of the governor.”
Ballard said he also had no issue with Nugent’s use of the Confederate battle flag.
“The flag is a part of Texas’ history and it doesn’t represent what a lot of people have come to believe that it does,” Ballard said. “You can’t try to restrict people’s freedom of expression – especially artists. I don’t think there was any political statement being made, it was just typical Ted Nugent.”
I don't have a problem of granting free speech as long as you're not going to harp on someone else's right to free speech when it offends you.
Just this week I read about people getting up in arms when the F-word was shown on TV, or when Howard Stern says something offensive but if there are threats of the government censoring them they claim freedom of speech as well.
Where do you draw the line? Can you draw the line and still grant freedom of speech?