I believe that the fear this could restrict freedom of speech is legitimate. Statistics show the top 2 uses of the Internet consist of porn and shopping. Whether you like this or not, condone it or not, its the truth! So, let us assume this .xxx top-level domain is created. Back in the days before silly top-level domains like .info and .me, you could only get .org, .net., .com, or your country-code.
Today, there have been many new top-level domains added for commercial purposes, and it is clearly assumed that .com,.net.,and .org will eventually run out. However, very few websites bother to use .info and so forth, and even search engines do not take these domain names as seriously as a .com. Everyone seems to know that you need a .com, .net. or .org to get exposure for your site, even if it contains an unimaingeable treasure trove of useful information.
The problem here, however is, that by making the .xxx domain, we specifically now know that this is for pornographic content. And so, any Internet Service Provider around the world, if it even takes off, could arbitrarily block any requests to domains that fall in the .xxx top-level domain range. This could, hypothetically, result in increased censorship, especially in many authoritarian countries.
Many content guardian tools already exist to prevent this content from being found, and entire rating systems have been created to monitor pornographic websites. It is not fair to throw an entire industry into a category in such a way. For example, I am almost entirely certain that if ".computer" was created, we would not use it, nor would most websites. Its just silly. Everyone still knows the .COM's are where it is at.
Now, decades from now, I could be wrong. Maybe it will become useful. But noone that I know of is registering .info domains, .jobs, .travel, .coop, etc. I bet many people reading this never knew these top-levels existed, because even I didn't. I had to look them up on ICANN's website. And if you register a domain that end with this at the top level, you are usually offered an enormous discount, because noone, at this time, truly takes them seriously. It will be hard to get listed in DMOZ (A human reviewed web directory that is used by almost all the major search engines to index quality websites) or for anyone to take your website seriously.
Recently ".co" was created for Columbia, and this did not require proof of residency or business registration in Columbia. Thus, this domain is now used for people who have a broken M key on their keyboard. For instance, even we have windows7forums.co in case someone forgets the "M" at the end. So that is the only use we see it for - bringing people who spelt our domain name wrong to the right website.
As more governments push to firewall their country from the rest of the Internet whenever there is civil unrest, I believe there should be serious concern about censorship in this matter. Censorship is appropriate when individuals, who are capable of making intelligent, informed decisions, are the ones doing it. On the Internet, it is not appropriate when governments or large corporations make the call.
I believe the potential for abuse in .XXX is very grave, and thus, most serious content providers of pornography will avoid the domain, fearing censorship from ISPs and authoritarian governments.