If you have a table in a publication you won't be able to rename the table, unless you drop the publication, rename the table, and then recreate the publication.

To know if the table is in any publication, execute SELECT article FROM Distribution.dbo.MSArticles WHERE Article = 'yourTable'

from the distributor.