Sure, the confederate flag is a symbol of racism for a lot of people. It should be taken down for that reason. But few people have asked a more basic question: didn't the confederacy lose?
Since when does a country allow its opposition to fly their flag on their property? A flag is not just a symbol of ideologies and purpose, but of entitlement and ownership. When the confederate rebels that tried to leave the United States lost the war, they lost their entitlement and ownership over the property of the United States of America.