Both "that said" and "that being said" are common (possibly too common) and perfectly grammatical, and sufficiently formal as well. "Having said that" is also correct, but to be correct the subject in what follows must be whoever said that (usually "I"). For instance, you can say: Roses are usually red. That [being] said, they are also… More @Wikipedia
Hover over any link to get a description of the article. Please note that search keywords are sometimes hidden within the full article and don't appear in the description or title.