When encountering guys with tough guy reputations of their own (William Smith, Richard Egan, and now Bob Mitchum), Charlie backed down when put on the spot. Part of this might be the realization that not only were these men not going to back down from Charlie's tough talk, but that physically they were all larger and knew how to fight. More @Wikipedia
Hover over any link to get a description of the article. Please note that search keywords are sometimes hidden within the full article and don't appear in the description or title.