NEW YORK (Reuters) - Nasdaq OMX Group recently talked with private equity firm Carlyle Group about taking the trans-Atlantic exchange operator private, but the talks broke down because of price disagreements, according to people familiar with the matter.