Massachusetts has the most expensive family health insurance premiums in the country, according to a new analysis that highlights the state’s challenge in trying to rein in medical costs after passage of a landmark 2006 law that mandated coverage for nearly everyone.