In a study of 135 mothers and their infants, researchers at Harvard Medical School in Boston found that the greater a woman's fish intake during the second trimester, the better her child performed on a standard test of mental development at six months.
But when mothers had high mercury levels, their babies tended to have poorer test scores.
The findings underscore the need to avoid fish with high mercury levels.
Writing in this month's issue of Environmental Health Perspectives (vol 113, no 10), the researchers say that "women should continue to eat fish during pregnancy but choose varieties with lower mercury contamination".
The study gathered data on maternal fish intake during pregnancy and tested for mercury levels in hair samples obtained around the time of delivery.
Infant cognition was assessed by visual recognition memory (VRM) testing at six months of age.
After adjustment for hair mercury level, each additional weekly fish serving was associated with a four point increase in VRM score, said the researchers.
However, an increase of 1ppm in mercury was associated with a decrement in VRM score of 7.5 points.
VRM scores were highest among infants of women who consumed more than two weekly fish servings but had mercury levels of 1.2 ppm or less.
Other studies have already underlined the benefits of fish, as well as one of its key nutrients - omega-3 fatty acids - on the mental development of children.
In a study reported last year, women who ate fish regularly during pregnancy had children with better language and communication skills by the age of 18 months.
Further studies have looked at how raising intake of DHA, an omega-3 fatty acid, in the maternal diet can improve a baby's learning.
But the new findings show that not all fish meals can offer the same benefits. Varieties that are less likely to be contaminated with mercury such as light tuna or salmon may be better for infants' brains than swordfish or albacore tuna that is more often contaminated.