Kiddushin, Chapter 4, Mishnah 4. Our Mishnah discusses a kohen who comes to marry a woman about whom a rumor or whisper has spread that there is some flaw in her family. In such a case there is a need to investigate the family background, to a certain extent. But as long as no one has cast aspersions on the family, there is no reason for concern, and we assume they are of fit lineage.
The text of the Mishnah:
"Hanosei ishah kohenes - tzarich livdok achareha arba imahos shehen shemoneh" - a kohen who comes to marry a woman who is the daughter of a kohen: even though she is a kohen's daughter, if a challenge was raised about a possible problem in the family, he must investigate her lineage, four mothers in her background, which are really eight. At the end of the Mishnah we will see the law when she is not the daughter of a kohen.
Why is it specifically the mothers who are investigated?
The Gemara explains that there is no need to investigate the fathers, only the mothers. The reason: when men quarrel with one another and want to say ugly things about each other, they raise the issue of tainted lineage and mention the blemishes in each other's ancestry, and so the matter comes to light on its own. When women quarrel with one another, however, they accuse each other of being unfaithful to their husbands, and a blemish of lineage does not come to light. For this reason it is necessary to investigate the mothers.
The four mothers that are eight:
"Imah" - the woman's mother, "ve'em imah" - and her mother's mother.
"Em avi imah" - the mother of her mother's father, "ve'imah" - and that mother's mother.
"Em aviha" - her father's mother, "ve'imah" - and that mother's mother.
"Em avi aviha" - the mother of her father's father, "ve'imah" - and her mother.
These, then, are four groups, eight mothers in all.
The daughter of a Levi and the daughter of a Yisrael:
If the young woman is not the daughter of a kohen but comes from a family of Levi or of Yisrael, "mosifin aleihen od achas" - one more is added to them. That is, in each of the groups an additional generation is added: for example, after her mother and her mother's mother, her mother's mother's mother is added as well. This addition is made in each of the four groups, so that twelve mothers are investigated in all, instead of eight.
In summary: in this Mishnah we learned that a kohen marrying a woman about whose family lineage a challenge has been raised must investigate her background; that the investigation is made specifically among the mothers, because a blemish of lineage does not come to light in a quarrel between women; and that the count is four mothers that are eight in the case of a kohen's daughter, while for the daughter of a Levi or a Yisrael an additional generation is added in each group, up to twelve.