Yevamos, Chapter Ten, Mishnah Ten. The Mishnah opens: "Pikachas ve'chereshes" - the case is of a man who was married to two women: one who was of full mental capacity, meaning an ordinary woman, and the other a chereshes, a deaf-mute. His marriage to the first woman is a full marriage by Torah law, while his marriage to the chereshes is only rabbinic.
Both women were widowed and fell before the yavam for yibum, and the Mishnah distinguishes between two cases according to the order of the relations.
The yavam first had relations with the woman of full capacity:
"Ba yavam al hapikachas ve'chazar u'va'al es hachereshes, o sheba achiv al hachereshes - lo pasal es hapikachas" - the yavam had relations with the woman of full capacity, which constitutes a complete yibum by Torah law, and afterward he had relations with the chereshes, or his brother had relations with the chereshes. This does not disqualify his bond with the woman of full capacity, since the first act of relations was a full and complete marriage and yibum, and that is what took effect. The bond with the chereshes is viewed as mere licentiousness, which carries no legal force.
The yavam first had relations with the chereshes:
If, however, the yavam first had relations with the chereshes - an act whose force is only rabbinic - and then went and had relations with the woman of full capacity, whose relations take effect by Torah law, or his brother afterward had relations with the woman of full capacity, he has disqualified the chereshes. These are the results:
The chereshes: His bond with her is disqualified, and she must leave with a get, because the woman of full capacity is his by Torah law.
The woman of full capacity: He cannot remain with her either, since his relations with the chereshes, which carry rabbinic force, came first; therefore she requires both a get and chalitzah.
In summary: everything depends on the order of the relations. When the relations with the woman of full capacity came first, which are by Torah law, the full yibum takes effect, and the relations with the chereshes that follow disqualify nothing. But when the relations with the chereshes came first, which are only rabbinic, and the relations with the woman of full capacity followed, the chereshes is disqualified and leaves with a get, and the woman of full capacity also requires a get and chalitzah.