In July 1968, ethologist John B. Calhoun built a “mouse utopia,” a metal enclosure 9 feet square with unlimited food, water, and nesting material. He introduced four pairs of mice, and within a year they had multiplied to 620. But after that the society began to fall apart — males became aggressive, females began neglecting their young, and the weaker mice were crowded to the center of the pen, where resources were scarce. After 600 days the females stopped reproducing and the males withdrew from them entirely, and by January 1973 the whole colony was dead. Even when the population had returned to its former levels, the mice’s behavior had remained permanently changed.
I think that this was also done with rats. That one found that a some point in over crowding the males started having sex with each other.