The human visual system is fine-tuned to detect not only individuals’ physical features (e.g., faces and eyes), but also complex social group interactions. For example, recent research has shown that in visual search tasks, humans are faster at locating groups of two (dyads) when they face one another versus when they face away. However, real-world scenarios often involve groups larger than two, with humans usually congregating in groups of four or less. Here we investigated if visual search advantage also exists for such larger groups, and if this advantage is constrained by group size. Participants searched for a facing or a non-facing group (among non-facing and facing distractors, respectively). Group size varied from two to eight individuals. Overall, participants found facing groups faster than non-facing ones. However, this search advantage was modulated by group size such that it was greater for groups of two and three individuals and diminished for larger groups. This suggests that the perceptual advantage for group interactions is not specific to dyads, but extends to larger groups, and that there is a group size constraint on this advantage. Thus, there may be a correspondence between our perceptual system’s tuning to detect social interactions, and our social system’s preference for congregating in groups of four or less.