Through programmable self-assembly, simple building blocks can be made to form highly complex structures following local rules of interaction. However, materials systems that are most commonly utilized for programmable assembly often lack interactions that exhibit the strength, specificity, and long ranges, which would, as a result, allow for robust and rapid hierarchical self-assembly processes. “Magnetic handshake” building blocks resolve many of these challenges at once, incorporating strong, long-range, and specific magnetic interactions through patterning of magnetic dipoles onto rigid panels. When appropriately designed, the panels organize hierarchically: first into chains, and subsequently those chains combine to form dense stacks. Differences in phase behavior and morphology was examined for four panel types. It was delineated how perpendicular chaining and stacking interactions between panels compete and how they can be manipulated to reverse the sequence of the hierarchical assembly pathway. Collectively, this work shows the enormous potential for using magnetic handshake materials for self-assembly of hierarchically organized complex structures.