Assembling 3D objects from primitive bricks is challenging due to complex constraints and numerous possible combinations. Recent studies have demonstrated promising results on sequential LEGO brick assembly by graph modeling. However, existing approaches are class-specific and require significant computational and 3D annotation resources. In this work, we first propose a computationally efficient breadth-first search (BFS) LEGO-Tree structure to model sequential assembly actions. Based on the LEGO-Tree structure, we then design a class-agnostic tree-transformer framework to predict assembly actions from multi-view images. A major challenge is the costly acquisition of step-wise action labels. We address this by leveraging synthetic-to-real transfer learning. Specifically, our model pre-trains on synthetic data with full action label supervision. We further circumvent the requirement for real data action labels by introducing an action-to-silhouette projection for self-supervision. With no real data annotation, our model outperforms existing methods with 3D supervision by 7.8% and 11.3% in mIoU on the MNIST and ModelNet Construction datasets, respectively.