Timber structures have grown in popularity in recent decades as a substitute for steel and concrete because they reduce greenhouse gas emissions. Cross-laminated timber (CLT) is the primary material for the mass timber structure. On the other hand, CLT is a lightweight material. Due to this, high-rise timber structures have experienced tremendous vibration due to external dynamic loads such as wind, earthquake, etc. An outrigger structure is a viable solution to reduce the dynamic force induced vibration of a tall timber structure as it reduces structural deformation by increasing the lateral resistance of the entire structure. It is seen that as the outrigger is connected in between the core of the structure and the perimeter column, the perimeter column experiences a huge load demand while bending. As a result, the size of the column increases, which results in an increase in the construction costs. To reduce the load demand in the column, a shape memory alloy (SMA) based outrigger is proposed in this study. As the proposed controller system is a passive system, it needs to be designed before installation. In this viewpoint, the tuning parameters of the SMA and the location of the outrigger are designed using a reliability-based design optimization. The probability of failure of the combined system is estimated using the probability density evolution method (PDEM). The proposed framework for optimization is computationally expensive. To reduce the computational burden, a combination of autoencoder (deep learning) and Gaussian process regression (GPR) is used as a surrogate model. A moving trust region is also employed to ensure the accuracy of the surrogate model. To solve the reliability-based optimization, a two-step algorithm is employed in this study. Firstly, a feasible search direction is computed using Karush-Kuhn-Tucker conditions, and finally, a line search technique is used to increase the speed of the convergence of the optimization. The numerical results show the effectiveness of the proposed outrigger system to reduce the structural vibration of the tall timber structure.