Structural monitoring of masonry buildings is crucial for ensuring safety, preserving historical heritage, and preventing structural damage. This process allows for the early detection of signs of deterioration or settlement, helping to plan maintenance and restoration interventions in a timely manner. Discriminating between structural damage and benign conditions using monitoring data is made difficult by the inherent unpredictability in climatic and operational variables, such as temperature and humidity, as well as by the complex constitutive behavior of masonry. In order to improve diagnostic reliability, this study integrates acceleration time histories with ambient temperature measurements to propose a strong anomaly detection framework for masonry structures. Convolutional Autoencoder (CAE), a deep learning model optimised for unsupervised anomaly detection, is used in the suggested method. By learning latent representations of thermal patterns and structural dynamics, the CAE makes it possible to distinguish between anomalies brought on by damage and typical seasonal variations. The framework uses Singular Value Decomposition (SVD) representations and extracts dominant spectral characteristics in order to handle the high dimensionality of time-domain accelerometer signals, which are computationally demanding and prone to noise. This condensed representation enhances model generalisation and makes training more effective. The use of reconstruction-based metrics on dominant SV distributions in conjunction with temperature-informed error analysis is a fundamental component of the framework. This combined spectral-thermal method keeps robustness against environmental noise while improving sensitivity to structural changes. Real-world data from the Consoli Palace in Gubbio, Italy—a historic masonry building outfitted with accelerometers and thermocouples—is used to validate the framework. The anomaly detection framework performs well in identifying damage after seismic events, demonstrating high accuracy and adaptability.