AI for Cutural Heritage
Teaching a Machine to Listen: PICUS and Artificial Intelligence for the Diagnosis of Detachments in Wall Paintings
For centuries, the most reliable way to find out whether a fresco is quietly coming away from its wall has been to tap it and listen. Where the plaster layers have separated, a thin cushion of air is trapped between them, and the surface answers with an unmistakable hollow sound. A trained conservator moves across the wall, taps, listens, and marks the areas that need to be consolidated.
It is a remarkable skill, and it has an equally remarkable limitation: it lives entirely inside one person's ear. Two conservators may hear the same wall differently. Sound zones judged by one operator as sound may be classified as defective by another — false positives and false negatives that translate directly into money spent where it was not needed, or damage left where it will grow. Transcribing what has been heard onto a map is a further step open to interpretation, and therefore hard to repeat. On large surfaces the work is also physically punishing, and fatigue does not improve anyone's hearing.
Our research group at the Department of Civil, Computer Science and Aeronautical Technologies Engineering of Roma Tre University has spent several years turning this expert gesture into a measurement: an instrument that taps with a controlled, repeatable force, records what the surface answers, and a set of neural networks trained to interpret that answer the way an experienced restorer would.
PICUS: an instrument that taps and listens
PICUS is a portable, low-cost electro-acoustic device. An electro-mechanical percussion element gently strikes the surface under examination; the sound produced is converted into a voltage signal by a microphone and digitised by an inexpensive Arduino-class board. The analogue signal is sampled at 48 kHz, and the system stores the short window that contains the complete acoustic response of the stimulated point.
Two design choices matter as much as the electronics. First, the force applied by the percussor can be calibrated, so the instrument can be adjusted to the specific surface being analysed and never applies more energy than the restorer's own gentle touch. Second, an on-board infrared camera detects an infrared LED that illuminates the scene, allowing the position of the probe to be tracked — which is what makes it possible to turn a sequence of individual taps into a map.
PICUS is intended for use by professional restorers. Operating it safely on a real painted surface requires exactly the expertise it is designed to support, not to replace.
The PICUS probe: percussion element, microphone and infrared tracking camera in a hand-held unit.
The test object: a fresco with known defects
Supervised learning needs ground truth, and a real fresco cannot supply it: nobody knows exactly where its detachments begin and end. The team therefore built a mock-up reproducing a stratified antique plaster, following historical and direct sources on the art of plastering.
- The support is a handmade brick tile, 50 × 50 × 7.5 cm.
- A 3 cm layer of arriccio — lime, pozzolana and river sand — was applied over it.
- Four circular cavities with sharp edges and known profiles, 13, 10, 6 and 8 cm in diameter, were created in the overlying intonaco at depths of 2.5, 2, 2.5 and 2 cm.
- A 1 cm finishing layer of finer mortar, the intonachino, closed the stratigraphy.
Around the largest defect, a set of concentric circles was laid out so that the severity of the detachment could be sampled gradually rather than as a yes-or-no condition. Eight classes were defined and labelled by conservation experts: C0 for defect-free material, and C1 to C7 for increasing depth of the defect from the edge towards the centre. One hundred acoustic samples were acquired for each class.
The test object and the concentric arrangement of classes C0–C7 around defect A.
Three steps of research
1. Learning the sound of a defect
The first study fed the raw acoustic waveforms directly into a one-dimensional convolutional neural network (CNN1D), designed specifically for this task. Working in the time domain, the network learns its own filters and extracts the latent features that distinguish one class from another, without any hand-crafted feature engineering — and therefore without the biases that manual feature selection introduces.
Because the dataset was small, the acoustic arrays were augmented by circularly shifting each input vector ten times, producing temporally displaced versions of the original signals and reducing overfitting. The model was validated with a five-fold cross-validation.
- In the eight-class task — classifying how severe the detachment is — the network reached an average accuracy of 82% (± 2%).
- Almost all residual errors fell between physically contiguous classes, such as C5 and C6, which correspond to adjacent rings only millimetres apart on the real object. Confusions between distant classes were rare.
- The defect-free class C0 was recognised with precision and recall close to 1.00 — the single most important result in practice, because it is the boundary between "intervene" and "leave alone".
That observation led to a second, binary network trained on 1,500 acquisitions (700 sound, 800 defective) to answer the simpler question: defect or no defect. It reached 99% (± 1%) on the test area.
2. From single points to a map of the wall
A diagnosis is only useful if it can be drawn. The binary network was applied to the whole test object, divided into a 16 × 16 grid — 256 acquisitions — and each classified point was mapped onto its acquisition coordinates to generate a colour map of the surface.
The significance of this step is easy to miss: the network had been trained on a single defect, and was then asked to inspect regions of the object it had never seen. It correctly identified the defect used for training and two of the remaining three. The fourth was only partially recognised, an outcome attributable to an internal collapse of that cavity, partly caused by the laboratory conditions that affected the setting and curing of the mortars. Superimposed on the traditional auscultation map drawn by the restorer, the machine-generated map showed consistent detachment areas.
Left: raw network output over the acquisition grid. Right: the interpolated defect map superimposed on the conservator's traditional auscultation map.
3. Looking at sound: spectrograms and a 2D network
The second study asked whether the acoustic signals carry information that a purely temporal analysis leaves on the table. Each acquisition was converted into a spectrogram through a short-time Fourier transform, computed with a 256-point window and a hop length of 32 samples, producing a picture of how the frequency content of the tap evolves over time.
Spectrograms are, structurally, images — which makes the whole toolkit of image recognition available. A purpose-built two-dimensional convolutional network (CNN2D) with three convolutional blocks of 128, 256 and 512 filters was trained on 800 acquisitions, 640 for training and 160 for evaluation on unseen data.
- Multi-class accuracy rose to 93%, a clear improvement over the time-domain approach on the same eight classes.
- Several classes were classified without a single error; the residual confusion again concentrated on the adjacent C5 and C6 rings.
- Just as importantly, the spectrogram makes the data legible to a human expert. Where a raw waveform is opaque, a time–frequency image shows where the energy of the tap sits — opening the way to explainable diagnostics that a conservation scientist can check against the physics of the material.
The same taps in the time domain (above) and as spectrograms (below): variations that are invisible in the waveform become structure in the time–frequency image.
4. Moving to a new surface: transfer learning
Every wall is different. A model painstakingly trained on one plaster stratigraphy is of limited use if it has to be retrained from scratch — with thousands of expert-labelled samples — for every new surface. The third study addressed this directly.
Two planar test objects were prepared, both mounted on a 200 mm aluminium cube used as a rigid backing, both containing the same cylindrical cavities (50 mm and 25 mm radius) bonded with an epoxy resin to simulate debonding. The first was a 3 mm plate of PMMA; the second a 1 mm plate of PVC — a different material with a different thickness, and therefore a substantially different acoustic behaviour. Each was scanned on a dense 40 × 40 grid, for 1,600 measurements per object.
- Trained on the first object, the network classified its defective and sound regions with 96% accuracy.
- Applied without modification to the second, unseen object, it still reached 88% precision on the defect class — evidence that what the network had learned was not tied to one specimen.
- Fine-tuning only the final layers on 160 samples from the second object — 10% of its data, with all earlier layers frozen — raised precision on the defect class to 95%, with an overall accuracy of 98%.
- A control experiment confirmed that this small subset would not have been enough to train a model from scratch: the knowledge transferred from the first surface is what makes the adaptation possible.
In operational terms, this is the difference between a laboratory result and a working instrument. A restorer arriving at a new site would need to label only a small number of points before the system adapts to that particular surface.
What this changes
Taken together, the three studies describe a complete chain — from a controlled tap, to a classified signal, to a map of the surface — that addresses the weaknesses of the traditional procedure without discarding the expertise behind it.
- Repeatability. The excitation force is standardised and the interpretation criteria are fixed, so two inspections of the same wall are comparable.
- Objectivity. The method remains human-operated but is far less exposed to individual bias and to the boundary conditions of a given working day.
- Scale. Large surfaces can be scanned systematically rather than sampled by a tiring operator, and the results are recorded rather than transcribed.
- Monitoring over time. Because every acquisition is stored with its coordinates, the conservation state of a surface can be documented and compared across campaigns — detecting evolution, not only presence.
- Better-targeted intervention. Accurate defect maps support informed decisions about where consolidation is actually required, with direct consequences for planning and for the use of restoration funds.
Where the research goes next
Three directions are already open. The first is explainability: techniques such as saliency maps and Grad-CAM would show which portions of a recording drive a classification, allowing experts to validate the machine's verdict against the physical and material properties of the stratigraphy rather than trusting it blindly. The second is architectural: expanding the datasets and comparing the present networks with alternative models, including Vision Transformers. The third is deployment — a field-capable device able to fine-tune its own network on site through transfer learning, extending the method to further materials and more complex defect patterns.
Beyond conservation, the same acoustic and algorithmic chain applies wherever bonded layers can separate: the polymeric specimens used in the transfer learning study point towards structural and materials health monitoring more generally.
The tap test is one of the oldest diagnostic gestures in the conservation of built heritage. What this research adds is not a replacement for the restorer's ear, but a way of making what that ear knows measurable, repeatable, and shareable — so that the judgement of experienced conservators can be applied consistently across surfaces far larger than any single professional could examine alone.
Publications
- G. Caliano, F. Mariani, P. Calicchia, "PICUS: a pocket-sized system for simple and fast non-destructive evaluation of the detachments in ancient artifacts", Applied Sciences, 11(8), 3382, 2021. DOI: 10.3390/app11083382
- M. Lo Giudice, F. Mariani, G. Caliano, A. Salvini, "Deep learning for the detection and classification of adhesion defects in antique plaster layers", Journal of Cultural Heritage, 69, 78–85, 2024. DOI: 10.1016/j.culher.2024.07.012
- M. Lo Giudice, F. Mariani, G. Caliano, F. C. Morabito, A. Salvini, "Automated Detection of Defects in Antique Plaster Using Spectrograms and Deep Convolutional Neural Networks", in: Esposito, A., Faundez-Zanuy, M., Morabito, F.C., Pasero, E., Cordasco, G. (eds) Neural Networks: Overview of Current Theories and Applications. Smart Innovation, Systems and Technologies, vol 459. Springer, Singapore. https://doi.org/10.1007/978-981-95-4072-3_24.
- M. Lo Giudice, F. Mariani, G. Caliano, A. Salvini, "Enhancing Defect Detection on Surfaces Using Transfer Learning and Acoustic Non-Destructive Testing", Information, 16(7), 516, 2025. DOI: 10.3390/info16070516