Course Project
Audio Classification using AudioSet
Rijul Dahiya1, Grishma Deshmukh1, Ansh Kumar Sharma1
Abstract
Audio classification is challenging due to the unstructured and diverse nature of sound events. In this study, we employ BEATs, a prominent self-supervised learning algorithm, to evaluate the effectiveness of large pre-trained models as feature extractors. We investigate various machine learning approaches, from simple models like logistic regression to advanced deep learning architectures such as GRU. We further compare performance across different BEATs layers as features and contrast these with traditional feature extraction techniques. Experiments on a subset of AudioSet (AS-20K) demonstrate that features from pre-trained models and deeper layers consistently outperform conventional methods.
Key Contributions
- Systematic evaluation of BEATs pre-trained features across multiple layers for audio classification.
- Comparison of logistic regression, SVM, GRU, and CNN classifiers on AS-20K.
- Demonstration that deeper BEATs layers provide significantly better representations.
- Analysis of SSL vs. traditional feature extraction (MFCC, Mel-spectrogram) performance gap.
Method
Audio clips are processed through the BEATs self-supervised model, and activations from various transformer layers are extracted as fixed features. Multiple classifiers, logistic regression, SVM, GRU, and CNN, are trained on these features. Traditional features (MFCC, Mel-spectrogram) serve as baselines. Evaluation is performed on AS-20K using mean average precision (mAP).
Results — feature layer
MAP by which BEATs layer supplies the features. Performance climbs sharply with depth: last-layer features are roughly 7× better than first-layer for the MLP and 60× for the LSTM.
| Layer | MLP | CNN | RNN | LSTM | GRU |
|---|---|---|---|---|---|
| First | 0.0544 | 0.0698 | 0.0107 | 0.0060 | 0.0128 |
| Middle | 0.1782 | 0.1466 | 0.0166 | 0.0507 | 0.1150 |
| Last | 0.3912 | 0.3320 | 0.1961 | 0.3572 | 0.3810 |
Results — classifier and reduction
- Handcrafted features (Mel spectrogram, MFCC, Chroma) did not converge at all on this task — the gap to pre-trained representations is categorical, not incremental.
- GRU reaches the best F1 (0.5143), followed by LSTM (0.5023); MLP takes the highest MAP (0.3924) thanks to PCA-based flattening.
- Reduction technique matters: PCA-flatten (MAP 0.3924 / F1 0.4882) beats mean (0.3912 / 0.4718) beats max (0.3186 / 0.3851).
- Tree-based classifiers collapse on high-dimensional sequential data — Random Forest manages F1 0.0536.
Figures
