FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing

Overview

FactorizePhys introduces a breakthrough in remote photoplethysmography (rPPG) through the novel Factorized Self-Attention Module (FSAM), which leverages nonnegative matrix factorization to jointly compute multidimensional attention across spatial, temporal, and channel dimensions. This work demonstrates that matrix factorization can effectively serve as a multidimensional attention mechanism, outperforming existing state-of-the-art methods, including transformer-based approaches, for remote physiological sensing.

Figure 1: Our poster at NeurIPS 2024, where we discussed FSAM with fellow researchers from the computer vision and machine learning community.

Technical Details

The Compression-as-Attention Paradigm

Using compression as an attention mechanism isn't new. In the CNN-dominated era, squeeze-and-excitation (SE) attention [2] was among the most popular mechanisms. However, a fundamental limitation emerges when working with multidimensional feature spaces: existing attention mechanisms compute attention disjointly across spatial, temporal, and channel dimensions.

For tasks like rPPG estimation that require joint modeling of these dimensions, squeezing individual dimensions can result in information loss, causing learned attention to miss comprehensive multidimensional feature relationships.

FSAM: Factorized Self-Attention Module

FSAM uses Non-negative Matrix Factorization (NMF) [3] to factorize multidimensional feature space into a low-rank approximation, serving as a compressed representation that preserves interdependencies across all dimensions.

Figure 2: Overview of the Factorized Self-Attention Module (FSAM) showing how multidimensional voxel embeddings are transformed into a 2D matrix, factorized using NMF, and reconstructed to provide attention weights.

Figure 3: FactorizePhys Architecture. (A) Proposed FactorizePhys with FSAM; (B) FSAM Adapted for EfficientPhys showing 3D-CNN architecture and 2D-CNN adaptation.

The NMF Attention Mechanism

The factorization process uses iterative multiplicative updates:

def fsam(E):
    S = 1
    rank = 1
    MD_STEPS = 4
    ε = 1e-4
    batch, channel_dim, temporal_dim, alpha, beta = E.shape
    spatial_dim = alpha * beta

    # Preprocessing: ensure non-negativity for NMF
    x = ReLU(Conv3D(E - E.min()))

    # Transform to factorization matrix
    V_st = reshape(x, (batch * S, temporal_dim, spatial_dim * channel_dim))

    # Initialize bases and coefficients
    bases = torch.ones(batch * S, temporal_dim, rank)
    coef = softmax(V_st.transpose(-2, -1) @ bases)

    # Iterative multiplicative updates (4-8 steps)
    for step in range(MD_STEPS):
        # Update coefficients
        numerator = V_st.transpose(-2, -1) @ bases
        denominator = coef @ (bases.transpose(-2, -1) @ bases)
        coef = coef * (numerator / (denominator + ε))
        
        # Update bases
        numerator = V_st @ coef
        denominator = bases @ (coef.transpose(-2, -1) @ coef)
        bases = bases * (numerator / (denominator + ε))

    # Reconstruct attention
    V̂_st = bases @ coef.transpose(-2, -1)
    Ê = reshape(V̂_st, (batch, channel_dim, temporal_dim, alpha, beta))

    # Apply attention with residual connection
    output = E + InstanceNorm(E * ReLU(Conv3D(Ê)))
    
    return output

Why Rank-1 Factorization Works

The paper's ablation studies confirm that rank-1 factorization performs optimally for rPPG estimation. This aligns with the physiological assumption that there's a single underlying blood volume pulse signal across different facial regions.

Table 1: Ablation study results showing performance across different factorization ranks. Rank-1 achieves optimal performance, supporting the single signal source assumption for rPPG estimation.

Performance Analysis

Why FSAM Outperforms Transformers

1. Task-Specific vs Generic Design

Transformers use generic self-attention that treats all positions equally:

FSAM is specifically designed for spatial-temporal signal extraction:

• Temporal vectors as the primary dimension (signals evolve over time, directly supervised through loss function)
• Spatial-channel features as descriptors (different facial regions contribute differently)
• Rank-1 constraint enforces single signal source assumption

2. Computational Efficiency

Cross-Dataset Generalization Performance

FactorizePhys demonstrates superior performance across all evaluation metrics when tested on unseen datasets:

Table 2: Cross-dataset Performance Evaluation. Comprehensive comparison showing FactorizePhys outperforming SOTA methods across all datasets with MAE, RMSE, MAPE, Correlation, SNR, and MACC metrics.

Performance vs. Latency Analysis

Figure 4: Performance vs. Latency. Cumulative cross-dataset performance (MAE) vs. latency plot with model parameters as sphere size, showing FactorizePhys achieving best performance with fewer parameters.

Attention Visualization

In this work, we introduced an intuitive visualization of learned attention features, by computing cosine similarity between temporal embeddings and ground-truth PPG signals. In the below figure, higher correlation scores and better spatial selectivity can be observed for FactorizePhys trained with FSAM.

Figure 5: Attention visualization comparing baseline model (left) and FactorizePhys with FSAM (right). Higher cosine similarity scores (brighter regions) indicate better spatial selectivity for rPPG-rich facial regions.

The Inference-Time Advantage

A remarkable finding: FactorizePhys trained with FSAM retains performance even when FSAM is dropped during inference. This suggests FSAM enhances saliency of relevant features during training, guiding the network to learn such salient feature representations that persist without the attention module.

# Training: FSAM influences 3D convolutional kernels
factorized_embeddings = fsam(voxel_embeddings)
loss = compute_loss(head(factorized_embeddings), ground_truth)

# Inference: Can drop FSAM without performance loss
output = head(voxel_embeddings)  # No FSAM needed!

This dramatically reduces inference latency while maintaining accuracy - ideal for real-time applications.

Key Contributions and Broader Implications

For the rPPG Community:

• First matrix factorization-based attention specifically designed for physiological signal extraction
• State-of-the-art cross-dataset generalization with dramatically fewer parameters
• Real-time deployment capability without performance degradation

For the Broader AI Community:

• Novel attention paradigm demonstrating that domain-specific designs can outperform generic mechanisms
• Efficiency breakthrough: 138× parameter reduction compared to transformers with superior performance
• New perspective on attention: Factorization as compression can be more effective than dimension squeezing

Future Research Directions

FSAM's success opens several promising avenues:

1. Extended Applications: Video understanding, action recognition, medical time-series analysis
2. Enhanced NMF Variants: Incorporating temporal smoothness or frequency domain constraints
3. Multi-physiological Signals: Check out our subsequent work (MMRPhys) that explores this direction for robust estimation of multiple physiological signals
4. Hybrid Architectures: Combining factorization-based attention with other mechanisms for different modalities
5. Theoretical Analysis: Understanding why rank-1 factorization generalizes so well across datasets

Code & Resources