Avva Sai Pranav

← Back to projects

Project · Data Science & Machine Learning

MoBI: Mobile Brain/Body Imaging Data Analysis

Johns Hopkins University · Biomedical Data Design · Team MoBI · Sep 2024 – Nov 2024

61Subjects
~17kData streams/subject
14Input features
0.8–1.0R² (within-subject)

This research project at Johns Hopkins University focused on developing machine learning models to analyze Mobile Brain/Body Imaging (MoBI) data and predict behavioral errors. The goal was to reduce the number of required data streams for MoBI experiments while maintaining predictive accuracy—ultimately working toward simplified diagnostic tools for the 17.4% of children affected by mental, behavioral, and developmental disorders. I implemented and compared Ridge Regression and Random Forest models, applied multiple normalization techniques (Z-score, Min-Max, Robust Scaling), and conducted extensive cross-validation including 5-fold CV and Leave-One-Out CV across 61 subjects with ~17,000 data streams each. Key findings showed that Random Forest achieved R² values of 0.8-1.0 at sample fractions above 0.8, while cross-subject generalization remained challenging. The project established a foundation for feature importance analysis and transfer learning approaches planned for manuscript publication.

61 Subjects Analyzed
~17,000 Data Streams per Subject
14 Input Features
0.8–1.0 R² (RF, high sample fraction)

Team Members: Isha Dev, Sai Pranav Avva, Taiming Chen, Liora Dsilva, Deborah Roman, Riya Modi

Problem Statement

  • Time & Cost: Running complete MoBI experiments is time-consuming and expensive
  • Model Limitations: Existing approaches rely on simple ridge regression with high prediction error
  • Core Challenge: Could we reduce the number of data streams required for MoBI experiments while maintaining predictive accuracy?
  • Clinical Impact: Could we find a relationship between MoBI data and behavioral errors that enables simplified diagnostic tools for the 17.4% of children affected by mental, behavioral, and developmental disorders?

Our Novel Approach:

  • Comparing ridge regression with nonlinear regression models (Random Forest)
  • Characterize feature importance
  • Implement time as a feature in the ML model for this dataset

Methodology

The project followed a comprehensive data preprocessing, model development, and validation workflow comparing linear and nonlinear regression approaches. We replicated results across all team members' systems to ensure reproducibility before applying advanced techniques. Click a stage to jump there.

Data Preprocessing

Dataset Overview: Processed and analyzed data from 61 subjects with approximately 17,000 data streams per subject across 14 input features including:

  • ECG metrics: ecg_rri, ecg_task_ts, ecg_heart_rate, ecg_rri_mean_5, ecg_rri_mean_10, ecg_rri_std_5, ecg_rri_std_10
  • EDA signals: eda_filtered, eda_peaks
  • Respiration data: resp_filtered, resp_resp_rate
  • Motion and EMG: abs_motion_filtered, emg1_filtered, emg2_filtered

Normalization Techniques: To ensure fair contribution of all features to the regression models, three normalization preprocessing techniques were applied with comparative heatmap visualization:

  • Min-Max Scaling: Rescales features to [0, 1] range
  • Z-Score Normalization: Standardizes to zero mean and unit variance
  • Robust Scaling: Uses median and IQR, robust to outliers
Comparison of normalization techniques: Before normalization, Min-Max, Z-Score, and Robust Scaling heatmaps showing Subject 1 data across all 14 features and 17000 time steps
Heatmap visualization comparing raw data with three normalization preprocessing techniques applied to Subject 1 MoBI dataset across 14 features

Machine Learning Models

Ridge Regression: Implemented Ridge Regression from scratch and validated against Scikit-learn implementation. Optimized the penalty parameter (alpha) to 0.01 through systematic tuning. Ridge regression served as the baseline linear model for comparison. Replication across all team members' systems ensured consistent results by using identical random states.

Random Forest Regression: Implemented nonlinear Random Forest regression to capture complex relationships in the MoBI data. Initial replication encountered inconsistencies due to multithreading issues, which were resolved by explicitly calling the random seed (RS = 23) prior to every trial. Evaluated performance across varying sample fractions on a logarithmic scale from 1000 to the total number of data streams.

0.01Ridge alpha (optimized)
RS = 23Random seed
Log scaleSample fraction sweep

Cross-Validation

Implemented comprehensive cross-validation strategies to evaluate model generalization:

  • 5-Fold Cross-Validation: Within-subject validation to assess model performance on held-out data from the same subject
  • Leave-One-Out Cross-Validation (LOOCV): Cross-subject validation where models trained on N-1 subjects predict on the held-out subject — critical for assessing generalizability

Identified optimal training-to-test split ratio of 0.4:0.6 for computational efficiency. Resolved cross-system reproducibility issues through explicit random seed initialization (RS = 23).

Results

Random Forest Regression — 5-Fold Cross-Validation

Random Forest regression with 5-fold cross-validation across all 61 subjects (~17,000 raw data streams each). Sample fractions above approximately 0.8 consistently achieved R² values between 0.8 and 1.0, indicating excellent within-subject prediction performance.

Random Forest Regression for Position Difference: R2 Score vs Sample Fraction showing all 61 subjects reaching 0.8-1.0 R² at high sample fractions
Random Forest — Target: Position Difference | 5-fold CV | 61 subjects | Truncated at R² = 0
Random Forest Regression for Stimulus Position: R2 Score vs Sample Fraction showing convergence to R² near 1.0
Random Forest — Target: Stimulus Position | 5-fold CV | 61 subjects
Random Forest Regression for Stimulus Speed: R2 Score vs Sample Fraction
Random Forest — Target: Stimulus Speed (change_rate_x) | 5-fold CV | 61 subjects
Random Forest Mean R² Scores for Stimulus Position comparing normalization methods
Random Forest Mean R² — Stimulus Position | Comparing normalization techniques
Random Forest Mean R² Scores for Stimulus Speed comparing normalization methods
Random Forest Mean R² — Stimulus Speed | Comparing normalization techniques

Ridge Regression — 5-Fold Cross-Validation

Ridge regression showed significantly lower R² values (max ~0.08) compared to Random Forest, confirming the nonlinear nature of the MoBI data relationships. Results shown for 1 subject with ~17,000 normalized data streams.

Ridge Regression Mean R² Scores for Position Difference showing max R² around 0.07-0.08
Ridge Regression Mean R² — Position Difference | 5-fold CV | Comparing normalization techniques
Ridge Regression Mean R² Scores for Stimulus Position
Ridge Regression Mean R² — Stimulus Position | 5-fold CV | Comparing normalization techniques
Ridge Regression Mean R² Scores for Stimulus Speed showing max R² around 0.014
Ridge Regression Mean R² — Stimulus Speed | 5-fold CV | Comparing normalization techniques

Leave-One-Out Cross-Validation (LOOCV)

Cross-Subject Generalization Challenge: LOOCV revealed significant challenges in cross-subject generalization. When training on N-1 subjects and predicting on the held-out subject, R² values dropped dramatically, ranging between -3.0 and 0.1 — indicating high inter-subject variability.

Random Forest LOOCV results showing negative R² values for 3 subjects, indicating poor cross-subject generalization
Random Forest LOOCV — 3 subjects | Before vs Min-Max Normalization | Negative R² indicates model performs worse than mean prediction
Ridge Regression LOOCV results across all 61 subjects showing consistently negative R² values
Ridge Regression LOOCV — 61 subjects | All three targets | Before Normalization, Min-Max, and Robust Scaling
Key Finding Random Forest outperforms Ridge Regression
  • Random Forest achieved R² values of 0.8–1.0 at sample fractions above 0.8 (within-subject)
  • Ridge Regression maxed at R² ~0.08, confirming nonlinear data relationships
  • Cross-subject generalization remains challenging: LOOCV R² between -3.0 and 0.1
  • Established reproducible ML pipeline across multiple development environments

Key Achievements

  • Established reproducible ML pipeline across multiple development environments
  • Demonstrated Random Forest superiority over Ridge Regression for within-subject prediction
  • Identified challenges in cross-subject generalization (R² between -3.0 and 0.1 for LOOCV)
  • Created foundation for feature importance analysis and transfer learning implementation

Next Steps

  • Implement transfer learning Sillyf algorithm for improved cross-subject generalization
  • Apply feature importance algorithm to highlight key predictors
  • Apply the selected regression model with Sillyf and feature importance
  • Target R² values range between 0.5 and 1 for cross-subject prediction
  • Write submission draft for manuscript publication
Skills Tools & technologies used
  • Languages: Python
  • ML Libraries: Scikit-learn, NumPy, Pandas
  • Visualization: Matplotlib, Seaborn
  • Techniques: Ridge Regression, Random Forest, 5-Fold CV, LOOCV
  • Data Processing: Z-score, Min-Max, Robust Scaling normalization
Team Contributions & credits

I implemented Ridge Regression and Random Forest models, applied multiple normalization techniques, conducted 5-fold and Leave-One-Out cross-validation, and contributed to reproducibility testing across team systems.

Team Members: Isha Dev, Sai Pranav Avva, Taiming Chen, Liora Dsilva, Deborah Roman, Riya Modi

Additional Information

Heilmeier Catechism and final results presentation slides detailing the MoBI project goals, methodology, and experimental findings.