D365 Record Saver

Automate opening and saving Dynamics 365 records in the browser to trigger client-side automation (JavaScript OnSave, Business Rules). Designed to handle 450,000+ records using parallel browser instances.

Overview

This application automates the process of opening and saving D365 records through the browser UI, which triggers client-side automation that only fires when a real user interacts with the form. It uses browser automation (Playwright) to simulate user actions at scale.

Features

• Parallel Browser Processing: Run multiple browser instances simultaneously (1-50 configurable)
• Persistent Authentication: Login once, reuse session across runs
• Progress Tracking: SQLite database tracks every record, survives restarts
• Error Recovery: Auto-retry failures, screenshot on error, resume after crashes
• Flexible Filtering: Filter by date range, views, custom FetchXML, or field values
• Real-time Monitoring: Live progress updates, speed metrics, ETA calculations
• Export & Reporting: CSV export of results, error reports, summary statistics

Prerequisites

1. Python 3.11+
2. Azure AD App Registration with:
   • API Permission: Dynamics CRM > user_impersonation
   • Client Secret generated
   • Redirect URI configured (if using delegated auth)

Installation

1. Clone/Create Project

cd d365-record-saver

2. Create Virtual Environment

# Create virtual environment
python -m venv venv

# Activate it
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate  # Windows

3. Install Dependencies

pip install -r requirements.txt

4. Install Playwright Browsers

playwright install chromium

First Run

streamlit run app.py

Setup Steps

1. Configure Connection

   • Enter your D365 URL (e.g., https://yourorg.crm.dynamics.com)
   • Enter Azure AD Tenant ID
   • Enter Azure AD Client ID
   • Enter Client Secret
   • Click "Connect" to test API connection

2. Setup Browser Login

   • Click "Setup Browser Login"
   • Complete D365 login manually (including MFA)
   • Session will be saved for future use

3. Select Records

   • Choose your target table/entity
   • Apply filters if needed
   • Click "Fetch Records"

4. Configure Execution

   • Set parallel browsers (start with 5-10 for testing)
   • Adjust timeouts if needed
   • Enable/disable error screenshots

5. Start Processing

   • Click "Start" to begin
   • Monitor progress in real-time
   • Pause/Stop as needed

Configuration

Parallel Browsers

• Recommended Start: 5-10 browsers
• Maximum Safe: 15 browsers (higher may trigger D365 throttling)
• High Risk: 30+ browsers (risk of account lockout)

Timeouts

• Form Load Timeout: How long to wait for D365 form to load (default: 60s)
• Save Timeout: How long to wait for save operation to complete (default: 30s)
• Delay Between Saves: Wait time after each save (default: 2s)

Safety Features

• Throttling detection and auto-adjustment
• Health checks every 100 records
• Graceful shutdown (waits for in-progress saves)
• Session validation before processing

Database

Progress is tracked in progress.db (SQLite):

• runs: Execution sessions
• records: Individual record status, errors, timestamps

This allows:

• Resume after crash
• View historical runs
• Export results to CSV
• Retry only failed records

Logging

Logs are written to:

logs/
├── app.log                    # General application log
└── run_YYYYMMDD_HHMMSS/      # Per-run directory
    ├── run.log               # Detailed run log
    ├── errors.log            # Errors only
    └── summary.json          # Run statistics

Troubleshooting

"Session Expired" Errors

• Click "Re-authenticate Browser" in the UI
• Complete the login process again

Slow Performance

• Reduce parallel browsers
• Increase delay between saves
• Check D365 environment performance

Throttling Detected

• Application will auto-reduce browser count
• Add delay between requests
• Contact D365 admin about service limits

Form Selectors Not Working

• D365 UI varies by version and customization
• Update selectors in automation/record_saver.py
• Check browser console for element IDs

Architecture

┌─────────────┐
│  Streamlit  │  User interface
│     UI      │
└──────┬──────┘
       │
       ├─────────────┐
       │             │
┌──────▼──────┐ ┌───▼────────┐
│    MSAL     │ │ Playwright │  Authentication
│   Auth API  │ │  Browser   │
└──────┬──────┘ └───┬────────┘
       │            │
┌──────▼────────────▼──────┐
│   Dataverse Web API      │  Fetch records metadata
└──────────────────────────┘
       │
┌──────▼──────────────────┐
│   Browser Pool Manager   │  Parallel automation
│   (10 browsers)          │
└──────┬──────────────────┘
       │
┌──────▼──────────────────┐
│   Record Saver Logic     │  Open form, save, verify
│   (per browser)          │
└──────┬──────────────────┘
       │
┌──────▼──────────────────┐
│   SQLite Progress DB     │  Track status, errors
└──────────────────────────┘

Security Notes

• Client secrets stored in config.json (add to .gitignore)
• Browser session data stored in browser_data/ directory
• Never commit credentials to source control
• Use environment variables or Azure Key Vault for production

Performance Tips

1. Start Small: Test with 100 records first
2. Monitor Throttling: Watch for 429 errors or slowdowns
3. Off-Peak Hours: Run during low-usage times
4. Headless Mode: Enable for better performance (no UI rendering)
5. Network: Run from same region as D365 instance

Limitations

• Only works with forms accessible via web UI
• Requires valid D365 user license
• Subject to D365 API and service protection limits
• Cannot bypass form validation or business rules (that's the point!)

License

MIT License - See LICENSE file

Support

For issues and questions:

• Check logs in logs/ directory
• Review error screenshots in screenshots/ directory
• Consult D365 admin for environment-specific issues

Contributing

Contributions welcome! Please:

1. Test with your D365 environment
2. Update selectors if needed for your version
3. Document any environment-specific changes