Domain Validation and Screenshot Capture Application

A comprehensive Node.js application that processes domain validation and screenshot capture with parallel processing capabilities and a professional web-based control panel.

Table of Contents

• Features
• Installation
• Quick Start
• Usage
• API Documentation
• Configuration
• File Structure
• Data Storage
• Performance
• Troubleshooting
• Contributing
• License

Features

🔍 Domain Validation

• HTTP/HTTPS Connectivity Testing: Tests both HTTP and HTTPS access to domain.com/ip.php
• SSL Certificate Validation: Chrome-like SSL certificate validation for HTTPS connections
• Response Code Capture: Records HTTP status codes for both protocols
• Comprehensive Logging: Detailed logging of all validation attempts

📸 Screenshot Capture

• Automated Screenshots: Captures homepage screenshots for valid domains
• Multiple URL Attempts: Tries various URL combinations (http/https, with/without www)
• Organized Storage: Screenshots stored with timestamp and domain name
• Browser Automation: Uses Puppeteer for reliable screenshot capture
• Fixed Compatibility: Updated to work with latest Puppeteer versions
• Optimized File Sizes: JPEG compression with configurable quality (60-90%)
• Dimension Control: Configurable maximum width/height limits
• Format Options: Support for both PNG and JPEG formats
• Smart Compression: Automatic page size detection and clipping for large pages

📊 Data Management

• CSV Input Processing: Reads domains from sites_ssl.csv (website_cad column)
• JSON Results Storage: Structured storage of all validation results
• Duplicate Prevention: Automatically skips already processed domains
• Resumable Processing: Can continue from where it stopped
• Export Functionality: Export results to CSV format

⚡ Parallel Processing

• Configurable Concurrency: Process multiple domains simultaneously (default: 3 concurrent)
• Batch Processing: Domains processed in configurable batches (default: 10 per batch)
• Resource Management: Proper memory and browser resource management
• Performance Optimization: Significantly faster processing than sequential mode
• Environment Configuration: Configurable via environment variables or API

🎛️ Web Control Panel

• Real-time Monitoring: Live progress tracking and status updates
• Start/Stop Controls: Easy processing control with web interface
• Results Visualization: View recent validation results in a table
• Processing Statistics: Total, processed, valid, and remaining domain counts
• Live Logging: Real-time processing logs in the web interface
• Configuration Management: Adjust parallel processing settings via web interface

🔍 Results Management Interface

• Advanced Search & Filtering: Search by domain name, validation status, SSL status, screenshot availability
• Detailed Analytics: Comprehensive statistics and error categorization
• Export Filtered Results: Export search results to CSV format
• Error Analysis: Easy identification and troubleshooting of validation errors
• Pagination: Efficient browsing of large result sets
• Detailed View: Modal dialogs with complete validation details

Installation

Prerequisites

• Node.js 18+ and npm
• At least 2GB RAM (4GB+ recommended for large datasets)
• 1GB free disk space for screenshots and results

Setup Steps

1. Clone the repository:

   git clone https://github.com/kayquer/domain-validation-app.git
   cd domain-validation-app

2. Install dependencies:

   npm install

3. Prepare your data:

   • Create a CSV file named sites_ssl.csv in the root directory
   • Ensure it has a column named website_cad containing the domains to validate
   • Example format:

     "website_cad"
     "www.example.com"
     "test.domain.org"

4. Create required directories (optional - they'll be created automatically):

   mkdir screenshots results logs

Quick Start

1. Start the application:

   npm start

2. Open your browser and navigate to http://localhost:3000

3. Begin processing:

   • Click "Start Processing" to begin domain validation
   • Monitor progress in real-time
   • View results in the "Results Management" section

4. Access results:

   • Navigate to http://localhost:3000/results for advanced filtering
   • Export results to CSV for further analysis

Usage

Starting the Application

npm start

The application will start on http://localhost:3000

Using the Web Control Panel

1. Open your browser and navigate to http://localhost:3000
2. Configure processing settings (optional):
   • Adjust concurrency settings via the configuration API
   • Set batch sizes and delays between processing
3. Click "Start Processing" to begin domain validation
4. Monitor progress in real-time through the dashboard
5. Stop processing at any time using the "Stop Processing" button
6. Export results using the "Export CSV" button

Using the Results Management Interface

1. Navigate to Results: Click "Results Management" or go to http://localhost:3000/results
2. Search and Filter: Use the search and filter controls to find specific domains
3. View Statistics: Review comprehensive processing statistics and error analysis
4. Export Filtered Data: Export search results to CSV for further analysis
5. View Details: Click "Details" on any result for complete validation information

API Endpoints

Main Interface

• GET / - Web control panel interface
• GET /results - Results management interface

Processing Control

• POST /api/start - Start domain processing
• POST /api/stop - Stop domain processing
• GET /api/status - Get current processing status (includes parallel processing info)

Configuration Management

• GET /api/config - Get current parallel processing configuration
• POST /api/config - Update parallel processing configuration

  {
    "maxConcurrency": 5,
    "batchSize": 20,
    "delayBetweenBatches": 3000,
    "delayBetweenTasks": 1000
  }

Results and Data

• GET /api/results?limit=N - Get recent results (default limit: 10)
• GET /api/results/all - Get all validation results
• GET /api/results/stats - Get comprehensive processing statistics
• GET /api/results/export?filters - Export filtered results as CSV
• GET /api/export/csv - Export all results as CSV

File Structure

ping_dns_check_node/
├── modules/
│   ├── domainValidator.js    # Domain validation logic
│   ├── screenshotCapture.js  # Screenshot capture functionality
│   └── dataManager.js        # Data processing and storage
├── public/
│   ├── index.html           # Web control panel interface
│   └── app.js               # Frontend JavaScript
├── screenshots/             # Generated screenshots
├── results/                 # JSON results and processing state
├── logs/                    # Application logs
├── sites_ssl.csv           # Input CSV file
├── index.js                # Main application
├── test.js                 # Test script
└── package.json            # Dependencies and scripts

Data Storage

Results Format

Results are stored in results/validation_results.json with the following structure:

{
  "metadata": {
    "created": "2025-06-20T01:00:00.000Z",
    "lastUpdated": "2025-06-20T01:30:00.000Z",
    "totalProcessed": 150,
    "totalValid": 120,
    "totalInvalid": 30
  },
  "results": [
    {
      "id": "result_1234567890_abc123",
      "domain": "example.com",
      "cleanedDomain": "example.com",
      "timestamp": "2025-06-20T01:15:00.000Z",
      "http_status": 200,
      "https_status": 200,
      "ssl_valid": true,
      "response_code": 200,
      "is_valid": true,
      "processing_time": 1250,
      "screenshot_path": "/screenshots/example_com_2025-06-20T01-15-00.png",
      "screenshot_url": "https://example.com",
      "validation_details": { /* detailed validation info */ },
      "screenshot_details": { /* detailed screenshot info */ }
    }
  ]
}

Processing State

Current processing state is maintained in results/processing_state.json:

{
  "isProcessing": false,
  "currentIndex": 150,
  "totalDomains": 4412,
  "startTime": "2025-06-20T01:00:00.000Z",
  "lastProcessedDomain": "example.com",
  "processedDomains": ["domain1.com", "domain2.com"]
}

Validation Criteria

A domain is considered valid if:

1. Either HTTP or HTTPS responds with status code 200-299 to /ip.php endpoint
2. If HTTPS works, the SSL certificate must be valid (Chrome-like validation)

Testing

Run the test script to verify all modules are working:

node test.js

Configuration

Environment Variables

• PORT - Server port (default: 3000)
• MAX_CONCURRENCY - Maximum concurrent domain processing (default: 3)
• BATCH_SIZE - Number of domains per batch (default: 10)
• SCREENSHOT_FORMAT - Screenshot format: 'png' or 'jpeg' (default: 'jpeg')
• JPEG_QUALITY - JPEG quality percentage: 10-100 (default: 75)
• MAX_SCREENSHOT_WIDTH - Maximum screenshot width in pixels (default: 1920)
• MAX_SCREENSHOT_HEIGHT - Maximum screenshot height in pixels (default: 1080)

Parallel Processing Settings

• Max Concurrency: Number of domains processed simultaneously (1-10 recommended)
• Batch Size: Domains processed per batch before delay (5-50 recommended)
• Delay Between Batches: Milliseconds to wait between batches (1000-5000ms)
• Delay Between Tasks: Milliseconds to wait between individual domains (100-1000ms)

Timeouts

• Domain validation: 10 seconds per request
• Screenshot capture: 30 seconds per page
• SSL handshake: 10 seconds

Parallel Processing Settings

• Max Concurrency: Number of domains processed simultaneously (1-10 recommended)
• Batch Size: Domains processed per batch before delay (5-50 recommended)
• Delay Between Batches: Milliseconds to wait between batches (1000-5000ms)
• Delay Between Tasks: Milliseconds to wait between individual domains (100-1000ms)

Screenshot Optimization Settings

• Format: 'jpeg' for smaller files, 'png' for pixel-perfect quality
• JPEG Quality: 60% (maximum compression) to 90% (high quality)
• Dimensions: Limit max width/height to reduce file sizes
• Compression: Enable smart compression for large pages

Performance Tuning

• Low Resource Systems: Use maxConcurrency=1, batchSize=5, JPEG quality=60%
• High Performance Systems: Use maxConcurrency=5-10, batchSize=20-50, JPEG quality=75%
• Rate Limiting Concerns: Increase delays between tasks and batches
• Storage Optimization: Use JPEG format with 75% quality (30% smaller than PNG)

Logging

Logs are stored in the logs/ directory:

• application.log - Main application logs
• domain-validation.log - Domain validation specific logs
• screenshot-capture.log - Screenshot capture logs
• data-manager.log - Data management logs

Troubleshooting

Common Issues

1. "CSV file not found"

   • Ensure sites_ssl.csv exists in the root directory
   • Check that the file has a website_cad column

2. "Browser initialization failed"

   • Puppeteer may need additional dependencies on some systems
   • Try running: npm install puppeteer --force

3. "Permission denied" for screenshots

   • Ensure the application has write permissions to the screenshots/ directory

4. High memory usage

   • The application processes domains sequentially to manage resources
   • Consider processing in smaller batches for very large CSV files

Performance Tips

• The application includes built-in delays between requests to avoid overwhelming target servers
• Screenshots are only captured for valid domains to save resources
• Browser instances are reused and properly closed to prevent memory leaks

License

ISC License - See package.json for details