[Feature Request] Implement Batch Processing (Chunking) to stabilize large library scans

[dgomesbr]

Is your feature request related to a problem? Please describe. Yes. Currently, in internal/media/processor.go, the application iterates through the entire movie/series list in a single continuous loop. For large libraries (e.g., 4,000+ items), this can flood the Radarr/Sonarr API with thousands of requests per second.

This causes:

1. Radarr/Sonarr Instability: The SQLite database locks up under the read pressure.
2. Container Crashes: The client.go HTTP requests consume file descriptors or memory until the container is killed.

Describe the solution you'd like I suggest modifying the main processing loop in internal/media/processor.go to process items in batches (e.g., 100 items at a time), with a configurable delay between batches.

Key Changes:

1. Batch Size: Default to 100.
2. Batch Delay: Default to 10 seconds (allows Radarr to "cool down" and write to DB).
3. Environment Variables: Add BATCH_SIZE and BATCH_DELAY_SECONDS to cmd/labelarr/main.go.

[dgomesbr]

Suggestions

1. Modify internal/media/processor.go Instead of a standard range loop, use a chunking loop.

package media

import (
	"time"
	"log"
)

// Suggestion: Add batching logic to the main Process loop
func (p *Processor) ProcessLibrary(items []MediaItem) {
	batchSize := 100 // Could be loaded from config
	delay := 10 * time.Second

	total := len(items)
	log.Printf("Starting processing of %d items in batches of %d...", total, batchSize)

	for i := 0; i < total; i += batchSize {
		end := i + batchSize
		if end > total {
			end = total
		}

		batch := items[i:end]
		log.Printf("Processing batch %d-%d of %d", i, end, total)

		// Process the current batch
		for _, item := range batch {
			p.processItem(item) // Your existing single-item logic
		}

		// The Brake Pedal: Pause between batches to let Radarr/Sonarr DB recover
		if end < total {
			log.Printf("Sleeping for %v to cool down...", delay)
			time.Sleep(delay)
		}
	}
}

2. Modify internal/radarr/client.go (Optional Optimization) If the HTTP client is creating too many connections, ensure the http.Client is reused and not created new for every request.

// internal/radarr/client.go

type Client struct {
    httpClient *http.Client
    // ...
}

func NewClient(...) *Client {
    return &Client{
        httpClient: &http.Client{
            Timeout: 30 * time.Second,
            Transport: &http.Transport{
                MaxIdleConns:        100,
                MaxIdleConnsPerHost: 100, // Important for batching
                IdleConnTimeout:     90 * time.Second,
            },
        },
    }
}

[dgomesbr]

@nullable-eth happy new year! hopefully this is an acceptable case, I'm having issues with a few low-end folks running labelarr and this PR should fix the issues.

[dgomesbr]

for slow servers, with the new retry logic, and the chunking, no more DDoS to your own Radarr and Sonarr servers.

[dgomesbr]

Tested with https://hub.docker.com/repository/docker/dgomesbr/labelarr/tags/batch-processing-chunking