Mc Smog 9164527f58 feat: enhanced Magnit scraper with streaming mode and retry logic ...

- Add streaming mode for memory-efficient large catalog scraping
- Implement retry logic with exponential backoff
- Add auto session reinitialization on 403 errors
- Add configurable options (pageSize, maxProducts, rateLimitDelay)
- Add maxIterations protection against infinite loops
- Add retry.ts utility module with withRetry and withRetryAndReinit
- Update .env.example with new scraping options
- Add pgAdmin and CloudBeaver to docker-compose

Co-Authored-By: Claude <noreply@anthropic.com>

2026-01-21 22:14:04 +05:00

1.2 KiB

Raw Blame History

View Raw