supermarket

awo/supermarket

Fork 0

Commit Graph

Author	SHA1	Message	Date
Mc Smog	9164527f58	feat: enhanced Magnit scraper with streaming mode and retry logic - Add streaming mode for memory-efficient large catalog scraping - Implement retry logic with exponential backoff - Add auto session reinitialization on 403 errors - Add configurable options (pageSize, maxProducts, rateLimitDelay) - Add maxIterations protection against infinite loops - Add retry.ts utility module with withRetry and withRetryAndReinit - Update .env.example with new scraping options - Add pgAdmin and CloudBeaver to docker-compose Co-Authored-By: Claude <noreply@anthropic.com>	2026-01-21 22:14:04 +05:00
Mc Smog	19c0426cdc	Initial commit: Supermarket scraper MVP	2025-12-28 23:29:30 +05:00

Author

SHA1

Message

Date

Mc Smog

9164527f58

feat: enhanced Magnit scraper with streaming mode and retry logic

- Add streaming mode for memory-efficient large catalog scraping
- Implement retry logic with exponential backoff
- Add auto session reinitialization on 403 errors
- Add configurable options (pageSize, maxProducts, rateLimitDelay)
- Add maxIterations protection against infinite loops
- Add retry.ts utility module with withRetry and withRetryAndReinit
- Update .env.example with new scraping options
- Add pgAdmin and CloudBeaver to docker-compose

Co-Authored-By: Claude <noreply@anthropic.com>

2026-01-21 22:14:04 +05:00

Mc Smog

19c0426cdc

Initial commit: Supermarket scraper MVP

2025-12-28 23:29:30 +05:00

2 Commits