supermarket/AGENTS.md

AGENTS.md

Guide for coding agents working in this repository.

Repo reality

This repo is an ESM TypeScript scraper focused on Magnit supermarket data. The active stack is Playwright, Axios, PostgreSQL, and Drizzle ORM. Use package.json and src/ as source of truth. Treat README.md carefully. It still documents useful local setup like docker-compose up -d, but it also contains stale Prisma-era commands and descriptions that no longer match the code. Do not treat README Prisma commands as the current workflow.

Package manager and commands

Use pnpm.

pnpm install
pnpm exec playwright install chromium
pnpm type-check
pnpm build
pnpm dev
pnpm enrich
pnpm test-db
pnpm db:generate
pnpm db:migrate
pnpm db:push
pnpm db:studio

Command mapping:

• pnpm type-check, tsc --noEmit
• pnpm build, compile TypeScript to dist/
• pnpm dev, run src/scripts/scrape-magnit-products.ts
• pnpm enrich, run src/scripts/enrich-product-details.ts
• pnpm test-db, run src/scripts/test-db-connection.ts
• pnpm db:*, Drizzle migration and studio commands

There is no lint command, no automated test framework, and no single-test execution command today.

Local DB setup and verification reality

The README still documents docker-compose up -d for local PostgreSQL, and that is still the documented bootstrap step. Current source uses Drizzle, not Prisma, so do not describe pnpm prisma:* commands as current.

Use the commands that actually exist for validation:

• pnpm type-check
• pnpm build
• pnpm test-db, when DB setup or connectivity is involved
• pnpm dev, when scraper behavior is involved
• pnpm enrich, when enrichment flow is involved

There is no Jest, Vitest, Mocha, or similar runner configured, no per-file test command, and no single-test pattern to follow.

Direct script entry points

Main scripts are src/scripts/scrape-magnit-products.ts, src/scripts/enrich-product-details.ts, and src/scripts/test-db-connection.ts.

Direct execution example:

tsx src/scripts/scrape-magnit-products.ts
MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts

TypeScript and module conventions

The project is ESM TypeScript. package.json sets "type": "module". tsconfig.json uses strict: true, module: "ESNext", and moduleResolution: "bundler".

Observed conventions:

• Local imports use the .js extension
• External imports are usually first, then local imports, but the repo is mixed
• Match nearby file style instead of rewriting unrelated imports
• Public APIs and service methods often have explicit types
• Pragmatic any and casts are present around ORM and script boundaries
• Do not invent a fake rule that every type must be explicit everywhere

Naming and file naming

Follow the naming already in the repo.

• Classes, interfaces, and major service types use PascalCase
• Functions, variables, and object properties use camelCase
• Constants often use UPPER_SNAKE_CASE
• Class and service files are usually PascalCase, for example MagnitApiScraper.ts
• Utility, config, database, and schema files are usually lowercase, for example logger.ts, errors.ts, client.ts, schema.ts
• Script files are kebab-case, for example scrape-magnit-products.ts

Logging, comments, and errors

Prefer Logger from src/utils/logger.ts for application code. It provides info, error, warn, and debug, with debug gated by DEBUG === 'true'.

• Logger is preferred in scraper and service code
• Plain console.log and console.error still appear in infra and test-style scripts
• src/config/database.ts and src/scripts/test-db-connection.ts are current examples
• Comments and log messages are often Russian, so keep language and tone consistent with nearby code

Active error classes live in src/utils/errors.ts.

• DatabaseError is actively used in ProductService
• APIError is actively used in MagnitApiScraper
• ScraperError exists, but it is not a common active pattern today

Common style is to catch low-level failures, log the original error, and re-throw a domain-specific error when that pattern already exists nearby.

Database and schema patterns

The active DB layer is Drizzle.

Source-backed facts:

• DB client is src/database/client.ts
• It exports db = drizzle(pool, { schema })
• src/config/database.ts re-exports db and provides connect and disconnect helpers
• Schema lives in src/db/schema.ts
• Products have a composite unique constraint on (externalId, storeId)
• Product price columns are decimal values
• Parser code converts inbound kopecks to rubles before persistence

Common query style uses Drizzle builders such as:

• .select().from(...).where(...)
• .insert().values(...).returning()
• .update().set(...).where(...).returning()

Batch size 50 is a common working number in product saving and scraper pagination defaults.

Service and scraper flow

Keep the existing responsibilities intact.

• MagnitApiScraper owns Playwright session setup, cookie and device-id capture, API requests, retries, reinit behavior, pagination, and streaming batches
• ProductParser transforms API payloads into CreateProductData and enrichment fields
• ProductService owns store, category, and product persistence

Main persistence flow:

1. MagnitApiScraper fetches product batches
2. ProductParser maps API items into DB-ready structures
3. ProductService creates or updates stores, categories, and products

Prefer extending that flow over bypassing it. Avoid dropping raw Drizzle writes into scraper fetch logic when ProductService already owns the path.

Scripts and direct DB access

Production-style scraper flow leans on services. Some debug and utility scripts are more direct.

• src/scripts/scrape-magnit-products.ts uses connectDatabase, disconnectDatabase, and passes db into scraper save methods
• src/scripts/enrich-product-details.ts instantiates ProductService directly and coordinates enrichment batches
• Debug or infra-style code may query db directly when narrowly focused on setup or inspection

Match the local pattern instead of forcing extra abstraction into one-off scripts.

Environment variables

Common variables confirmed by source and docs:

DATABASE_URL=postgresql://user:password@localhost:5432/supermarket
MAGNIT_STORE_CODE=992301
DEBUG=true

Other runtime variables used by scripts include MAGNIT_USE_STREAMING, MAGNIT_MAX_PRODUCTS, MAGNIT_STORE_TYPE, MAGNIT_CATALOG_TYPE, MAGNIT_HEADLESS, MAGNIT_PAGE_SIZE, MAGNIT_RATE_LIMIT_DELAY, MAGNIT_MAX_ITERATIONS, MAGNIT_RETRY_ATTEMPTS, and FETCH_OBJECT_INFO.

Cursor Requestly rule

The repo includes a Cursor rule at .cursor/rules/requestly-test-rules.mdc for .requestly-supermarket/**/*.json.

Preserve these points:

• Use rq.test() for tests
• Use rq.expect() for assertions
• Read response data from rq.response.body and parse it as JSON before working with fields
• Prices in these Requestly payloads are in kopecks, for example 24999 means 249.99 rubles

Practical guidance

Read nearby files before standardizing style. Prefer source over README when they disagree, document Drizzle not Prisma, and do not claim linting, automated tests, or single-test support that does not exist. When changing scraper persistence, check MagnitApiScraper, ProductParser, and ProductService together.