# AGENTS.md

Guide for coding agents working in this repository.

## Repo reality
This repo is an ESM TypeScript scraper focused on Magnit supermarket data.
The active stack is Playwright, Axios, PostgreSQL, and Drizzle ORM.
Use `package.json` and `src/` as source of truth.
Treat `README.md` carefully.
It still documents useful local setup like `docker-compose up -d`, but it also contains stale Prisma-era commands and descriptions that no longer match the code.
Do not treat README Prisma commands as the current workflow.

## Package manager and commands
Use `pnpm`.

```bash
pnpm install
pnpm exec playwright install chromium
pnpm type-check
pnpm build
pnpm dev
pnpm enrich
pnpm test-db
pnpm db:generate
pnpm db:migrate
pnpm db:push
pnpm db:studio
```

Command mapping:
* `pnpm type-check`, `tsc --noEmit`
* `pnpm build`, compile TypeScript to `dist/`
* `pnpm dev`, run `src/scripts/scrape-magnit-products.ts`
* `pnpm enrich`, run `src/scripts/enrich-product-details.ts`
* `pnpm test-db`, run `src/scripts/test-db-connection.ts`
* `pnpm db:*`, Drizzle migration and studio commands

There is no lint command, no automated test framework, and no single-test execution command today.

## Local DB setup and verification reality
The README still documents `docker-compose up -d` for local PostgreSQL, and that is still the documented bootstrap step.
Current source uses Drizzle, not Prisma, so do not describe `pnpm prisma:*` commands as current.

Use the commands that actually exist for validation:
* `pnpm type-check`
* `pnpm build`
* `pnpm test-db`, when DB setup or connectivity is involved
* `pnpm dev`, when scraper behavior is involved
* `pnpm enrich`, when enrichment flow is involved

There is no Jest, Vitest, Mocha, or similar runner configured, no per-file test command, and no single-test pattern to follow.

## Direct script entry points
Main scripts are `src/scripts/scrape-magnit-products.ts`, `src/scripts/enrich-product-details.ts`, and `src/scripts/test-db-connection.ts`.

Direct execution example:

```bash
tsx src/scripts/scrape-magnit-products.ts
MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts
```

## TypeScript and module conventions
The project is ESM TypeScript. `package.json` sets `"type": "module"`. `tsconfig.json` uses `strict: true`, `module: "ESNext"`, and `moduleResolution: "bundler"`.

Observed conventions:
* Local imports use the `.js` extension
* External imports are usually first, then local imports, but the repo is mixed
* Match nearby file style instead of rewriting unrelated imports
* Public APIs and service methods often have explicit types
* Pragmatic `any` and casts are present around ORM and script boundaries
* Do not invent a fake rule that every type must be explicit everywhere

## Naming and file naming
Follow the naming already in the repo.
* Classes, interfaces, and major service types use PascalCase
* Functions, variables, and object properties use camelCase
* Constants often use UPPER_SNAKE_CASE
* Class and service files are usually PascalCase, for example `MagnitApiScraper.ts`
* Utility, config, database, and schema files are usually lowercase, for example `logger.ts`, `errors.ts`, `client.ts`, `schema.ts`
* Script files are kebab-case, for example `scrape-magnit-products.ts`

## Logging, comments, and errors
Prefer `Logger` from `src/utils/logger.ts` for application code.
It provides `info`, `error`, `warn`, and `debug`, with `debug` gated by `DEBUG === 'true'`.

* `Logger` is preferred in scraper and service code
* Plain `console.log` and `console.error` still appear in infra and test-style scripts
* `src/config/database.ts` and `src/scripts/test-db-connection.ts` are current examples
* Comments and log messages are often Russian, so keep language and tone consistent with nearby code

Active error classes live in `src/utils/errors.ts`.
* `DatabaseError` is actively used in `ProductService`
* `APIError` is actively used in `MagnitApiScraper`
* `ScraperError` exists, but it is not a common active pattern today

Common style is to catch low-level failures, log the original error, and re-throw a domain-specific error when that pattern already exists nearby.

## Database and schema patterns
The active DB layer is Drizzle.

Source-backed facts:
* DB client is `src/database/client.ts`
* It exports `db = drizzle(pool, { schema })`
* `src/config/database.ts` re-exports `db` and provides connect and disconnect helpers
* Schema lives in `src/db/schema.ts`
* Products have a composite unique constraint on `(externalId, storeId)`
* Product price columns are decimal values
* Parser code converts inbound kopecks to rubles before persistence

Common query style uses Drizzle builders such as:
* `.select().from(...).where(...)`
* `.insert().values(...).returning()`
* `.update().set(...).where(...).returning()`

Batch size `50` is a common working number in product saving and scraper pagination defaults.

## Service and scraper flow
Keep the existing responsibilities intact.
* `MagnitApiScraper` owns Playwright session setup, cookie and device-id capture, API requests, retries, reinit behavior, pagination, and streaming batches
* `ProductParser` transforms API payloads into `CreateProductData` and enrichment fields
* `ProductService` owns store, category, and product persistence

Main persistence flow:
1. `MagnitApiScraper` fetches product batches
2. `ProductParser` maps API items into DB-ready structures
3. `ProductService` creates or updates stores, categories, and products

Prefer extending that flow over bypassing it.
Avoid dropping raw Drizzle writes into scraper fetch logic when `ProductService` already owns the path.

## Scripts and direct DB access
Production-style scraper flow leans on services.
Some debug and utility scripts are more direct.
* `src/scripts/scrape-magnit-products.ts` uses `connectDatabase`, `disconnectDatabase`, and passes `db` into scraper save methods
* `src/scripts/enrich-product-details.ts` instantiates `ProductService` directly and coordinates enrichment batches
* Debug or infra-style code may query `db` directly when narrowly focused on setup or inspection

Match the local pattern instead of forcing extra abstraction into one-off scripts.

## Environment variables
Common variables confirmed by source and docs:

```bash
DATABASE_URL=postgresql://user:password@localhost:5432/supermarket
MAGNIT_STORE_CODE=992301
DEBUG=true
```

Other runtime variables used by scripts include `MAGNIT_USE_STREAMING`, `MAGNIT_MAX_PRODUCTS`, `MAGNIT_STORE_TYPE`, `MAGNIT_CATALOG_TYPE`, `MAGNIT_HEADLESS`, `MAGNIT_PAGE_SIZE`, `MAGNIT_RATE_LIMIT_DELAY`, `MAGNIT_MAX_ITERATIONS`, `MAGNIT_RETRY_ATTEMPTS`, and `FETCH_OBJECT_INFO`.

## Cursor Requestly rule
The repo includes a Cursor rule at `.cursor/rules/requestly-test-rules.mdc` for `.requestly-supermarket/**/*.json`.

Preserve these points:
* Use `rq.test()` for tests
* Use `rq.expect()` for assertions
* Read response data from `rq.response.body` and parse it as JSON before working with fields
* Prices in these Requestly payloads are in kopecks, for example `24999` means `249.99` rubles

## Practical guidance
Read nearby files before standardizing style.
Prefer source over README when they disagree, document Drizzle not Prisma, and do not claim linting, automated tests, or single-test support that does not exist.
When changing scraper persistence, check `MagnitApiScraper`, `ProductParser`, and `ProductService` together.