165 lines
7.3 KiB
Markdown
165 lines
7.3 KiB
Markdown
# AGENTS.md
|
|
|
|
Guide for coding agents working in this repository.
|
|
|
|
## Repo reality
|
|
This repo is an ESM TypeScript scraper focused on Magnit supermarket data.
|
|
The active stack is Playwright, Axios, PostgreSQL, and Drizzle ORM.
|
|
Use `package.json` and `src/` as source of truth.
|
|
Treat `README.md` carefully.
|
|
It still documents useful local setup like `docker-compose up -d`, but it also contains stale Prisma-era commands and descriptions that no longer match the code.
|
|
Do not treat README Prisma commands as the current workflow.
|
|
|
|
## Package manager and commands
|
|
Use `pnpm`.
|
|
|
|
```bash
|
|
pnpm install
|
|
pnpm exec playwright install chromium
|
|
pnpm type-check
|
|
pnpm build
|
|
pnpm dev
|
|
pnpm enrich
|
|
pnpm test-db
|
|
pnpm db:generate
|
|
pnpm db:migrate
|
|
pnpm db:push
|
|
pnpm db:studio
|
|
```
|
|
|
|
Command mapping:
|
|
* `pnpm type-check`, `tsc --noEmit`
|
|
* `pnpm build`, compile TypeScript to `dist/`
|
|
* `pnpm dev`, run `src/scripts/scrape-magnit-products.ts`
|
|
* `pnpm enrich`, run `src/scripts/enrich-product-details.ts`
|
|
* `pnpm test-db`, run `src/scripts/test-db-connection.ts`
|
|
* `pnpm db:*`, Drizzle migration and studio commands
|
|
|
|
There is no lint command, no automated test framework, and no single-test execution command today.
|
|
|
|
## Local DB setup and verification reality
|
|
The README still documents `docker-compose up -d` for local PostgreSQL, and that is still the documented bootstrap step.
|
|
Current source uses Drizzle, not Prisma, so do not describe `pnpm prisma:*` commands as current.
|
|
|
|
Use the commands that actually exist for validation:
|
|
* `pnpm type-check`
|
|
* `pnpm build`
|
|
* `pnpm test-db`, when DB setup or connectivity is involved
|
|
* `pnpm dev`, when scraper behavior is involved
|
|
* `pnpm enrich`, when enrichment flow is involved
|
|
|
|
There is no Jest, Vitest, Mocha, or similar runner configured, no per-file test command, and no single-test pattern to follow.
|
|
|
|
## Direct script entry points
|
|
Main scripts are `src/scripts/scrape-magnit-products.ts`, `src/scripts/enrich-product-details.ts`, and `src/scripts/test-db-connection.ts`.
|
|
|
|
Direct execution example:
|
|
|
|
```bash
|
|
tsx src/scripts/scrape-magnit-products.ts
|
|
MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts
|
|
```
|
|
|
|
## TypeScript and module conventions
|
|
The project is ESM TypeScript. `package.json` sets `"type": "module"`. `tsconfig.json` uses `strict: true`, `module: "ESNext"`, and `moduleResolution: "bundler"`.
|
|
|
|
Observed conventions:
|
|
* Local imports use the `.js` extension
|
|
* External imports are usually first, then local imports, but the repo is mixed
|
|
* Match nearby file style instead of rewriting unrelated imports
|
|
* Public APIs and service methods often have explicit types
|
|
* Pragmatic `any` and casts are present around ORM and script boundaries
|
|
* Do not invent a fake rule that every type must be explicit everywhere
|
|
|
|
## Naming and file naming
|
|
Follow the naming already in the repo.
|
|
* Classes, interfaces, and major service types use PascalCase
|
|
* Functions, variables, and object properties use camelCase
|
|
* Constants often use UPPER_SNAKE_CASE
|
|
* Class and service files are usually PascalCase, for example `MagnitApiScraper.ts`
|
|
* Utility, config, database, and schema files are usually lowercase, for example `logger.ts`, `errors.ts`, `client.ts`, `schema.ts`
|
|
* Script files are kebab-case, for example `scrape-magnit-products.ts`
|
|
|
|
## Logging, comments, and errors
|
|
Prefer `Logger` from `src/utils/logger.ts` for application code.
|
|
It provides `info`, `error`, `warn`, and `debug`, with `debug` gated by `DEBUG === 'true'`.
|
|
|
|
* `Logger` is preferred in scraper and service code
|
|
* Plain `console.log` and `console.error` still appear in infra and test-style scripts
|
|
* `src/config/database.ts` and `src/scripts/test-db-connection.ts` are current examples
|
|
* Comments and log messages are often Russian, so keep language and tone consistent with nearby code
|
|
|
|
Active error classes live in `src/utils/errors.ts`.
|
|
* `DatabaseError` is actively used in `ProductService`
|
|
* `APIError` is actively used in `MagnitApiScraper`
|
|
* `ScraperError` exists, but it is not a common active pattern today
|
|
|
|
Common style is to catch low-level failures, log the original error, and re-throw a domain-specific error when that pattern already exists nearby.
|
|
|
|
## Database and schema patterns
|
|
The active DB layer is Drizzle.
|
|
|
|
Source-backed facts:
|
|
* DB client is `src/database/client.ts`
|
|
* It exports `db = drizzle(pool, { schema })`
|
|
* `src/config/database.ts` re-exports `db` and provides connect and disconnect helpers
|
|
* Schema lives in `src/db/schema.ts`
|
|
* Products have a composite unique constraint on `(externalId, storeId)`
|
|
* Product price columns are decimal values
|
|
* Parser code converts inbound kopecks to rubles before persistence
|
|
|
|
Common query style uses Drizzle builders such as:
|
|
* `.select().from(...).where(...)`
|
|
* `.insert().values(...).returning()`
|
|
* `.update().set(...).where(...).returning()`
|
|
|
|
Batch size `50` is a common working number in product saving and scraper pagination defaults.
|
|
|
|
## Service and scraper flow
|
|
Keep the existing responsibilities intact.
|
|
* `MagnitApiScraper` owns Playwright session setup, cookie and device-id capture, API requests, retries, reinit behavior, pagination, and streaming batches
|
|
* `ProductParser` transforms API payloads into `CreateProductData` and enrichment fields
|
|
* `ProductService` owns store, category, and product persistence
|
|
|
|
Main persistence flow:
|
|
1. `MagnitApiScraper` fetches product batches
|
|
2. `ProductParser` maps API items into DB-ready structures
|
|
3. `ProductService` creates or updates stores, categories, and products
|
|
|
|
Prefer extending that flow over bypassing it.
|
|
Avoid dropping raw Drizzle writes into scraper fetch logic when `ProductService` already owns the path.
|
|
|
|
## Scripts and direct DB access
|
|
Production-style scraper flow leans on services.
|
|
Some debug and utility scripts are more direct.
|
|
* `src/scripts/scrape-magnit-products.ts` uses `connectDatabase`, `disconnectDatabase`, and passes `db` into scraper save methods
|
|
* `src/scripts/enrich-product-details.ts` instantiates `ProductService` directly and coordinates enrichment batches
|
|
* Debug or infra-style code may query `db` directly when narrowly focused on setup or inspection
|
|
|
|
Match the local pattern instead of forcing extra abstraction into one-off scripts.
|
|
|
|
## Environment variables
|
|
Common variables confirmed by source and docs:
|
|
|
|
```bash
|
|
DATABASE_URL=postgresql://user:password@localhost:5432/supermarket
|
|
MAGNIT_STORE_CODE=992301
|
|
DEBUG=true
|
|
```
|
|
|
|
Other runtime variables used by scripts include `MAGNIT_USE_STREAMING`, `MAGNIT_MAX_PRODUCTS`, `MAGNIT_STORE_TYPE`, `MAGNIT_CATALOG_TYPE`, `MAGNIT_HEADLESS`, `MAGNIT_PAGE_SIZE`, `MAGNIT_RATE_LIMIT_DELAY`, `MAGNIT_MAX_ITERATIONS`, `MAGNIT_RETRY_ATTEMPTS`, and `FETCH_OBJECT_INFO`.
|
|
|
|
## Cursor Requestly rule
|
|
The repo includes a Cursor rule at `.cursor/rules/requestly-test-rules.mdc` for `.requestly-supermarket/**/*.json`.
|
|
|
|
Preserve these points:
|
|
* Use `rq.test()` for tests
|
|
* Use `rq.expect()` for assertions
|
|
* Read response data from `rq.response.body` and parse it as JSON before working with fields
|
|
* Prices in these Requestly payloads are in kopecks, for example `24999` means `249.99` rubles
|
|
|
|
## Practical guidance
|
|
Read nearby files before standardizing style.
|
|
Prefer source over README when they disagree, document Drizzle not Prisma, and do not claim linting, automated tests, or single-test support that does not exist.
|
|
When changing scraper persistence, check `MagnitApiScraper`, `ProductParser`, and `ProductService` together.
|