From aeaf871a577fd6e52d2b8ada4ce778afdceb390f Mon Sep 17 00:00:00 2001 From: Mc_Smog Date: Sat, 11 Jul 2026 19:04:47 +0500 Subject: [PATCH] chore: update agent instructions --- AGENTS.md | 246 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 126 insertions(+), 120 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8625fb3..fa7b21b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,158 +1,164 @@ # AGENTS.md -Guidelines for AI coding agents working on this repository. +Guide for coding agents working in this repository. -## Project Overview +## Repo reality +This repo is an ESM TypeScript scraper focused on Magnit supermarket data. +The active stack is Playwright, Axios, PostgreSQL, and Drizzle ORM. +Use `package.json` and `src/` as source of truth. +Treat `README.md` carefully. +It still documents useful local setup like `docker-compose up -d`, but it also contains stale Prisma-era commands and descriptions that no longer match the code. +Do not treat README Prisma commands as the current workflow. -TypeScript-based scraper for Russian supermarkets (Magnit). Uses Playwright for sessions, Axios for API, PostgreSQL with Drizzle ORM. - -## Build & Run Commands - -**Package Manager**: Use `pnpm` (not npm/yarn) +## Package manager and commands +Use `pnpm`. ```bash -pnpm install # Install dependencies -pnpm exec playwright install chromium # Install browsers (once) -pnpm type-check # Type checking (validation) -pnpm build # Build TypeScript to dist/ -pnpm dev # Run main scraper -pnpm enrich # Run product enrichment -pnpm test-db # Test database connection +pnpm install +pnpm exec playwright install chromium +pnpm type-check +pnpm build +pnpm dev +pnpm enrich +pnpm test-db +pnpm db:generate +pnpm db:migrate +pnpm db:push +pnpm db:studio ``` -### Drizzle Commands +Command mapping: +* `pnpm type-check`, `tsc --noEmit` +* `pnpm build`, compile TypeScript to `dist/` +* `pnpm dev`, run `src/scripts/scrape-magnit-products.ts` +* `pnpm enrich`, run `src/scripts/enrich-product-details.ts` +* `pnpm test-db`, run `src/scripts/test-db-connection.ts` +* `pnpm db:*`, Drizzle migration and studio commands -```bash -pnpm db:generate # Generate migration files -pnpm db:migrate # Apply migrations -pnpm db:push # Push schema changes directly (dev only) -pnpm db:studio # Open database GUI -``` +There is no lint command, no automated test framework, and no single-test execution command today. -### Running Scripts Directly +## Local DB setup and verification reality +The README still documents `docker-compose up -d` for local PostgreSQL, and that is still the documented bootstrap step. +Current source uses Drizzle, not Prisma, so do not describe `pnpm prisma:*` commands as current. + +Use the commands that actually exist for validation: +* `pnpm type-check` +* `pnpm build` +* `pnpm test-db`, when DB setup or connectivity is involved +* `pnpm dev`, when scraper behavior is involved +* `pnpm enrich`, when enrichment flow is involved + +There is no Jest, Vitest, Mocha, or similar runner configured, no per-file test command, and no single-test pattern to follow. + +## Direct script entry points +Main scripts are `src/scripts/scrape-magnit-products.ts`, `src/scripts/enrich-product-details.ts`, and `src/scripts/test-db-connection.ts`. + +Direct execution example: ```bash tsx src/scripts/scrape-magnit-products.ts MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts ``` -## Testing +## TypeScript and module conventions +The project is ESM TypeScript. `package.json` sets `"type": "module"`. `tsconfig.json` uses `strict: true`, `module: "ESNext"`, and `moduleResolution: "bundler"`. -No test framework configured. Manual testing via `pnpm test-db`, `pnpm dev`, Prisma Studio. +Observed conventions: +* Local imports use the `.js` extension +* External imports are usually first, then local imports, but the repo is mixed +* Match nearby file style instead of rewriting unrelated imports +* Public APIs and service methods often have explicit types +* Pragmatic `any` and casts are present around ORM and script boundaries +* Do not invent a fake rule that every type must be explicit everywhere -## Code Style +## Naming and file naming +Follow the naming already in the repo. +* Classes, interfaces, and major service types use PascalCase +* Functions, variables, and object properties use camelCase +* Constants often use UPPER_SNAKE_CASE +* Class and service files are usually PascalCase, for example `MagnitApiScraper.ts` +* Utility, config, database, and schema files are usually lowercase, for example `logger.ts`, `errors.ts`, `client.ts`, `schema.ts` +* Script files are kebab-case, for example `scrape-magnit-products.ts` -### Imports +## Logging, comments, and errors +Prefer `Logger` from `src/utils/logger.ts` for application code. +It provides `info`, `error`, `warn`, and `debug`, with `debug` gated by `DEBUG === 'true'`. -1. External packages first, then internal modules -2. **Always include `.js` extension** for local imports (ESM) -3. Use named imports from Drizzle schema +* `Logger` is preferred in scraper and service code +* Plain `console.log` and `console.error` still appear in infra and test-style scripts +* `src/config/database.ts` and `src/scripts/test-db-connection.ts` are current examples +* Comments and log messages are often Russian, so keep language and tone consistent with nearby code -```typescript -import { chromium, Browser } from 'playwright'; -import axios from 'axios'; -import { Logger } from '../../../utils/logger.js'; -import { db } from '../../../config/database.js'; -import { products, stores, categories } from '../../../db/schema.js'; -import { eq, and, asc } from 'drizzle-orm'; -``` +Active error classes live in `src/utils/errors.ts`. +* `DatabaseError` is actively used in `ProductService` +* `APIError` is actively used in `MagnitApiScraper` +* `ScraperError` exists, but it is not a common active pattern today -### Naming Conventions +Common style is to catch low-level failures, log the original error, and re-throw a domain-specific error when that pattern already exists nearby. -| Type | Convention | Example | -|------|------------|---------| -| Classes/Interfaces | PascalCase | `MagnitApiScraper`, `CreateProductData` | -| Functions/variables | camelCase | `scrapeAllProducts`, `deviceId` | -| Constants | UPPER_SNAKE_CASE | `ACTUAL_API_PAGE_SIZE` | -| Class files | PascalCase | `MagnitApiScraper.ts` | -| Util files | camelCase | `logger.ts`, `errors.ts` | +## Database and schema patterns +The active DB layer is Drizzle. -### TypeScript Patterns +Source-backed facts: +* DB client is `src/database/client.ts` +* It exports `db = drizzle(pool, { schema })` +* `src/config/database.ts` re-exports `db` and provides connect and disconnect helpers +* Schema lives in `src/db/schema.ts` +* Products have a composite unique constraint on `(externalId, storeId)` +* Product price columns are decimal values +* Parser code converts inbound kopecks to rubles before persistence -- **Strict mode** - all types explicit -- Interfaces for data, optional props with `?`, `readonly` for constants +Common query style uses Drizzle builders such as: +* `.select().from(...).where(...)` +* `.insert().values(...).returning()` +* `.update().set(...).where(...).returning()` -```typescript -export interface MagnitScraperConfig { - storeCode: string; - headless?: boolean; -} -``` +Batch size `50` is a common working number in product saving and scraper pagination defaults. -### Error Handling +## Service and scraper flow +Keep the existing responsibilities intact. +* `MagnitApiScraper` owns Playwright session setup, cookie and device-id capture, API requests, retries, reinit behavior, pagination, and streaming batches +* `ProductParser` transforms API payloads into `CreateProductData` and enrichment fields +* `ProductService` owns store, category, and product persistence -Use custom error classes from `src/utils/errors.ts`: -- `ScraperError` - scraping failures -- `DatabaseError` - database operations -- `APIError` - HTTP/API failures (includes statusCode) +Main persistence flow: +1. `MagnitApiScraper` fetches product batches +2. `ProductParser` maps API items into DB-ready structures +3. `ProductService` creates or updates stores, categories, and products -```typescript -try { - // operation -} catch (error) { - Logger.error('Ошибка операции:', error); - throw new APIError( - `Не удалось: ${error instanceof Error ? error.message : String(error)}`, - statusCode - ); -} -``` +Prefer extending that flow over bypassing it. +Avoid dropping raw Drizzle writes into scraper fetch logic when `ProductService` already owns the path. -### Logging +## Scripts and direct DB access +Production-style scraper flow leans on services. +Some debug and utility scripts are more direct. +* `src/scripts/scrape-magnit-products.ts` uses `connectDatabase`, `disconnectDatabase`, and passes `db` into scraper save methods +* `src/scripts/enrich-product-details.ts` instantiates `ProductService` directly and coordinates enrichment batches +* Debug or infra-style code may query `db` directly when narrowly focused on setup or inspection -Use static `Logger` class from `src/utils/logger.ts`: +Match the local pattern instead of forcing extra abstraction into one-off scripts. -```typescript -Logger.info('Message'); // Always shown -Logger.error('Error:', error); // Always shown -Logger.debug('Debug'); // Only when DEBUG=true -``` - -### Async/Class Patterns - -- All async methods return `Promise` with explicit return types -- Class order: private props -> constructor -> public methods -> private methods -- Lifecycle: `initialize()` -> operations -> `close()` - -### Services Pattern - -- Services receive `db` (Drizzle instance) via constructor (DI) -- Use `getOrCreate` for idempotent operations -- Never call Drizzle directly from scrapers - -### Database Patterns - -- Upsert via composite unique constraint on `(externalId, storeId)` -- Batch processing: 50 items per batch -- Prices: Decimal (rubles), stored as decimal type -- Use `.select().from().where()` for queries -- Use `.insert().values()` for inserts -- Use `.update().set().where()` for updates -- Use `.delete().where()` for deletes - -### Comments - -- JSDoc for public methods, inline comments in Russian - -```typescript -/** Инициализация сессии через Playwright */ -async initialize(): Promise { } -``` - -## Cursor Rules - -### Requestly API Tests (`.requestly-supermarket/**/*.json`) - -- Use `rq.test()` for tests, `rq.expect()` for assertions -- Access response via `rq.response.body` (parse as JSON) -- Prices in kopecks (24999 = 249.99 rubles) - -See `.cursor/rules/requestly-test-rules.mdc` for full docs. - -## Environment Variables +## Environment variables +Common variables confirmed by source and docs: ```bash DATABASE_URL=postgresql://user:password@localhost:5432/supermarket MAGNIT_STORE_CODE=992301 DEBUG=true ``` + +Other runtime variables used by scripts include `MAGNIT_USE_STREAMING`, `MAGNIT_MAX_PRODUCTS`, `MAGNIT_STORE_TYPE`, `MAGNIT_CATALOG_TYPE`, `MAGNIT_HEADLESS`, `MAGNIT_PAGE_SIZE`, `MAGNIT_RATE_LIMIT_DELAY`, `MAGNIT_MAX_ITERATIONS`, `MAGNIT_RETRY_ATTEMPTS`, and `FETCH_OBJECT_INFO`. + +## Cursor Requestly rule +The repo includes a Cursor rule at `.cursor/rules/requestly-test-rules.mdc` for `.requestly-supermarket/**/*.json`. + +Preserve these points: +* Use `rq.test()` for tests +* Use `rq.expect()` for assertions +* Read response data from `rq.response.body` and parse it as JSON before working with fields +* Prices in these Requestly payloads are in kopecks, for example `24999` means `249.99` rubles + +## Practical guidance +Read nearby files before standardizing style. +Prefer source over README when they disagree, document Drizzle not Prisma, and do not claim linting, automated tests, or single-test support that does not exist. +When changing scraper persistence, check `MagnitApiScraper`, `ProductParser`, and `ProductService` together.