awo/supermarket
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: update agent instructions

Browse Source
This commit is contained in:
Mc_Smog
2026-07-11 19:04:47 +05:00
parent b6f5138390
commit aeaf871a57
1 changed files with 126 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

246
AGENTS.md
View File

@@ -1,158 +1,164 @@
# AGENTS.md # AGENTS.md
Guidelines for AI coding agents working on this repository. Guide for coding agents working in this repository.
## Project Overview ## Repo reality
This repo is an ESM TypeScript scraper focused on Magnit supermarket data.
The active stack is Playwright, Axios, PostgreSQL, and Drizzle ORM.
Use `package.json` and `src/` as source of truth.
Treat `README.md` carefully.
It still documents useful local setup like `docker-compose up -d`, but it also contains stale Prisma-era commands and descriptions that no longer match the code.
Do not treat README Prisma commands as the current workflow.
TypeScript-based scraper for Russian supermarkets (Magnit). Uses Playwright for sessions, Axios for API, PostgreSQL with Drizzle ORM. ## Package manager and commands
Use `pnpm`.
## Build & Run Commands
**Package Manager**: Use `pnpm` (not npm/yarn)
```bash ```bash
pnpm install # Install dependencies pnpm install
pnpm exec playwright install chromium # Install browsers (once) pnpm exec playwright install chromium
pnpm type-check # Type checking (validation) pnpm type-check
pnpm build # Build TypeScript to dist/ pnpm build
pnpm dev # Run main scraper pnpm dev
pnpm enrich # Run product enrichment pnpm enrich
pnpm test-db # Test database connection pnpm test-db
pnpm db:generate
pnpm db:migrate
pnpm db:push
pnpm db:studio
``` ```
### Drizzle Commands Command mapping:
* `pnpm type-check`, `tsc --noEmit`
* `pnpm build`, compile TypeScript to `dist/`
* `pnpm dev`, run `src/scripts/scrape-magnit-products.ts`
* `pnpm enrich`, run `src/scripts/enrich-product-details.ts`
* `pnpm test-db`, run `src/scripts/test-db-connection.ts`
* `pnpm db:*`, Drizzle migration and studio commands
```bash There is no lint command, no automated test framework, and no single-test execution command today.
pnpm db:generate # Generate migration files
pnpm db:migrate # Apply migrations
pnpm db:push # Push schema changes directly (dev only)
pnpm db:studio # Open database GUI
```
### Running Scripts Directly ## Local DB setup and verification reality
The README still documents `docker-compose up -d` for local PostgreSQL, and that is still the documented bootstrap step.
Current source uses Drizzle, not Prisma, so do not describe `pnpm prisma:*` commands as current.
Use the commands that actually exist for validation:
* `pnpm type-check`
* `pnpm build`
* `pnpm test-db`, when DB setup or connectivity is involved
* `pnpm dev`, when scraper behavior is involved
* `pnpm enrich`, when enrichment flow is involved
There is no Jest, Vitest, Mocha, or similar runner configured, no per-file test command, and no single-test pattern to follow.
## Direct script entry points
Main scripts are `src/scripts/scrape-magnit-products.ts`, `src/scripts/enrich-product-details.ts`, and `src/scripts/test-db-connection.ts`.
Direct execution example:
```bash ```bash
tsx src/scripts/scrape-magnit-products.ts tsx src/scripts/scrape-magnit-products.ts
MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts MAGNIT_STORE_CODE=992301 tsx src/scripts/scrape-magnit-products.ts
``` ```
## Testing ## TypeScript and module conventions
The project is ESM TypeScript. `package.json` sets `"type": "module"`. `tsconfig.json` uses `strict: true`, `module: "ESNext"`, and `moduleResolution: "bundler"`.
No test framework configured. Manual testing via `pnpm test-db`, `pnpm dev`, Prisma Studio. Observed conventions:
* Local imports use the `.js` extension
* External imports are usually first, then local imports, but the repo is mixed
* Match nearby file style instead of rewriting unrelated imports
* Public APIs and service methods often have explicit types
* Pragmatic `any` and casts are present around ORM and script boundaries
* Do not invent a fake rule that every type must be explicit everywhere
## Code Style ## Naming and file naming
Follow the naming already in the repo.
* Classes, interfaces, and major service types use PascalCase
* Functions, variables, and object properties use camelCase
* Constants often use UPPER_SNAKE_CASE
* Class and service files are usually PascalCase, for example `MagnitApiScraper.ts`
* Utility, config, database, and schema files are usually lowercase, for example `logger.ts`, `errors.ts`, `client.ts`, `schema.ts`
* Script files are kebab-case, for example `scrape-magnit-products.ts`
### Imports ## Logging, comments, and errors
Prefer `Logger` from `src/utils/logger.ts` for application code.
It provides `info`, `error`, `warn`, and `debug`, with `debug` gated by `DEBUG === 'true'`.
1. External packages first, then internal modules * `Logger` is preferred in scraper and service code
2. **Always include `.js` extension** for local imports (ESM) * Plain `console.log` and `console.error` still appear in infra and test-style scripts
3. Use named imports from Drizzle schema * `src/config/database.ts` and `src/scripts/test-db-connection.ts` are current examples
* Comments and log messages are often Russian, so keep language and tone consistent with nearby code
```typescript Active error classes live in `src/utils/errors.ts`.
import { chromium, Browser } from 'playwright'; * `DatabaseError` is actively used in `ProductService`
import axios from 'axios'; * `APIError` is actively used in `MagnitApiScraper`
import { Logger } from '../../../utils/logger.js'; * `ScraperError` exists, but it is not a common active pattern today
import { db } from '../../../config/database.js';
import { products, stores, categories } from '../../../db/schema.js';
import { eq, and, asc } from 'drizzle-orm';
```
### Naming Conventions Common style is to catch low-level failures, log the original error, and re-throw a domain-specific error when that pattern already exists nearby.
| Type | Convention | Example | ## Database and schema patterns
|------|------------|---------| The active DB layer is Drizzle.
| Classes/Interfaces | PascalCase | `MagnitApiScraper`, `CreateProductData` |
| Functions/variables | camelCase | `scrapeAllProducts`, `deviceId` |
| Constants | UPPER_SNAKE_CASE | `ACTUAL_API_PAGE_SIZE` |
| Class files | PascalCase | `MagnitApiScraper.ts` |
| Util files | camelCase | `logger.ts`, `errors.ts` |
### TypeScript Patterns Source-backed facts:
* DB client is `src/database/client.ts`
* It exports `db = drizzle(pool, { schema })`
* `src/config/database.ts` re-exports `db` and provides connect and disconnect helpers
* Schema lives in `src/db/schema.ts`
* Products have a composite unique constraint on `(externalId, storeId)`
* Product price columns are decimal values
* Parser code converts inbound kopecks to rubles before persistence
- **Strict mode** - all types explicit Common query style uses Drizzle builders such as:
- Interfaces for data, optional props with `?`, `readonly` for constants * `.select().from(...).where(...)`
* `.insert().values(...).returning()`
* `.update().set(...).where(...).returning()`
```typescript Batch size `50` is a common working number in product saving and scraper pagination defaults.
export interface MagnitScraperConfig {
storeCode: string;
headless?: boolean;
}
```
### Error Handling ## Service and scraper flow
Keep the existing responsibilities intact.
* `MagnitApiScraper` owns Playwright session setup, cookie and device-id capture, API requests, retries, reinit behavior, pagination, and streaming batches
* `ProductParser` transforms API payloads into `CreateProductData` and enrichment fields
* `ProductService` owns store, category, and product persistence
Use custom error classes from `src/utils/errors.ts`: Main persistence flow:
- `ScraperError` - scraping failures 1. `MagnitApiScraper` fetches product batches
- `DatabaseError` - database operations 2. `ProductParser` maps API items into DB-ready structures
- `APIError` - HTTP/API failures (includes statusCode) 3. `ProductService` creates or updates stores, categories, and products
```typescript Prefer extending that flow over bypassing it.
try { Avoid dropping raw Drizzle writes into scraper fetch logic when `ProductService` already owns the path.
// operation
} catch (error) {
Logger.error('Ошибка операции:', error);
throw new APIError(
`Не удалось: ${error instanceof Error ? error.message : String(error)}`,
statusCode
);
}
```
### Logging ## Scripts and direct DB access
Production-style scraper flow leans on services.
Some debug and utility scripts are more direct.
* `src/scripts/scrape-magnit-products.ts` uses `connectDatabase`, `disconnectDatabase`, and passes `db` into scraper save methods
* `src/scripts/enrich-product-details.ts` instantiates `ProductService` directly and coordinates enrichment batches
* Debug or infra-style code may query `db` directly when narrowly focused on setup or inspection
Use static `Logger` class from `src/utils/logger.ts`: Match the local pattern instead of forcing extra abstraction into one-off scripts.
```typescript ## Environment variables
Logger.info('Message'); // Always shown Common variables confirmed by source and docs:
Logger.error('Error:', error); // Always shown
Logger.debug('Debug'); // Only when DEBUG=true
```
### Async/Class Patterns
- All async methods return `Promise<T>` with explicit return types
- Class order: private props -> constructor -> public methods -> private methods
- Lifecycle: `initialize()` -> operations -> `close()`
### Services Pattern
- Services receive `db` (Drizzle instance) via constructor (DI)
- Use `getOrCreate` for idempotent operations
- Never call Drizzle directly from scrapers
### Database Patterns
- Upsert via composite unique constraint on `(externalId, storeId)`
- Batch processing: 50 items per batch
- Prices: Decimal (rubles), stored as decimal type
- Use `.select().from().where()` for queries
- Use `.insert().values()` for inserts
- Use `.update().set().where()` for updates
- Use `.delete().where()` for deletes
### Comments
- JSDoc for public methods, inline comments in Russian
```typescript
/** Инициализация сессии через Playwright */
async initialize(): Promise<void> { }
```
## Cursor Rules
### Requestly API Tests (`.requestly-supermarket/**/*.json`)
- Use `rq.test()` for tests, `rq.expect()` for assertions
- Access response via `rq.response.body` (parse as JSON)
- Prices in kopecks (24999 = 249.99 rubles)
See `.cursor/rules/requestly-test-rules.mdc` for full docs.
## Environment Variables
```bash ```bash
DATABASE_URL=postgresql://user:password@localhost:5432/supermarket DATABASE_URL=postgresql://user:password@localhost:5432/supermarket
MAGNIT_STORE_CODE=992301 MAGNIT_STORE_CODE=992301
DEBUG=true DEBUG=true
``` ```
Other runtime variables used by scripts include `MAGNIT_USE_STREAMING`, `MAGNIT_MAX_PRODUCTS`, `MAGNIT_STORE_TYPE`, `MAGNIT_CATALOG_TYPE`, `MAGNIT_HEADLESS`, `MAGNIT_PAGE_SIZE`, `MAGNIT_RATE_LIMIT_DELAY`, `MAGNIT_MAX_ITERATIONS`, `MAGNIT_RETRY_ATTEMPTS`, and `FETCH_OBJECT_INFO`.
## Cursor Requestly rule
The repo includes a Cursor rule at `.cursor/rules/requestly-test-rules.mdc` for `.requestly-supermarket/**/*.json`.
Preserve these points:
* Use `rq.test()` for tests
* Use `rq.expect()` for assertions
* Read response data from `rq.response.body` and parse it as JSON before working with fields
* Prices in these Requestly payloads are in kopecks, for example `24999` means `249.99` rubles
## Practical guidance
Read nearby files before standardizing style.
Prefer source over README when they disagree, document Drizzle not Prisma, and do not claim linting, automated tests, or single-test support that does not exist.
When changing scraper persistence, check `MagnitApiScraper`, `ProductParser`, and `ProductService` together.