71 lines
3.2 KiB
Markdown
71 lines
3.2 KiB
Markdown
|
# CLAUDE.md — auto-reverse
|
||
|
|
|
||
|
|
## Active Context
|
||
|
|
|
||
|
|
- Swapped `anthropic` SDK → `litellm` for multi-provider support (OpenRouter, mimo, etc)
|
||
|
|
- Added `archive.jsonl` format to persist full request/response details
|
||
|
|
- Default model: `openrouter/xiaomi/mimo-v2.5-pro`
|
||
|
|
- Successfully reverse-engineered URA GLS APIs from `eservice.ura.gov.sg`
|
||
|
|
|
||
|
|
## Key Learnings
|
||
|
|
|
||
|
|
### JS Source Analysis Is The Real Breakthrough
|
||
|
|
The MITM proxy captures traffic but the **JavaScript source files** contain the real API documentation:
|
||
|
|
- Auth flows (JSONP tokens, Bearer headers, cookie patterns)
|
||
|
|
- Endpoint URLs and query structures
|
||
|
|
- Field names and data types
|
||
|
|
- Service registries and feature flags
|
||
|
|
|
||
|
|
**Always fetch and analyze JS source files** after capturing traffic. The files to look for:
|
||
|
|
- `*Service.js` — service-specific logic and API calls
|
||
|
|
- `Env.js` or `*Config.js` — host URLs, environment settings
|
||
|
|
- `*API.js` — auth patterns, token management
|
||
|
|
- `*Controller.js` — orchestration, service registry
|
||
|
|
|
||
|
|
### Archive Format Must Capture Everything
|
||
|
|
The original `archive.log` only saved `METHOD host/path status` — useless for replay.
|
||
|
|
The new `archive.jsonl` captures:
|
||
|
|
- Full request headers (cookies, auth tokens, referer)
|
||
|
|
- Request/query parameters
|
||
|
|
- Response headers (set-cookie, content-type)
|
||
|
|
- Response body (full JSON)
|
||
|
|
|
||
|
|
### LLM Agent Is Optional Overhead
|
||
|
|
For API reverse-engineering, the real value is:
|
||
|
|
1. **Proxy capture** (mitmproxy) — discovers endpoints
|
||
|
|
2. **JS source analysis** — reveals auth, structure, fields
|
||
|
|
3. **Standard API patterns** (ArcGIS REST, etc.) — enables replay
|
||
|
|
|
||
|
|
The LLM agent driving the browser adds cost and latency but wasn't essential for the URA workflow. Consider a "no-llm" mode that just captures + analyzes.
|
||
|
|
|
||
|
|
### Geo-blocking Awareness
|
||
|
|
Singapore government sites (URA, HDB, etc.) use Azure Application Gateway WAF that blocks non-SG IPs. The tool should:
|
||
|
|
- Detect 403 responses from WAFs and report geo-blocking
|
||
|
|
- Use the browser's context (same proxy) to fetch APIs, not direct `requests.get()`
|
||
|
|
- Document that the browser must be on an authorized IP
|
||
|
|
|
||
|
|
### ArcGIS REST Services Are Common
|
||
|
|
Government map sites often use ArcGIS REST services. Standard patterns:
|
||
|
|
```
|
||
|
|
GET /arcgis/rest/services/<name>/MapServer/<layer_id>?f=json # metadata
|
||
|
|
GET /arcgis/rest/services/<name>/MapServer/<layer_id>/query?where=1=1&outFields=*&f=json # data
|
||
|
|
GET /arcgis/rest/services/<name>/MapServer/export?... # map tiles
|
||
|
|
```
|
||
|
|
The tool should auto-detect ArcGIS endpoints and suggest these queries.
|
||
|
|
|
||
|
|
### Auth Pattern Detection Needed
|
||
|
|
The tool should automatically detect and document:
|
||
|
|
- JSONP token endpoints (strip callback wrapper, extract JWT)
|
||
|
|
- Bearer token auth headers
|
||
|
|
- Cookie-based sessions
|
||
|
|
- AWS Signature auth (different from simple Bearer)
|
||
|
|
|
||
|
|
## Improvements To Make
|
||
|
|
|
||
|
|
1. **Auto-analyze captured JS files** — extract API endpoints, auth patterns, headers
|
||
|
|
2. **Export replay scripts** — generate `requests.get()` or `curl` from captured flows
|
||
|
|
3. **ArcGIS-aware analysis** — detect MapServer endpoints, auto-query metadata
|
||
|
|
4. **Auth pattern detection** — notice JSONP tokens, Bearer headers, cookies
|
||
|
|
5. **Request/response diffing** — compare same endpoint with different params
|
||
|
|
6. **Skip LLM for simple sites** — proxy + JS analysis mode without agent
|