diff --git a/skills/archived/miaoshou-batch-import/SKILL.md b/skills/archived/miaoshou-batch-import/SKILL.md new file mode 100644 index 0000000..ea9934d --- /dev/null +++ b/skills/archived/miaoshou-batch-import/SKILL.md @@ -0,0 +1,350 @@ +--- +name: miaoshou-batch-import +description: Automate batch product import from AtomK to Miaoshou ERP via Excel upload +triggers: + - miaoshou import + - atomk to miaoshou + - batch upload miaoshou +--- + +# Miaoshou ERP Batch Import Automation + +## Overview +Import products from AtomK AI-generated listings into Miaoshou ERP collection box automatically. + +## Prerequisites +- Miaoshou cookies saved at `/home/ubuntu/.hermes/cookies/miaoshou_cookies.json` +- AtomK API credentials configured +- Python venv: `/home/ubuntu/.hermes/hermes-agent/venv/bin/python3` +- Dependencies: `requests`, `xlwt`, `DrissionPage`, `ddddocr` + +## Critical File Format +Miaoshou import page (`common_collect_box/index?fetchType=importCopy`) requires **exactly 5 columns**: + +``` +链接地址(必填) | 产品标题 | 价格(RMB) | 促销价(RMB) | 提示:促销价仅可用于采集到Lazada +``` + +⚠️ **Do NOT include "产品主编号" column** — it causes "解析文件数据错误" (parse error). + +Excel requirements: +- Sheet name: `Worksheet` +- Encoding: `utf_16_le` (use `xlwt` with `style_compression=2`) +- Save as `.xls` (NOT `.xlsx`) + +## Automation Script +Main script: `/home/ubuntu/auto_miaoshou_import.py` +Canonical wrapper (guarantees correct logging + verification): **`scripts/run-import.sh`** in this skill + +Key workflow: +1. Call AtomK `/api/v1/listings` to get latest unimported products +2. Filter out already-imported IDs (tracked in state file) +3. Build 5-column xls with `source_url`, AI title, AI price +4. Use DrissionPage to upload to Miaoshou import page +5. Wait for parsing result (max 120s) +6. Save imported IDs to `/home/ubuntu/.hermes/miaoshou_import_state.json` + +## Cron Setup +Use the venv Python, NOT `uv`. +Wrap execution with a timestamp header/footer so failures are easy to spot: + +```bash +# Check current tasks +uv run --from pycron cron list + +# Remove old broken tasks (if using wrong env) +uv run --from pycron cron remove + +# Add correct task (robust logging with timestamp + exit code — brace group redirect) +CRON_CMD='cd /home/ubuntu && { echo ""; echo "=== $(date \x27+%Y-%m-%d %H:%M:%S\x27) ==="; /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py; echo "=== END (exit code: $?) ==="; } >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1' + +uv run --from pycron cron add "0 * * * * $CRON_CMD" --name atomk-miaoshou-hourly +``` + +### Ad-hoc Execution from Agent (Terminal Tool) + +When triggering the script manually via the agent's terminal tool (not via cron), **you MUST combine all parts into a single terminal command**. Splitting across multiple terminal calls causes `>>` redirections to silently fail — the command returns exit 0 with empty stdout, but nothing is appended (see Pitfall #1). Even `bash -c` wrappers in separate calls don't reliably fix this. + +**Use `printf` for the timestamp header — NOT `echo -e`**: `echo -e` can silently fail to append even in a combined single command (observed May 2026: file mtime unchanged, nothing written). `printf` is more reliable for control characters in this context. + +**Method A — brace group redirect (preferred, most robust):** + +Wrap all commands in `{ ... ; } >> logfile 2>&1` so there is only ONE redirect operation. This sidesteps `echo -e` failures and multi-redirect fragility entirely. + +```bash +cd /home/ubuntu && { echo ""; echo "=== $(date '+%Y-%m-%d %H:%M:%S') ==="; /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py; echo "=== END (exit code: $?) ==="; } >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1; echo "Exit code: $?" +``` + +The outer `echo "Exit code: $?"` goes to terminal stdout (not the log) so the agent sees the result immediately. Inside the brace group, `$?` correctly captures the script's exit code. + +**Method B — printf + && chaining (alternative):** + +```bash +cd /home/ubuntu && printf '\n=== %s ===\n' "$(date '+%Y-%m-%d %H:%M:%S')" >> /home/ubuntu/.hermes/miaoshou_cron.log && /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1; EXIT_CODE=$?; echo "=== END (exit code: $EXIT_CODE) ===" >> /home/ubuntu/.hermes/miaoshou_cron.log; echo "Exit code was: $EXIT_CODE" +``` + +⚠️ Method B has three separate `>>` redirections — if the first `printf` silently fails (rare with `printf` but observed with `echo -e`), all subsequent appends still proceed and produce a malformed log entry missing the timestamp header. + +Then verify: +```bash +tail -20 /home/ubuntu/.hermes/miaoshou_cron.log +``` + +Always verify with `tail` that the log was actually written before reporting success. + +### Log Verification After Execution (Critical) + +After running ANY command that appends to `miaoshou_cron.log`, you **must** verify the log was actually written before reporting success. Use `terminal` with `tail` or `grep` — **never use `read_file`** for this check because `read_file` caches file content and may return stale or truncated results even after new data was appended. + +**Verification commands:** +```bash +# Quick check — last 20 lines +tail -n 20 /home/ubuntu/.hermes/miaoshou_cron.log + +# Scope to the current hour's entries +grep -n "=== $(date '+%Y-%m-%d %H')" /home/ubuntu/.hermes/miaoshou_cron.log + +# If the log is very large, check line count growth +wc -l /home/ubuntu/.hermes/miaoshou_cron.log +``` + +**If verification shows NO new entry** (last line is still from the previous run), the append silently failed. This happens most often with `echo -e` (see Pitfall #1) or when commands are split across multiple terminal calls. Do not report success. Instead, use the Fallback pattern: +1. Run the script bare (no redirect) to capture output in the terminal tool's stdout +2. Append the complete output block with `cat >> log << 'EOF'` +3. Verify again with `tail` + +**Why `read_file` fails for verification:** The `read_file` tool returns cached content if called multiple times on the same file in one conversation. Even when the file has grown, it may report "File unchanged since last read" and return the old truncated view. Terminal commands (`tail`, `cat`, `grep`) always read fresh from disk. + +### Cron Execution Behavior & Log Interpretation +The script uses **internal exception handling** (`try/except` inside `run_import()`) so the process almost always exits with code 0 even when the import itself fails. To know whether a run actually succeeded: +- Look for `✅` / `❌` markers inside the log, not just the final `=== END (exit code: 0) ===`. +- A Traceback appearing **before** the timestamped header is normal: the script prints it via `traceback.print_exc()` inside the except block, then logs the failure message. +- When reading the log, the **last non-empty block before `=== END`** tells you what happened. +- Common failure signatures in cron output: + - `❌ 流程异常: 'str' object has no attribute 'get'` → AtomK API returned malformed data (see Pitfall #10) + - `❌ 流程异常: 502 Server Error` → AtomK gateway temporarily unavailable; next hour usually recovers + - `✅ 没有新产品需要导入,本次跳过` → Healthy run, nothing to do + - `✅ 生成导入文件...` / `✅ 解析成功` → Healthy run, products imported + +### Cron Job Reporting Pattern (for scheduled agent runs) +When the agent executes this script as a cron job, use this pattern to extract a concise status report: + +1. **Run** the script with the brace-group redirect (Method A above). +2. **Read the last entry** from the log: + ```bash + grep -A 15 "^=== $(date '+%Y-%m-%d %H')" /home/ubuntu/.hermes/miaoshou_cron.log | tail -20 + ``` + This scopes to today's entries only and avoids reading the full (potentially 10K+ line) log. +3. **Parse the key metrics** from the output: + - `AtomK中有 N 个待导入产品` — how many were available + - `其中 N 个未导入过` — how many were new + - `累计已导入: N 个产品` — running total + - `✅ 没有新产品需要导入` → healthy idle run + - `✅ 解析成功 N/N/N` → successful import + - `❌ 流程异常: ...` → failure (check traceback before header) +4. **Fallback**: If the combined command produces no log output (silent `>>` failure — see Pitfall #1), run the script bare (no redirect), capture stdout from the terminal tool, then manually append the complete output block to the log. Always verify with `tail -15 logfile`. + +## Verification +After upload, check: +- Import history: `https://erp.91miaoshou.com/common_collect_box/index?fetchType=importCopy` bottom table +- Collection box count: `https://erp.91miaoshou.com/collect-box-aggregate/index` +- Logs: `/home/ubuntu/.hermes/miaoshou_cron.log` + +## Known Pitfalls +1. **Shell `>>` redirections via terminal tool — fragile when split across calls**: When using the Hermes terminal tool, `>>` appends can silently disappear if the echo/script/footer are split across multiple terminal invocations. The command returns exit 0 with empty stdout, but nothing is appended to the log file (file mtime unchanged). This affects BOTH bare commands and `bash -c '...'` wrappers when they're in separate calls (observed May 2026). **Most reliable fix**: use the brace-group redirect pattern — `{ echo "header"; script; echo "footer"; } >> logfile 2>&1` — which has only ONE redirect operation and avoids the `echo -e` failure mode entirely. As a secondary option, `printf` + `&&` chaining works but has three separate `>>` appends (see Ad-hoc Execution section for both patterns). **Never use `echo -e`** for the timestamp header: it silently fails to append even in combined single commands (confirmed May 2026, reconfirmed June 2026 — file mtime unchanged, nothing written). + +**Fallback when combined command still doesn't write to log**: If even the single-command approach produces no log output (file mtime unchanged, `tail` shows nothing new), recover by: +1. Run the script bare (no `>>` redirection) to capture output in the terminal tool's stdout. +2. Manually append the complete output block using `{ echo "..."; echo "..."; } >> logfile`. +3. Verify with `tail -15 logfile`. +This two-step pattern is slower but guaranteed to produce a log entry. +3. **6-column format**: Adding "产品主编号" breaks parsing entirely +4. **Headless detection**: Miaoshou may block headless Chrome; use `--disable-blink-features=AutomationControlled` +5. **Sub-account limitation**: `xiaochaoren2026` cannot access product publish page, only collection box/import functions +6. **Cookie login unreliable for import page**: Import page often redirects to public landing even with valid cookies. **Fresh login every run is more reliable**. +7. **Notification popups block "导入链接采集"**: Popups ("我知道了") must be closed before the button click registers. +8. **Duplicate source_url causes parse failure**: Miaoshou rejects repeated 1688 links in the same batch. Deduplicate by stripping `?_t=` params before generating XLS. +9. **OCR captcha errors**: ddddocr commonly misreads `o→0`, `l→1`, `S→5`, `B→8`, `Z→2`, `g→9`. Always clean the result before filling. +10. **Multiple captcha images on login page**: There are 2+ `captcha-img` elements (login, forgot password, register forms). Only the one linked to `input.J_captchaInput` is valid for the main login form. +11. **`'str' object has no attribute 'get'` from AtomK API**: The `/api/v1/listings` endpoint sometimes returns a string instead of a list of dicts (e.g., an error message or unexpected scalar). The script should verify `isinstance(products, list)` before iterating and calling `.get()` on each element. Without this check, the cron run crashes silently (caught internally) and imports 0 products. +12. **Log file grows unbounded**: `/home/ubuntu/.hermes/miaoshou_cron.log` accumulates ~10 lines per hour (~7200/month). At 13K+ lines (May 2026), `cat` and `wc -l` still work fine but avoid reading the entire file — use `tail -N` or `grep` instead. Consider periodic truncation: `echo "" > /home/ubuntu/.hermes/miaoshou_cron.log` or `tail -500 /home/ubuntu/.hermes/miaoshou_cron.log > /tmp/miaoshou_cron_trimmed.log && mv /tmp/miaoshou_cron_trimmed.log /home/ubuntu/.hermes/miaoshou_cron.log`. + +## Full Upload Workflow (Link Collection Import) + +**Recommended approach**: Do a fresh login every run instead of relying on stale cookies. + +### Step 1: Fresh Login with Captcha OCR + +关键点:妙手登录页面有多个表单(登录、注册、忘记密码),每个都有独立的 `captcha-img`。必须通过 `input.J_captchaInput` 定位主登录表单,向上遍历 parent 找关联的验证码图片,避免填写隐藏表单。 + +```python +import base64, ddddocr +from DrissionPage import ChromiumPage, ChromiumOptions + +def clean_captcha(text): + text = text.strip().replace(" ", "").replace("\n", "") + for old, new in [('o','0'),('O','0'),('l','1'),('I','1'),('i','1'), + ('S','5'),('s','5'),('B','8'),('b','6'), + ('Z','2'),('z','2'),('g','9'),('q','9')]: + text = text.replace(old, new) + import re + return re.sub(r'[^a-zA-Z0-9]', '', text) + +def login(page, username, password): + for attempt in range(5): + page.get("https://erp.91miaoshou.com/auth/login") + time.sleep(3) + page.ele("css:.account-input", timeout=5).clear().input(username) + page.ele("css:.password-input", timeout=5).clear().input(password) + + cap_input = page.ele("css:input.J_captchaInput", timeout=5) + parent = cap_input.parent() + cap_img = None + for _ in range(5): + cap_img = parent.ele("css:img.captcha-img", timeout=1) + if cap_img: break + parent = parent.parent() + if not parent: break + if not cap_img: + imgs = page.eles("css:img.captcha-img") + for img in imgs: + if (img.attr("src") or "").startswith("data:"): + cap_img = img + break + + src = cap_img.attr("src") or "" if cap_img else "" + code = None + if src.startswith("data:image/"): + b64 = src.split(",", 1)[1] + img_bytes = base64.b64decode(b64) + code = clean_captcha(ddddocr.DdddOcr().classification(img_bytes)) + if not code: + path = f"/tmp/miaoshou_cap_{attempt}.jpg" + cap_img.get_screenshot(path=path) + with open(path, "rb") as f: + code = clean_captcha(ddddocr.DdddOcr().classification(f.read())) + + cap_input.clear().input(code) + page.ele("css:.login.login-button", timeout=5).click() + time.sleep(4) + + url = page.url + body_text = page("tag:body").text + if "图形验证码不正确" in body_text: + continue + success = ["/welcome","/home","/dashboard","工作台","xiaochaoren","实时数据","产品采集"] + if any(s in url or s in body_text for s in success): + return True + return False +``` + +### Step 2: Close Notification Popups +Miaoshou shows "我知道了" notification popups that **must** be closed before clicking "导入链接采集". Failure to close them causes the click to do nothing. + +```python +def close_popups(page): + for _ in range(3): + page.run_js(''' + var btns = document.querySelectorAll("button, .el-message-box__headerbtn, .jx-dialog__close"); + for (var i = 0; i < btns.length; i++) { + var t = btns[i].textContent.trim(); + if (t === "关闭" || t === "我知道了") btns[i].click(); + } + ''') + time.sleep(1) +``` + +### Step 3: Open Import Dialog and Upload + +```python +page.get("https://erp.91miaoshou.com/common_collect_box/index?fetchType=importCopy") +time.sleep(6) +close_popups(page) + +# Click 导入链接采集 +btns = page.eles("css:button") +target = next((b for b in btns if b.text and "导入链接采集" in b.text), None) +if not target: + raise RuntimeError("Import button not found") +target.click() +time.sleep(6) + +# Find file input inside the dialog +all_inputs = page.eles("css:input") +file_input = next((inp for inp in all_inputs if (inp.attr("type") or "") == "file"), None) +if not file_input: + raise RuntimeError("No file input found in dialog") +file_input.input("/tmp/miaoshou_import_25.xls") +time.sleep(3) + +# Click 确认 +confirm_btns = page.eles("css:button") +confirm = next((cb for cb in confirm_btns if cb.text and "确认" in cb.text), None) +if confirm: + confirm.click() + print(f"Clicked: '{confirm.text}'") +time.sleep(3) +``` + +### Step 4: Wait for Parsing Result + +```python +import re + +for i in range(30): + time.sleep(5) + text = page("tag:body").text + if "解析成功" in text: + m = re.search(r"解析成功\s*(\d+)/(\d+)/(\d+)", text) + if m: + print(f"✅ 解析成功 {m.group(1)}/{m.group(2)}/{m.group(3)}") + else: + print("✅ 解析成功") + return True + elif "解析失败" in text: + print("❌ 解析失败") + return False + elif "等待解析" in text: + if i % 6 == 0: + print(f" [{i*5}s] 等待解析...") +``` + +### Step 5: Save Fresh Cookies (Optional) +```python +cookies = page.cookies(all_domains=True) +with open("/home/ubuntu/.hermes/cookies/miaoshou_cookies_new.json", "w") as f: + json.dump(cookies, f, indent=2) +``` + +**Key cookies to verify:** +- `autoLoginToken` (JWT) — critical for session persistence +- `mserp_sst` — session token +- `accountId` — user identifier + +## URL Deduplication & Link Quality + +Before generating the XLS, **deduplicate by `source_url`** and verify links are valid: + +```python +# Remove duplicates +seen = set() +unique = [] +for item in items: + url = item["source_url"].split("?")[0] # strip _t params + if url not in seen: + seen.add(url) + unique.append(item) + +# Result: 25 items → 19 unique → 6/6 success (vs 6/25 with duplicates) +``` + +Duplicate URLs cause "解析失败" because Miaoshou's scraper rejects repeated 1688 links in the same batch. + +## Cookie Expiration Detection & Recovery +When Miaoshou cookies expire, the import page **redirects to the public landing page** instead of staying logged in. + +**Symptoms:** +- URL becomes `https://erp.91miaoshou.com/?redirect=%2Fcommon_collect_box%2Findex%3FfetchType%3DimportCopy` +- Page shows "免费使用" / "立即登录" buttons instead of import interface +- `input[type="file"]` count is 0 +- Body text contains no "导入" or "采集" keywords