admin9webs/atomk-hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c18daa706dacdfa559259fbf53e7baaa5606c200
atomk-hermes-skills/skills/archived/miaoshou-batch-import/SKILL.md
T

18 KiB
Raw Blame History

name, description, triggers
name description triggers
miaoshou-batch-import Automate batch product import from AtomK to Miaoshou ERP via Excel upload
miaoshou import
atomk to miaoshou
batch upload miaoshou

Miaoshou ERP Batch Import Automation

Overview

Import products from AtomK AI-generated listings into Miaoshou ERP collection box automatically.

Prerequisites

  • Miaoshou cookies saved at /home/ubuntu/.hermes/cookies/miaoshou_cookies.json
  • AtomK API credentials configured
  • Python venv: /home/ubuntu/.hermes/hermes-agent/venv/bin/python3
  • Dependencies: requests, xlwt, DrissionPage, ddddocr

Critical File Format

Miaoshou import page (common_collect_box/index?fetchType=importCopy) requires exactly 5 columns:

链接地址（必填） | 产品标题 | 价格（RMB） | 促销价（RMB） | 提示：促销价仅可用于采集到Lazada

⚠️ Do NOT include "产品主编号" column — it causes "解析文件数据错误" (parse error).

Excel requirements:

  • Sheet name: Worksheet
  • Encoding: utf_16_le (use xlwt with style_compression=2)
  • Save as .xls (NOT .xlsx)

Automation Script

Main script: /home/ubuntu/auto_miaoshou_import.py Canonical wrapper (guarantees correct logging + verification): scripts/run-import.sh in this skill

Key workflow:

  1. Call AtomK /api/v1/listings to get latest unimported products
  2. Filter out already-imported IDs (tracked in state file)
  3. Build 5-column xls with source_url, AI title, AI price
  4. Use DrissionPage to upload to Miaoshou import page
  5. Wait for parsing result (max 120s)
  6. Save imported IDs to /home/ubuntu/.hermes/miaoshou_import_state.json

Cron Setup

Use the venv Python, NOT uv. Wrap execution with a timestamp header/footer so failures are easy to spot:

# Check current tasks
uv run --from pycron cron list

# Remove old broken tasks (if using wrong env)
uv run --from pycron cron remove <task-id>

# Add correct task (robust logging with timestamp + exit code — brace group redirect)
CRON_CMD='cd /home/ubuntu && { echo ""; echo "=== $(date \x27+%Y-%m-%d %H:%M:%S\x27) ==="; /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py; echo "=== END (exit code: $?) ==="; } >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1'

uv run --from pycron cron add "0 * * * * $CRON_CMD" --name atomk-miaoshou-hourly

Ad-hoc Execution from Agent (Terminal Tool)

When triggering the script manually via the agent's terminal tool (not via cron), you MUST combine all parts into a single terminal command. Splitting across multiple terminal calls causes >> redirections to silently fail — the command returns exit 0 with empty stdout, but nothing is appended (see Pitfall #1). Even bash -c wrappers in separate calls don't reliably fix this.

Use printf for the timestamp header — NOT echo -e: echo -e can silently fail to append even in a combined single command (observed May 2026: file mtime unchanged, nothing written). printf is more reliable for control characters in this context.

Method A — brace group redirect (preferred, most robust):

Wrap all commands in { ... ; } >> logfile 2>&1 so there is only ONE redirect operation. This sidesteps echo -e failures and multi-redirect fragility entirely.

cd /home/ubuntu && { echo ""; echo "=== $(date '+%Y-%m-%d %H:%M:%S') ==="; /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py; echo "=== END (exit code: $?) ==="; } >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1; echo "Exit code: $?"

The outer echo "Exit code: $?" goes to terminal stdout (not the log) so the agent sees the result immediately. Inside the brace group, $? correctly captures the script's exit code.

Method B — printf + && chaining (alternative):

cd /home/ubuntu && printf '\n=== %s ===\n' "$(date '+%Y-%m-%d %H:%M:%S')" >> /home/ubuntu/.hermes/miaoshou_cron.log && /home/ubuntu/.hermes/hermes-agent/venv/bin/python3 /home/ubuntu/auto_miaoshou_import.py >> /home/ubuntu/.hermes/miaoshou_cron.log 2>&1; EXIT_CODE=$?; echo "=== END (exit code: $EXIT_CODE) ===" >> /home/ubuntu/.hermes/miaoshou_cron.log; echo "Exit code was: $EXIT_CODE"

⚠️ Method B has three separate >> redirections — if the first printf silently fails (rare with printf but observed with echo -e), all subsequent appends still proceed and produce a malformed log entry missing the timestamp header.

Then verify:

tail -20 /home/ubuntu/.hermes/miaoshou_cron.log

Always verify with tail that the log was actually written before reporting success.

Log Verification After Execution (Critical)

After running ANY command that appends to miaoshou_cron.log, you must verify the log was actually written before reporting success. Use terminal with tail or grep — never use read_file for this check because read_file caches file content and may return stale or truncated results even after new data was appended.

Verification commands:

# Quick check — last 20 lines
tail -n 20 /home/ubuntu/.hermes/miaoshou_cron.log

# Scope to the current hour's entries
grep -n "=== $(date '+%Y-%m-%d %H')" /home/ubuntu/.hermes/miaoshou_cron.log

# If the log is very large, check line count growth
wc -l /home/ubuntu/.hermes/miaoshou_cron.log

If verification shows NO new entry (last line is still from the previous run), the append silently failed. This happens most often with echo -e (see Pitfall #1) or when commands are split across multiple terminal calls. Do not report success. Instead, use the Fallback pattern:

  1. Run the script bare (no redirect) to capture output in the terminal tool's stdout
  2. Append the complete output block with cat >> log << 'EOF'
  3. Verify again with tail

Why read_file fails for verification: The read_file tool returns cached content if called multiple times on the same file in one conversation. Even when the file has grown, it may report "File unchanged since last read" and return the old truncated view. Terminal commands (tail, cat, grep) always read fresh from disk.

Cron Execution Behavior & Log Interpretation

The script uses internal exception handling (try/except inside run_import()) so the process almost always exits with code 0 even when the import itself fails. To know whether a run actually succeeded:

  • Look for ✅ / ❌ markers inside the log, not just the final === END (exit code: 0) ===.
  • A Traceback appearing before the timestamped header is normal: the script prints it via traceback.print_exc() inside the except block, then logs the failure message.
  • When reading the log, the last non-empty block before === END tells you what happened.
  • Common failure signatures in cron output:
    • ❌ 流程异常: 'str' object has no attribute 'get' → AtomK API returned malformed data (see Pitfall #10)
    • ❌ 流程异常: 502 Server Error → AtomK gateway temporarily unavailable; next hour usually recovers
    • ✅ 没有新产品需要导入，本次跳过 → Healthy run, nothing to do
    • ✅ 生成导入文件... / ✅ 解析成功 → Healthy run, products imported

Cron Job Reporting Pattern (for scheduled agent runs)

When the agent executes this script as a cron job, use this pattern to extract a concise status report:

  1. Run the script with the brace-group redirect (Method A above).
  2. Read the last entry from the log: 
    grep -A 15 "^=== $(date '+%Y-%m-%d %H')" /home/ubuntu/.hermes/miaoshou_cron.log | tail -20
    This scopes to today's entries only and avoids reading the full (potentially 10K+ line) log.
  3. Parse the key metrics from the output:
    • AtomK中有 N 个待导入产品 — how many were available
    • 其中 N 个未导入过 — how many were new
    • 累计已导入: N 个产品 — running total
    • ✅ 没有新产品需要导入 → healthy idle run
    • ✅ 解析成功 N/N/N → successful import
    • ❌ 流程异常: ... → failure (check traceback before header)
  4. Fallback: If the combined command produces no log output (silent >> failure — see Pitfall #1), run the script bare (no redirect), capture stdout from the terminal tool, then manually append the complete output block to the log. Always verify with tail -15 logfile.

Verification

After upload, check:

  • Import history: https://erp.91miaoshou.com/common_collect_box/index?fetchType=importCopy bottom table
  • Collection box count: https://erp.91miaoshou.com/collect-box-aggregate/index
  • Logs: /home/ubuntu/.hermes/miaoshou_cron.log

Known Pitfalls

  1. Shell >> redirections via terminal tool — fragile when split across calls: When using the Hermes terminal tool, >> appends can silently disappear if the echo/script/footer are split across multiple terminal invocations. The command returns exit 0 with empty stdout, but nothing is appended to the log file (file mtime unchanged). This affects BOTH bare commands and bash -c '...' wrappers when they're in separate calls (observed May 2026). Most reliable fix: use the brace-group redirect pattern — { echo "header"; script; echo "footer"; } >> logfile 2>&1 — which has only ONE redirect operation and avoids the echo -e failure mode entirely. As a secondary option, printf + && chaining works but has three separate >> appends (see Ad-hoc Execution section for both patterns). Never use echo -e for the timestamp header: it silently fails to append even in combined single commands (confirmed May 2026, reconfirmed June 2026 — file mtime unchanged, nothing written).

Fallback when combined command still doesn't write to log: If even the single-command approach produces no log output (file mtime unchanged, tail shows nothing new), recover by:

  1. Run the script bare (no >> redirection) to capture output in the terminal tool's stdout.
  2. Manually append the complete output block using { echo "..."; echo "..."; } >> logfile.
  3. Verify with tail -15 logfile. This two-step pattern is slower but guaranteed to produce a log entry.
  4. 6-column format: Adding "产品主编号" breaks parsing entirely
  5. Headless detection: Miaoshou may block headless Chrome; use --disable-blink-features=AutomationControlled
  6. Sub-account limitation: xiaochaoren2026 cannot access product publish page, only collection box/import functions
  7. Cookie login unreliable for import page: Import page often redirects to public landing even with valid cookies. Fresh login every run is more reliable.
  8. Notification popups block "导入链接采集": Popups ("我知道了") must be closed before the button click registers.
  9. Duplicate source_url causes parse failure: Miaoshou rejects repeated 1688 links in the same batch. Deduplicate by stripping ?_t= params before generating XLS.
  10. OCR captcha errors: ddddocr commonly misreads o→0, l→1, S→5, B→8, Z→2, g→9. Always clean the result before filling.
  11. Multiple captcha images on login page: There are 2+ captcha-img elements (login, forgot password, register forms). Only the one linked to input.J_captchaInput is valid for the main login form.
  12. 'str' object has no attribute 'get' from AtomK API: The /api/v1/listings endpoint sometimes returns a string instead of a list of dicts (e.g., an error message or unexpected scalar). The script should verify isinstance(products, list) before iterating and calling .get() on each element. Without this check, the cron run crashes silently (caught internally) and imports 0 products.
  13. Log file grows unbounded: /home/ubuntu/.hermes/miaoshou_cron.log accumulates ~10 lines per hour (~7200/month). At 13K+ lines (May 2026), cat and wc -l still work fine but avoid reading the entire file — use tail -N or grep instead. Consider periodic truncation: echo "" > /home/ubuntu/.hermes/miaoshou_cron.log or tail -500 /home/ubuntu/.hermes/miaoshou_cron.log > /tmp/miaoshou_cron_trimmed.log && mv /tmp/miaoshou_cron_trimmed.log /home/ubuntu/.hermes/miaoshou_cron.log.

Full Upload Workflow (Link Collection Import)

Recommended approach: Do a fresh login every run instead of relying on stale cookies.

Step 1: Fresh Login with Captcha OCR

关键点：妙手登录页面有多个表单（登录、注册、忘记密码），每个都有独立的 captcha-img。必须通过 input.J_captchaInput 定位主登录表单，向上遍历 parent 找关联的验证码图片，避免填写隐藏表单。

import base64, ddddocr
from DrissionPage import ChromiumPage, ChromiumOptions

def clean_captcha(text):
    text = text.strip().replace(" ", "").replace("\n", "")
    for old, new in [('o','0'),('O','0'),('l','1'),('I','1'),('i','1'),
                     ('S','5'),('s','5'),('B','8'),('b','6'),
                     ('Z','2'),('z','2'),('g','9'),('q','9')]:
        text = text.replace(old, new)
    import re
    return re.sub(r'[^a-zA-Z0-9]', '', text)

def login(page, username, password):
    for attempt in range(5):
        page.get("https://erp.91miaoshou.com/auth/login")
        time.sleep(3)
        page.ele("css:.account-input", timeout=5).clear().input(username)
        page.ele("css:.password-input", timeout=5).clear().input(password)

        cap_input = page.ele("css:input.J_captchaInput", timeout=5)
        parent = cap_input.parent()
        cap_img = None
        for _ in range(5):
            cap_img = parent.ele("css:img.captcha-img", timeout=1)
            if cap_img: break
            parent = parent.parent()
            if not parent: break
        if not cap_img:
            imgs = page.eles("css:img.captcha-img")
            for img in imgs:
                if (img.attr("src") or "").startswith("data:"):
                    cap_img = img
                    break

        src = cap_img.attr("src") or "" if cap_img else ""
        code = None
        if src.startswith("data:image/"):
            b64 = src.split(",", 1)[1]
            img_bytes = base64.b64decode(b64)
            code = clean_captcha(ddddocr.DdddOcr().classification(img_bytes))
        if not code:
            path = f"/tmp/miaoshou_cap_{attempt}.jpg"
            cap_img.get_screenshot(path=path)
            with open(path, "rb") as f:
                code = clean_captcha(ddddocr.DdddOcr().classification(f.read()))

        cap_input.clear().input(code)
        page.ele("css:.login.login-button", timeout=5).click()
        time.sleep(4)

        url = page.url
        body_text = page("tag:body").text
        if "图形验证码不正确" in body_text:
            continue
        success = ["/welcome","/home","/dashboard","工作台","xiaochaoren","实时数据","产品采集"]
        if any(s in url or s in body_text for s in success):
            return True
    return False

Step 2: Close Notification Popups

Miaoshou shows "我知道了" notification popups that must be closed before clicking "导入链接采集". Failure to close them causes the click to do nothing.

def close_popups(page):
    for _ in range(3):
        page.run_js('''
            var btns = document.querySelectorAll("button, .el-message-box__headerbtn, .jx-dialog__close");
            for (var i = 0; i < btns.length; i++) {
                var t = btns[i].textContent.trim();
                if (t === "关闭" || t === "我知道了") btns[i].click();
            }
        ''')
        time.sleep(1)

Step 3: Open Import Dialog and Upload

page.get("https://erp.91miaoshou.com/common_collect_box/index?fetchType=importCopy")
time.sleep(6)
close_popups(page)

# Click 导入链接采集
btns = page.eles("css:button")
target = next((b for b in btns if b.text and "导入链接采集" in b.text), None)
if not target:
    raise RuntimeError("Import button not found")
target.click()
time.sleep(6)

# Find file input inside the dialog
all_inputs = page.eles("css:input")
file_input = next((inp for inp in all_inputs if (inp.attr("type") or "") == "file"), None)
if not file_input:
    raise RuntimeError("No file input found in dialog")
file_input.input("/tmp/miaoshou_import_25.xls")
time.sleep(3)

# Click 确认
confirm_btns = page.eles("css:button")
confirm = next((cb for cb in confirm_btns if cb.text and "确认" in cb.text), None)
if confirm:
    confirm.click()
    print(f"Clicked: '{confirm.text}'")
time.sleep(3)

Step 4: Wait for Parsing Result

import re

for i in range(30):
    time.sleep(5)
    text = page("tag:body").text
    if "解析成功" in text:
        m = re.search(r"解析成功\s*(\d+)/(\d+)/(\d+)", text)
        if m:
            print(f"✅ 解析成功 {m.group(1)}/{m.group(2)}/{m.group(3)}")
        else:
            print("✅ 解析成功")
        return True
    elif "解析失败" in text:
        print("❌ 解析失败")
        return False
    elif "等待解析" in text:
        if i % 6 == 0:
            print(f"  [{i*5}s] 等待解析...")

Step 5: Save Fresh Cookies (Optional)

cookies = page.cookies(all_domains=True)
with open("/home/ubuntu/.hermes/cookies/miaoshou_cookies_new.json", "w") as f:
    json.dump(cookies, f, indent=2)

Key cookies to verify:

  • autoLoginToken (JWT) — critical for session persistence
  • mserp_sst — session token
  • accountId — user identifier

URL Deduplication & Link Quality

Before generating the XLS, deduplicate by source_url and verify links are valid:

# Remove duplicates
seen = set()
unique = []
for item in items:
    url = item["source_url"].split("?")[0]  # strip _t params
    if url not in seen:
        seen.add(url)
        unique.append(item)

# Result: 25 items → 19 unique → 6/6 success (vs 6/25 with duplicates)

Duplicate URLs cause "解析失败" because Miaoshou's scraper rejects repeated 1688 links in the same batch.

Cookie Expiration Detection & Recovery

When Miaoshou cookies expire, the import page redirects to the public landing page instead of staying logged in.

Symptoms:

  • URL becomes https://erp.91miaoshou.com/?redirect=%2Fcommon_collect_box%2Findex%3FfetchType%3DimportCopy
  • Page shows "免费使用" / "立即登录" buttons instead of import interface
  • input[type="file"] count is 0
  • Body text contains no "导入" or "采集" keywords
Reference in New Issue View Git Blame Copy Permalink