first commit
This commit is contained in:
41 files changed
+4964
No files matched your search
+10
@@ -0,0 +1,10 @@
|
|||||||
|
data/
|
||||||
|
downloads/
|
||||||
|
local-records/
|
||||||
|
test/history/
|
||||||
|
test/artifacts/
|
||||||
|
发布去重记录.json
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
.venv/
|
||||||
|
.DS_Store
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
# 新闻工作台
|
||||||
|
|
||||||
|
将新闻下载、文章打包、浏览器导入、排版与换号联动整合到本地控制台。
|
||||||
|
|
||||||
|
**当前版本:0.1。可完成下载 → 入队 → 导入 → 排版 → 等待人工发表。结果页自动核对按用户决定暂留待接入;人工提交后会记录候选结果并暂停,不会自动计数、换号或处理下一篇。** 换号调度和计数接口已完成离线验证,完整在线链路仍需补齐结果页 HTML 后验证。
|
||||||
|
|
||||||
|
## 启动
|
||||||
|
|
||||||
|
双击本目录的 `start.command`,或在本目录运行:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
默认打开 http://127.0.0.1:8770/ 。保留启动窗口,按 Ctrl+C 停止本地服务。修改端口使用 `--port 8772`;不自动打开浏览器使用 `--no-browser`。首次运行 uv 可能下载 Python 或依赖,后续可以复用缓存。
|
||||||
|
|
||||||
|
页面必须通过本地服务打开,不要直接双击 `static/index.html`。Python 依赖仅用于下载和内容处理;浏览器脚本没有远程脚本依赖。
|
||||||
|
|
||||||
|
## 浏览器安装与配对
|
||||||
|
|
||||||
|
1. 在油猴安装控制台“浏览器连接”区域的两个脚本:
|
||||||
|
- `browser/local-bridge.user.js`:本地连接、配对与常驻终止按钮。
|
||||||
|
- `browser/auto-news.user.js`:导入、排版和任务执行。它由 `importer.js`、`formatter.js`、`runner.js` 构建。
|
||||||
|
2. 停用旧的独立导入与排版油猴脚本,避免同一编辑器重复挂载。
|
||||||
|
3. 换号插件仍由用户安装,位于项目根目录的 `dww11-account-switch-browser-extension/`,没有复制到本目录。本次在其 `content.js` 中增加了不含凭据的状态属性及终止事件监听。请在浏览器扩展管理页重新加载**这个原安装目录**,再刷新网站。旧 ZIP 未重新打包,不包含本次修改。
|
||||||
|
4. 在控制台点击“显示本次连接码”。在论坛左下角“新闻工作台 · 本地连接”展开设置,填写端口和连接码并保存。
|
||||||
|
5. 服务重启后连接码会改变,需要重新配对;任务和计数保留,但服务不会自动恢复运行。
|
||||||
|
|
||||||
|
本机 HTTP 通信通过油猴特权请求完成,不要求论坛跨域开放本地服务。真实浏览器的脚本管理器配置、本地网络权限和网站编辑器兼容性仍需首次实机验证。配对码保存在油猴存储,不写入页面全局变量。下载 API Key 不进入文章包,不发给论坛。
|
||||||
|
|
||||||
|
## 操作流程
|
||||||
|
|
||||||
|
1. 填写 Key,获取频道、选择数量并下载。完整保存的文章会自动校验并进入队列;下载失败和入队失败分别显示,已保存的文章不因入队失败而删除。
|
||||||
|
2. 也可展开“导入已有文章目录”,填写单篇目录或含多个文章子目录的根目录,点击检查。留空表示扫描本工具的 `downloads/`。扫描只处理根目录的一层文章子目录;文件须已完整落盘。
|
||||||
|
3. 填写目标版块 `fid`、每账号最少/最多篇数和处理间隔。先核对历史发帖记录,再勾选确认并保存。版块和篇数不会替用户选择。
|
||||||
|
4. 登录论坛,建议只保留一个执行标签。点击控制台“开始 / 继续”。执行器识别账号后领取一篇任务,进入指定新帖页、导入、排版,停止在人工审核阶段。
|
||||||
|
5. 在论坛检查账号、版块、标题、图片位置、正文和来源,并亲手点击“发表帖子”。程序不调用发帖提交函数,不发送发帖请求,也不自动保存草稿。
|
||||||
|
6. 当前版本会记录人工点击和提交快照,在跳转后的候选页面或等待超时后标记“发布结果待核对”并暂停。**需要补充结果页 HTML 后,才能自动核对受理结果并接续下一篇。**
|
||||||
|
|
||||||
|
待补齐结果页自动核对后:审核中算网站已受理一次,但不算已确认公开;达到账号随机目标篇数且仍有待处理文章时,回首页调用插件换号,核对新 UID 与插件空闲状态后创建下一账号批次。没有待处理文章时直接结束,不为达到数量重发。
|
||||||
|
|
||||||
|
## 终止、恢复与冲突
|
||||||
|
|
||||||
|
- 网站左下角常驻“终止流程”,点击后立即取消本标签可取消的导入,并通知其他安装脚本的标签和换号插件;持久状态禁止继续领取任务。
|
||||||
|
- 控制台“终止流程”还会停止下载,浏览器在下一次连接检查时停止(正常轮询约 2 秒)。网络断开时无法承诺固定延迟;网站按钮可立即停止本标签。已到达网站的图片、登录或发帖请求无法撤回。
|
||||||
|
- 暂停/终止后必须明确点击“开始 / 继续”。下载停止等待当前网络请求/单篇保存结束,不伪装成已经撤回在途请求。
|
||||||
|
- 非空编辑器、人工修改、账号变化、缺图、未知来源或上传失败会停在“需要处理”,不弹自动确认覆盖旧稿。
|
||||||
|
- 导入中刷新后不会重新上传。先核对网站附件和草稿,再决定如何处理。若需要换标签,先在控制台暂停,再在目标标签点击“接管已暂停任务”。接管不会自动重新导入内容。
|
||||||
|
- 只有人工确认网站**未受理**且已清理编辑器/草稿后,才能通过队列“人工核对 / 处理”重新排队或移出。审核中、已受理和结果不明不应重试。
|
||||||
|
- 换号插件暂停后需要通过插件现有“继续未完成流程”恢复,不重复抽号。若账号未发生变化,调度器不会猜测换号成功。
|
||||||
|
|
||||||
|
## 文件与数据
|
||||||
|
|
||||||
|
```text
|
||||||
|
app.py / store.py / articles.py 本地接口、任务队列、文章校验
|
||||||
|
modules/downloader/ 下载器副本与成功保存回调
|
||||||
|
modules/package.py 文章包生成器副本
|
||||||
|
browser/ 页面执行器、排版和连接脚本
|
||||||
|
static/ 操作前端
|
||||||
|
start.command 启动入口
|
||||||
|
build_browser.py 重建合并油猴脚本
|
||||||
|
test/ 离线测试、模拟素材、历史测试截图
|
||||||
|
data/queue.sqlite3 运行任务、计数、受理去重记录(本地)
|
||||||
|
downloads/ 本工具下载结果(本地)
|
||||||
|
```
|
||||||
|
|
||||||
|
- `test/history/` 保存从本目录移入的两张演示截图与演示任务 JSON。根目录其他测试文件未搬动。
|
||||||
|
- `发布去重记录.json` 是实际受理证据,继续留在原处,并在服务启动时迁入去重库。它不代表完整历史已对账。原文件可找到时还建立内容指纹;原文件丢失时仍保留原 Markdown 摘要。
|
||||||
|
- 原始下载文件不改写。长标题生成保守 UTF-16 长度不超过 80 的派生标题并加 `...`,原题仍展示在队列;网站真实计数规则尚需实测。
|
||||||
|
- 排版仅处理与文章来源一致的末尾标准尾注,保留正文中的其他“来源”文字。图片顺序、标识改变会回滚排版。
|
||||||
|
- WebP/AVIF/SVG 当前明确拦截,需用户另存支持格式并更新 Markdown;没有静默丢图或自动转码。单图须小于 20 MiB,图片 base64 总大小不超过 48 MiB。
|
||||||
|
- 本地库含文章内容快照和账号标识,不含账号密码、Cookie 或上传凭据。运行数据和历史截图已加入忽略规则。
|
||||||
|
|
||||||
|
## 验证
|
||||||
|
|
||||||
|
不连接新闻 API 或真实论坛:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python -m unittest discover -s test -p 'test_*.py' -v
|
||||||
|
node --test test/importer.test.cjs test/bridge.test.cjs
|
||||||
|
python3 build_browser.py
|
||||||
|
```
|
||||||
|
|
||||||
|
浏览器回归需要 Playwright 和本机 Chrome,使用临时独立浏览器,不使用用户配置或账号:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# 终端一:临时模拟数据服务,端口 8771
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python test/serve_fixture.py
|
||||||
|
# 终端二:Node 环境需能加载 playwright;CHROME_PATH 可指定浏览器可执行文件
|
||||||
|
node test/browser.test.cjs
|
||||||
|
```
|
||||||
|
|
||||||
|
浏览器回归拦截所有真实论坛请求,仅使用模拟 HTML。截图输出到 `test/artifacts/`。本轮离线通过不代表已在用户登录态的真实网站上完成上传、换号或发布。
|
||||||
|
|
||||||
|
## 下一步需要的页面资料
|
||||||
|
|
||||||
|
按用户决定,等待结果页 HTML:帖子标题、首帖作者及 UID 链接、版块及 fid 链接、“审核中”提示、公开帖子与失败提示。不要提供 Cookie、密码或表单令牌。将据此完成 `runner.js` 的结果观察适配,调用已经实现的受理记录接口,联通后续计数和换号。
|
||||||
@@ -0,0 +1,205 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Local controller: serves UI and authenticated browser bridge on loopback only."""
|
||||||
|
import argparse
|
||||||
|
import hmac
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from pathlib import Path
|
||||||
|
import secrets
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
import webbrowser
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
sys.path.insert(0, str(ROOT / 'modules/downloader'))
|
||||||
|
spec = importlib.util.spec_from_file_location('news_downloader', ROOT / 'modules/downloader/app.py')
|
||||||
|
downloader = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(downloader)
|
||||||
|
from store import Store
|
||||||
|
|
||||||
|
|
||||||
|
class Controller:
|
||||||
|
def __init__(self, data=ROOT / 'data', downloads=ROOT / 'downloads', ledger=ROOT / '发布去重记录.json'):
|
||||||
|
self.store = Store(Path(data) / 'queue.sqlite3', ledger)
|
||||||
|
self.ui_token, self.bridge_token = secrets.token_urlsafe(32), secrets.token_urlsafe(32)
|
||||||
|
self.manager = downloader.Downloader(downloads, on_saved=self.saved)
|
||||||
|
self.scan_lock = threading.Lock()
|
||||||
|
|
||||||
|
def saved(self, article, result, keys):
|
||||||
|
directory = Path(result.directory)
|
||||||
|
marker = directory / '.article.json'
|
||||||
|
pending = directory / '.article.json.tmp'
|
||||||
|
pending.write_text(json.dumps({'fingerprints': keys, 'source_url': result.source_url}, ensure_ascii=False), encoding='utf-8')
|
||||||
|
pending.replace(marker)
|
||||||
|
self.store.enqueue(directory)
|
||||||
|
|
||||||
|
def scan(self, root):
|
||||||
|
if self.manager.thread and self.manager.thread.is_alive():
|
||||||
|
raise ValueError('下载进行中,请等待完成后扫描;新下载文章会自动入队')
|
||||||
|
root = Path(root).expanduser().resolve(strict=True)
|
||||||
|
if not root.is_dir():
|
||||||
|
raise ValueError('文章根目录不存在')
|
||||||
|
if not self.scan_lock.acquire(blocking=False):
|
||||||
|
raise ValueError('正在扫描,请稍候')
|
||||||
|
try:
|
||||||
|
directories = [root] if list(root.glob('*.md')) else sorted(p for p in root.iterdir() if p.is_dir() and not p.name.startswith('.'))
|
||||||
|
result = []
|
||||||
|
for directory in directories:
|
||||||
|
try:
|
||||||
|
status = self.store.enqueue(directory)
|
||||||
|
except (ValueError, OSError) as error:
|
||||||
|
status = str(error)
|
||||||
|
result.append({'directory': directory.name, 'status': status})
|
||||||
|
return {'results': result}
|
||||||
|
finally:
|
||||||
|
self.scan_lock.release()
|
||||||
|
|
||||||
|
def ui(self, route, body):
|
||||||
|
if route == 'state':
|
||||||
|
return {**self.store.state(), 'download': self.manager.snapshot()}
|
||||||
|
if route == 'pair':
|
||||||
|
return {'token': self.bridge_token}
|
||||||
|
if route == 'channels':
|
||||||
|
key = str(body.get('key', '')).strip()
|
||||||
|
if not key or len(key) > 512:
|
||||||
|
raise ValueError('请填写 API Key')
|
||||||
|
items = downloader.result_list(downloader.api_request(key, 'channel'))
|
||||||
|
if not items or not all(isinstance(x, str) and x for x in items):
|
||||||
|
raise ValueError('频道响应格式不正确')
|
||||||
|
return {'channels': items}
|
||||||
|
if route == 'download':
|
||||||
|
key, channel, target = str(body.get('key', '')).strip(), body.get('channel'), body.get('target')
|
||||||
|
if not key or len(key) > 512 or not isinstance(channel, str) or not channel.strip() or type(target) is not int or not 1 <= target <= 100000:
|
||||||
|
raise ValueError('请填写 Key、频道和有效下载数量')
|
||||||
|
self.manager.start(key, channel, target)
|
||||||
|
return {'ok': True}
|
||||||
|
if route == 'download-stop':
|
||||||
|
self.manager.stop.set()
|
||||||
|
return {'ok': True}
|
||||||
|
if route == 'scan':
|
||||||
|
return self.scan(body.get('root', '')) if body.get('root') else self.scan(self.manager.root)
|
||||||
|
if route == 'configure':
|
||||||
|
self.store.configure(body)
|
||||||
|
return {'ok': True}
|
||||||
|
if route in ('start', 'pause', 'stop'):
|
||||||
|
if route == 'stop':
|
||||||
|
self.manager.stop.set()
|
||||||
|
return self.store.control(route)
|
||||||
|
if route == 'resolve':
|
||||||
|
return self.store.resolve(body)
|
||||||
|
if route == 'history':
|
||||||
|
records = body.get('records')
|
||||||
|
if not isinstance(records, list):
|
||||||
|
raise ValueError('历史记录需包含 records 数组')
|
||||||
|
self.store.import_history(records)
|
||||||
|
return {'ok': True}
|
||||||
|
raise ValueError('未知操作')
|
||||||
|
|
||||||
|
def bridge(self, route, body):
|
||||||
|
methods = {'heartbeat': self.store.heartbeat, 'next': self.store.next, 'package': self.store.package,
|
||||||
|
'event': self.store.event, 'accept': self.store.accept, 'switch-begin': self.store.switch_begin,
|
||||||
|
'takeover': self.store.takeover, 'current': self.store.current}
|
||||||
|
if route == 'stop':
|
||||||
|
self.manager.stop.set()
|
||||||
|
return self.store.control('stop')
|
||||||
|
if route not in methods:
|
||||||
|
raise ValueError('未知浏览器操作')
|
||||||
|
return methods[route](body)
|
||||||
|
|
||||||
|
|
||||||
|
def make_handler(controller):
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
def log_message(self, *args):
|
||||||
|
pass # Never echo request bodies, API keys or pairing tokens.
|
||||||
|
|
||||||
|
def send(self, code, payload, mime='application/json; charset=utf-8'):
|
||||||
|
content = payload if isinstance(payload, bytes) else json.dumps(payload, ensure_ascii=False).encode()
|
||||||
|
self.send_response(code)
|
||||||
|
self.send_header('Content-Type', mime)
|
||||||
|
self.send_header('Content-Length', str(len(content)))
|
||||||
|
self.send_header('Cache-Control', 'no-store')
|
||||||
|
self.send_header('X-Content-Type-Options', 'nosniff')
|
||||||
|
self.send_header('Content-Security-Policy', "default-src 'self'; script-src 'self'; style-src 'self'; connect-src 'self'; frame-ancestors 'none'; base-uri 'none'")
|
||||||
|
self.end_headers()
|
||||||
|
try:
|
||||||
|
self.wfile.write(content)
|
||||||
|
except (BrokenPipeError, ConnectionResetError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def local(self):
|
||||||
|
return self.headers.get('Host') in (f'127.0.0.1:{self.server.server_port}', f'localhost:{self.server.server_port}')
|
||||||
|
|
||||||
|
def origin_ok(self):
|
||||||
|
origin = self.headers.get('Origin')
|
||||||
|
return not origin or origin in (f'http://127.0.0.1:{self.server.server_port}', f'http://localhost:{self.server.server_port}')
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
if not self.local() or not self.origin_ok():
|
||||||
|
return self.send(403, {'error': '仅允许本机控制台访问'})
|
||||||
|
route = urlsplit(self.path).path
|
||||||
|
if route == '/api/session':
|
||||||
|
return self.send(200, {'token': controller.ui_token})
|
||||||
|
files = {'/': ('static/index.html', 'text/html; charset=utf-8'),
|
||||||
|
'/app.js': ('static/app.js', 'text/javascript; charset=utf-8'),
|
||||||
|
'/style.css': ('static/style.css', 'text/css; charset=utf-8'),
|
||||||
|
'/browser/auto-news.user.js': ('browser/auto-news.user.js', 'text/javascript; charset=utf-8'),
|
||||||
|
'/browser/local-bridge.user.js': ('browser/local-bridge.user.js', 'text/javascript; charset=utf-8')}
|
||||||
|
if route not in files:
|
||||||
|
return self.send(404, {'error': '不存在'})
|
||||||
|
name, mime = files[route]
|
||||||
|
file = ROOT / name
|
||||||
|
if not file.exists():
|
||||||
|
return self.send(404, {'error': '请先运行 build_browser.py'})
|
||||||
|
return self.send(200, file.read_bytes(), mime)
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
if not self.local():
|
||||||
|
return self.send(403, {'error': 'Host 不匹配'})
|
||||||
|
route = urlsplit(self.path).path
|
||||||
|
bridge = route.startswith('/bridge/')
|
||||||
|
expected = controller.bridge_token if bridge else controller.ui_token
|
||||||
|
provided = self.headers.get('X-Session-Token', '')
|
||||||
|
if not hmac.compare_digest(expected, provided) or (not bridge and not self.origin_ok()):
|
||||||
|
return self.send(403, {'error': '连接凭据无效,请重新配对'})
|
||||||
|
if not bridge and not route.startswith('/api/'):
|
||||||
|
return self.send(404, {'error': '不存在'})
|
||||||
|
try:
|
||||||
|
length = int(self.headers.get('Content-Length', 0))
|
||||||
|
if not 0 < length <= 2 * 1024 * 1024:
|
||||||
|
raise ValueError('请求大小无效')
|
||||||
|
body = json.loads(self.rfile.read(length))
|
||||||
|
if not isinstance(body, dict):
|
||||||
|
raise ValueError('请求必须是对象')
|
||||||
|
result = (controller.bridge if bridge else controller.ui)(route.rsplit('/', 1)[1], body)
|
||||||
|
self.send(200, result)
|
||||||
|
except (ValueError, KeyError, OSError) as error:
|
||||||
|
self.send(400, {'error': str(error)[:1000]})
|
||||||
|
except Exception:
|
||||||
|
self.send(500, {'error': '内部错误;请停止当前任务并检查本地服务'})
|
||||||
|
return Handler
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--port', type=int, default=8770)
|
||||||
|
parser.add_argument('--no-browser', action='store_true')
|
||||||
|
args = parser.parse_args()
|
||||||
|
controller = Controller()
|
||||||
|
server = ThreadingHTTPServer(('127.0.0.1', args.port), make_handler(controller))
|
||||||
|
url = f'http://127.0.0.1:{args.port}/'
|
||||||
|
print('新闻工作台:' + url, flush=True)
|
||||||
|
if not args.no_browser:
|
||||||
|
webbrowser.open(url)
|
||||||
|
try:
|
||||||
|
server.serve_forever()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
controller.manager.stop.set()
|
||||||
|
controller.store.control('stop')
|
||||||
|
finally:
|
||||||
|
server.server_close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
+116
@@ -0,0 +1,116 @@
|
|||||||
|
"""Local article validation and portable packages. Never changes downloaded files."""
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import unicodedata
|
||||||
|
from urllib.parse import unquote
|
||||||
|
|
||||||
|
from modules.package import make_package
|
||||||
|
|
||||||
|
MAX_PACKAGE_BYTES = 48 * 1024 * 1024
|
||||||
|
FOOTER = re.compile(r'(?:^|\n)来源:([^\n]+),如有侵权,请联系删除。\s*$')
|
||||||
|
|
||||||
|
|
||||||
|
def validate_images(body, directory):
|
||||||
|
"""Conservative inline-image preflight; the importer remains the final parser."""
|
||||||
|
if re.search(r'!\[[^\]]*\]\s*\[', body):
|
||||||
|
raise ValueError('不支持引用式图片,请改成内联图片路径')
|
||||||
|
for start in re.finditer(r'!\[[^\]]*\]\(', body):
|
||||||
|
pos, depth, quote, angle = start.end(), 1, '', False
|
||||||
|
begin = pos
|
||||||
|
while pos < len(body):
|
||||||
|
ch = body[pos]
|
||||||
|
if ch == '\\':
|
||||||
|
pos += 2
|
||||||
|
continue
|
||||||
|
if quote:
|
||||||
|
if ch == quote:
|
||||||
|
quote = ''
|
||||||
|
elif not angle and depth == 1 and body[pos - 1].isspace() and ch in ('"', "'"):
|
||||||
|
quote = ch
|
||||||
|
elif ch == '<':
|
||||||
|
angle = True
|
||||||
|
elif ch == '>':
|
||||||
|
angle = False
|
||||||
|
elif not angle and ch == '(':
|
||||||
|
depth += 1
|
||||||
|
elif not angle and ch == ')':
|
||||||
|
depth -= 1
|
||||||
|
if not depth:
|
||||||
|
break
|
||||||
|
pos += 1
|
||||||
|
if depth:
|
||||||
|
raise ValueError('图片引用括号不完整')
|
||||||
|
target = body[begin:pos].strip()
|
||||||
|
target = target[1:target.index('>')] if target.startswith('<') and '>' in target else re.sub(r'\s+["\'][\s\S]*["\']\s*$', '', target)
|
||||||
|
target = unquote(re.sub(r'\\([()\\ ])', r'\1', target))
|
||||||
|
if re.match(r'^(?:[a-zA-Z][\w+.-]*:|/|\\)', target):
|
||||||
|
raise ValueError('图片必须使用目录内相对路径:' + target)
|
||||||
|
image = (directory / target).resolve()
|
||||||
|
if not image.is_relative_to(directory) or not image.is_file():
|
||||||
|
raise ValueError('缺图或图片路径超出目录:' + target)
|
||||||
|
if image.suffix.lower() not in ('.jpg', '.jpeg', '.png', '.gif'):
|
||||||
|
raise ValueError('不支持的配图格式:' + target)
|
||||||
|
if not 0 < image.stat().st_size < 20 * 1024 * 1024:
|
||||||
|
raise ValueError('配图为空或达到 20 MiB:' + target)
|
||||||
|
|
||||||
|
|
||||||
|
def short_title(title, budget=80):
|
||||||
|
# Conservative UTF-16 budget: do not split astral characters / surrogate pairs.
|
||||||
|
if len(title.encode('utf-16-le')) // 2 <= budget:
|
||||||
|
return title
|
||||||
|
out = ''
|
||||||
|
for ch in title:
|
||||||
|
if len((out + ch).encode('utf-16-le')) // 2 > budget - 3:
|
||||||
|
break
|
||||||
|
out += ch
|
||||||
|
return out + '...'
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_article(directory):
|
||||||
|
directory = Path(directory).resolve(strict=True)
|
||||||
|
candidates = sorted(directory.glob('*.md'))
|
||||||
|
if len(candidates) != 1:
|
||||||
|
raise ValueError('每个文章目录必须恰好包含一个 Markdown 文件')
|
||||||
|
md = candidates[0]
|
||||||
|
if not md.resolve().is_relative_to(directory):
|
||||||
|
raise ValueError('Markdown 不得链接到目录外')
|
||||||
|
raw = md.read_text(encoding='utf-8-sig')
|
||||||
|
heading = re.match(r'^\s*#\s+([^\n]+)\n?', raw)
|
||||||
|
title = (heading[1] if heading else md.stem).strip()
|
||||||
|
body = raw[heading.end():].strip() if heading else raw.strip()
|
||||||
|
footer = FOOTER.search(body)
|
||||||
|
if not footer:
|
||||||
|
raise ValueError('文末缺少标准来源尾注,请补充来源后再扫描')
|
||||||
|
source = footer[1].strip()
|
||||||
|
body = body[:footer.start()].strip()
|
||||||
|
if not title or not body:
|
||||||
|
raise ValueError('标题或正文为空')
|
||||||
|
validate_images(body, directory)
|
||||||
|
for file in directory.rglob('*'):
|
||||||
|
if file.suffix.lower() in ('.webp', '.avif', '.svg'):
|
||||||
|
raise ValueError('发现暂不支持的图片格式:' + file.name + ';请另存为 JPG/PNG/GIF 并更新引用')
|
||||||
|
pack = make_package(directory)
|
||||||
|
if sum(len(i['base64']) for i in pack['images']) > MAX_PACKAGE_BYTES:
|
||||||
|
raise ValueError('文章包超过 48 MiB,请减少单篇图片体积')
|
||||||
|
pack['title'] = short_title(title)
|
||||||
|
# Preserve body and exact final footer for the formatter; the original file stays intact.
|
||||||
|
pack['markdown'] = '# ' + pack['title'] + '\n\n' + body + '\n\n来源:' + source + ',如有侵权,请联系删除。\n'
|
||||||
|
content = unicodedata.normalize('NFC', body)
|
||||||
|
# Hash file contents, not directory/title; renamed copies share their article identity.
|
||||||
|
image_hashes = []
|
||||||
|
for item in pack['images']:
|
||||||
|
import base64
|
||||||
|
digest = hashlib.sha256(base64.b64decode(item['base64'])).hexdigest()
|
||||||
|
content = content.replace(item['path'], 'image:' + digest)
|
||||||
|
image_hashes.append(digest)
|
||||||
|
identity = hashlib.sha256((content + '\n' + '\n'.join(sorted(set(image_hashes)))).encode()).hexdigest()
|
||||||
|
metadata = {}
|
||||||
|
marker = directory / '.article.json'
|
||||||
|
if marker.is_file():
|
||||||
|
metadata = json.loads(marker.read_text())
|
||||||
|
return dict(article_key=identity, markdown_sha256=hashlib.sha256(md.read_bytes()).hexdigest(),
|
||||||
|
directory=str(directory), original_title=title, title=pack['title'], source=source,
|
||||||
|
image_count=len(pack['images']), package=pack,
|
||||||
|
source_keys=metadata.get('fingerprints', []))
|
||||||
@@ -0,0 +1,850 @@
|
|||||||
|
// ==UserScript==
|
||||||
|
// @name 新闻工作台 · 导入排版与流程
|
||||||
|
// @namespace local.dww11.auto-news
|
||||||
|
// @version 0.1.0
|
||||||
|
// @match https://www.dww11.com/*
|
||||||
|
// @match https://dww11.com/*
|
||||||
|
// @match https://house.dww11.com/*
|
||||||
|
// @grant none
|
||||||
|
// @run-at document-idle
|
||||||
|
// @noframes
|
||||||
|
// ==/UserScript==
|
||||||
|
|
||||||
|
|
||||||
|
(function (root, factory) {
|
||||||
|
'use strict';
|
||||||
|
const core = factory();
|
||||||
|
if (typeof module === 'object' && module.exports) module.exports = core;
|
||||||
|
else core.mount(root);
|
||||||
|
})(typeof window === 'undefined' ? null : window, function () {
|
||||||
|
'use strict';
|
||||||
|
const VERSION = '1.0.0';
|
||||||
|
const MAX_IMAGE_SIZE = 20 * 1024 * 1024;
|
||||||
|
const IMAGE_EXT = /\.(?:jpe?g|png|gif)$/i;
|
||||||
|
|
||||||
|
function normalizePath(path) {
|
||||||
|
const parts = [];
|
||||||
|
const value = String(path).replace(/\\/g, '/').normalize('NFC');
|
||||||
|
if (/^(?:\/|[a-z]:|[a-z]+:)/i.test(value)) throw new Error('请使用目录内的相对路径:' + path);
|
||||||
|
for (const part of value.split('/')) {
|
||||||
|
if (!part || part === '.') continue;
|
||||||
|
if (part === '..') {
|
||||||
|
if (!parts.length) throw new Error('图片路径超出所选目录:' + path);
|
||||||
|
parts.pop();
|
||||||
|
} else parts.push(part);
|
||||||
|
}
|
||||||
|
return parts.join('/');
|
||||||
|
}
|
||||||
|
|
||||||
|
function imagePath(mdPath, target) {
|
||||||
|
if (/^(?:[a-z][\w+.-]*:|\/\/|\/)/i.test(target)) throw new Error('图片必须在所选目录内,请先下载:' + target);
|
||||||
|
let decoded;
|
||||||
|
try { decoded = decodeURIComponent(target); } catch { throw new Error('图片路径编码无效:' + target); }
|
||||||
|
return normalizePath(mdPath.slice(0, mdPath.lastIndexOf('/') + 1) + decoded);
|
||||||
|
}
|
||||||
|
|
||||||
|
// A small, intentionally bounded Markdown reader. Raw HTML is never executed.
|
||||||
|
// Links/images support balanced parentheses, angle destinations and optional titles.
|
||||||
|
function readLink(text, start) {
|
||||||
|
const image = text.startsWith('![', start);
|
||||||
|
if (!image && text[start] !== '[') return null;
|
||||||
|
const left = start + (image ? 2 : 1);
|
||||||
|
let end = left;
|
||||||
|
for (; end < text.length; end++) {
|
||||||
|
if (text[end] === '\\') { end++; continue; }
|
||||||
|
if (text[end] === ']') break;
|
||||||
|
}
|
||||||
|
if (text.slice(end, end + 2) !== '](') return null;
|
||||||
|
let pos = end + 2, depth = 1, quoted = '', angle = false;
|
||||||
|
for (; pos < text.length; pos++) {
|
||||||
|
const ch = text[pos];
|
||||||
|
if (ch === '\\') { pos++; continue; }
|
||||||
|
if (quoted) { if (ch === quoted) quoted = ''; continue; }
|
||||||
|
if (!angle && depth === 1 && /\s/.test(text[pos - 1]) && (ch === '"' || ch === "'")) { quoted = ch; continue; }
|
||||||
|
if (ch === '<') { angle = true; continue; }
|
||||||
|
if (ch === '>') { angle = false; continue; }
|
||||||
|
if (!angle && ch === '(') depth++;
|
||||||
|
if (!angle && ch === ')' && --depth === 0) break;
|
||||||
|
}
|
||||||
|
if (depth) return null;
|
||||||
|
let target = text.slice(end + 2, pos).trim();
|
||||||
|
const destination = target.match(/^<([^>]+)>\s*(?:["'][\s\S]*["'])?$/);
|
||||||
|
target = destination ? destination[1] : target.replace(/\s+["'][\s\S]*["']\s*$/, '');
|
||||||
|
target = target.replace(/\\([()\\ ])/g, '$1');
|
||||||
|
return { image, label: text.slice(left, end), target, end: pos + 1 };
|
||||||
|
}
|
||||||
|
|
||||||
|
function inline(text, onImage) {
|
||||||
|
let out = '';
|
||||||
|
for (let pos = 0; pos < text.length;) {
|
||||||
|
if (text[pos] === '\\' && pos + 1 < text.length) { out += text[pos + 1]; pos += 2; continue; }
|
||||||
|
const link = readLink(text, pos);
|
||||||
|
if (link) {
|
||||||
|
if (link.image) out += onImage(link.target, link.label);
|
||||||
|
else if (/^(https?:\/\/|mailto:)/i.test(link.target) && !/[\[\]\r\n"']/.test(link.target)) {
|
||||||
|
out += '[url=' + link.target + ']' + inline(link.label, onImage) + '[/url]';
|
||||||
|
} else out += link.label + ' (' + link.target + ')';
|
||||||
|
pos = link.end;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (text[pos] === '`') {
|
||||||
|
const end = text.indexOf('`', pos + 1);
|
||||||
|
if (end > pos + 1) { out += text.slice(pos + 1, end); pos = end + 1; continue; }
|
||||||
|
}
|
||||||
|
let matched = false;
|
||||||
|
for (const [mark, tag] of [['**', 'b'], ['__', 'b'], ['~~', 's'], ['*', 'i'], ['_', 'i']]) {
|
||||||
|
if (!text.startsWith(mark, pos) || (mark === '_' && /\w/.test(text[pos - 1] || ''))) continue;
|
||||||
|
const end = text.indexOf(mark, pos + mark.length);
|
||||||
|
if (end <= pos + mark.length) continue;
|
||||||
|
out += '[' + tag + ']' + inline(text.slice(pos + mark.length, end), onImage) + '[/' + tag + ']';
|
||||||
|
pos = end + mark.length; matched = true; break;
|
||||||
|
}
|
||||||
|
if (!matched) out += text[pos++];
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function markdownToBBCode(markdown, onImage) {
|
||||||
|
const lines = markdown.replace(/\r\n?/g, '\n').split('\n');
|
||||||
|
const output = [];
|
||||||
|
let list = '', quote = false, code = false, fence = '';
|
||||||
|
const closeList = () => { if (list) output.push('[/list]'); list = ''; };
|
||||||
|
for (let index = 0; index < lines.length; index++) {
|
||||||
|
const line = lines[index];
|
||||||
|
const codeMark = line.match(/^\s*(`{3,}|~{3,})/);
|
||||||
|
if (codeMark && (!code || codeMark[1][0] === fence)) {
|
||||||
|
closeList();
|
||||||
|
output.push(code ? '[/code]' : '[code]');
|
||||||
|
code = !code; fence = codeMark[1][0]; continue;
|
||||||
|
}
|
||||||
|
if (code) { output.push(line.replace(/\[\/code\]/gi, '[/code]')); continue; }
|
||||||
|
const quoted = /^\s*>\s?/.test(line);
|
||||||
|
if (quoted !== quote) { closeList(); output.push(quoted ? '[quote]' : '[/quote]'); quote = quoted; }
|
||||||
|
const content = quoted ? line.replace(/^\s*>\s?/, '') : line;
|
||||||
|
const item = content.match(/^\s*(?:([-+*])|\d+[.)])\s+(.+)$/);
|
||||||
|
if (item) {
|
||||||
|
const type = item[1] ? '[list]' : '[list=1]';
|
||||||
|
if (list !== type) { closeList(); output.push(type); list = type; }
|
||||||
|
output.push('[*]' + inline(item[2], onImage)); continue;
|
||||||
|
}
|
||||||
|
closeList();
|
||||||
|
const heading = content.match(/^\s{0,3}(#{1,6})\s+(.+?)\s*#*$/);
|
||||||
|
if (heading) output.push('[b]' + inline(heading[2], onImage) + '[/b]');
|
||||||
|
else if (/^\s*(?:---+|\*\*\*+|___+)\s*$/.test(content)) output.push('[hr]');
|
||||||
|
else output.push(inline(content, onImage));
|
||||||
|
}
|
||||||
|
closeList();
|
||||||
|
if (quote) output.push('[/quote]');
|
||||||
|
if (code) output.push('[/code]');
|
||||||
|
return output.join('\n').trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function compile(markdown, mdPath, fileMap, titleOverride) {
|
||||||
|
let body = String(markdown).replace(/^\uFEFF/, '').replace(/\r\n?/g, '\n');
|
||||||
|
const firstHeading = body.match(/^\s*#\s+([^\n]+)\n?/);
|
||||||
|
const title = String(titleOverride || firstHeading?.[1] || mdPath.split('/').pop().replace(/\.md$/i, '')).trim();
|
||||||
|
if (firstHeading) body = body.slice(firstHeading[0].length).trim();
|
||||||
|
if (!title || Array.from(title).length > 80) throw new Error('标题为空或超过 80 字,请修改 Markdown 标题后重试。');
|
||||||
|
if (!body.trim()) throw new Error('正文为空。');
|
||||||
|
const warnings = [];
|
||||||
|
if (/<\/?[a-z][^>]*>/i.test(body)) warnings.push('存在 HTML,将作为文字保留;请检查排版。');
|
||||||
|
if (/^\s*\|.*\|\s*$/m.test(body)) warnings.push('表格暂按原文保留,需手动检查。');
|
||||||
|
if (/!\[[^\]]*\]\s*\[[^\]]*\]/.test(body) || /^\s*\[[^\]]+\]:/m.test(body)) {
|
||||||
|
throw new Error('暂不支持引用式 Markdown 图片/链接,请改成  或 [文字](网址)。');
|
||||||
|
}
|
||||||
|
const images = new Map();
|
||||||
|
// Validate every image before any network request; repeated references upload once.
|
||||||
|
markdownToBBCode(body, (target) => {
|
||||||
|
const path = imagePath(mdPath, target);
|
||||||
|
const file = fileMap.get(path);
|
||||||
|
if (!file) throw new Error('找不到配图:' + path);
|
||||||
|
if (!IMAGE_EXT.test(path)) throw new Error('网站仅支持 JPG、JPEG、PNG、GIF:' + path);
|
||||||
|
if (!file.size || file.size >= MAX_IMAGE_SIZE) throw new Error('配图为空或达到 20 MB:' + path);
|
||||||
|
images.set(path, file);
|
||||||
|
return '[图片:' + path + ']';
|
||||||
|
});
|
||||||
|
return { title, body, mdPath, images, warnings,
|
||||||
|
render(aids) {
|
||||||
|
return markdownToBBCode(body, target => {
|
||||||
|
const path = imagePath(mdPath, target);
|
||||||
|
const aid = aids.get(path);
|
||||||
|
if (!/^[1-9]\d*$/.test(String(aid))) throw new Error('配图尚未上传:' + path);
|
||||||
|
return '[attachimg]' + aid + '[/attachimg]';
|
||||||
|
});
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function prepareFiles(files, markdownPath) {
|
||||||
|
const fileMap = new Map();
|
||||||
|
for (const file of Array.from(files)) {
|
||||||
|
const path = normalizePath(file.webkitRelativePath || file.name);
|
||||||
|
if (fileMap.has(path)) throw new Error('文件路径重复:' + path);
|
||||||
|
fileMap.set(path, file);
|
||||||
|
}
|
||||||
|
const candidates = Array.from(fileMap.keys()).filter(p => /\.md$/i.test(p)).sort();
|
||||||
|
if (!candidates.length) throw new Error('所选目录中没有 .md 文件。');
|
||||||
|
if (!markdownPath && candidates.length > 1) return { candidates, files: Array.from(files) };
|
||||||
|
const mdPath = markdownPath || candidates[0];
|
||||||
|
const md = fileMap.get(mdPath);
|
||||||
|
if (!md) throw new Error('指定的 Markdown 文件不存在。');
|
||||||
|
return compile(await md.text(), mdPath, fileMap);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function preparePackage(data) {
|
||||||
|
if (!data || data.schemaVersion !== 1 || typeof data.markdown !== 'string' || !Array.isArray(data.images)) {
|
||||||
|
throw new Error('文章包格式错误,需要 schemaVersion: 1、markdown 和 images。');
|
||||||
|
}
|
||||||
|
const map = new Map();
|
||||||
|
for (const item of data.images) {
|
||||||
|
const path = normalizePath(item.path);
|
||||||
|
if (map.has(path)) throw new Error('文章包图片路径重复:' + path);
|
||||||
|
let blob = item.blob;
|
||||||
|
if (!blob && typeof item.base64 === 'string') {
|
||||||
|
const binary = atob(item.base64);
|
||||||
|
blob = new Blob([Uint8Array.from(binary, ch => ch.charCodeAt(0))], { type: item.mime || 'application/octet-stream' });
|
||||||
|
}
|
||||||
|
if (!blob || typeof blob.arrayBuffer !== 'function') throw new Error('文章包缺少图片数据:' + path);
|
||||||
|
map.set(path, blob);
|
||||||
|
}
|
||||||
|
return compile(data.markdown, normalizePath(data.markdownPath || 'article.md'), map, data.title);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseAid(text) {
|
||||||
|
const value = String(text).trim();
|
||||||
|
// The site's actual WebUploader adapter uses a positive integer response.
|
||||||
|
if (!/^(?:[1-9]\d*|"[1-9]\d*")$/.test(value)) throw new Error('服务器未返回有效附件编号(可能登录过期、超出权限或上传失败)。');
|
||||||
|
return value.replace(/"/g, '');
|
||||||
|
}
|
||||||
|
|
||||||
|
function mount(win) {
|
||||||
|
const doc = win.document;
|
||||||
|
const params = new URLSearchParams(win.location.search);
|
||||||
|
if (params.get('mod') !== 'post' || params.get('action') !== 'newthread' || !doc.querySelector('#subject') || !doc.querySelector('#e_textarea')) return;
|
||||||
|
if (win.DWW11Importer) return;
|
||||||
|
let article = null, busy = false, candidates = null, lastResult = null, savedBackup = null, stageRevision = 0;
|
||||||
|
const cache = new Map(), urls = [], logs = [];
|
||||||
|
const host = doc.createElement('div');
|
||||||
|
host.id = 'dww11-importer';
|
||||||
|
host.style.cssText = 'position:fixed;right:18px;bottom:18px;z-index:2147483000';
|
||||||
|
const shadow = host.attachShadow({ mode: 'open' });
|
||||||
|
shadow.innerHTML = `<style>
|
||||||
|
:host{all:initial}*{box-sizing:border-box}details{font:14px/1.6 system-ui,sans-serif;color:#203044;background:#fff;border:1px solid #d4dee8;border-radius:12px;box-shadow:0 6px 28px #0003;width:360px;max-width:92vw}summary{cursor:pointer;padding:12px 16px;font-weight:650}section{padding:0 16px 16px;max-height:70vh;overflow:auto}button,select{font:inherit;padding:7px 10px;border-radius:6px;border:1px solid #b8c9da;background:#f4f8fc;color:#203044;cursor:pointer;margin:4px 4px 4px 0}button.primary{background:#1767b3;color:white;border-color:#1767b3}button:disabled{opacity:.5;cursor:default}p{margin:8px 0}small{color:#617183}pre{font:12px/1.6 system-ui;white-space:pre-wrap;overflow-wrap:anywhere;max-height:170px;overflow:auto;background:#f4f6f9;padding:8px}img{max-width:90px;max-height:80px;margin:4px;object-fit:contain}.error{color:#b32424}.success{color:#18713e}select{max-width:100%}[hidden]{display:none!important}
|
||||||
|
</style><details open><summary>本地文章导入 · 1.0</summary><section>
|
||||||
|
<small>选择整个文章文件夹,检查后上传配图并填写。发布由你操作。网站自带的自动保存仍可能运行。</small>
|
||||||
|
<p><button id="choose" type="button">选择文章目录</button><button id="package" type="button">导入文章包</button></p>
|
||||||
|
<input id="folder" type="file" webkitdirectory multiple hidden><input id="json" type="file" accept=".json,application/json" hidden>
|
||||||
|
<select id="md" hidden aria-label="选择 Markdown"></select>
|
||||||
|
<p id="status" role="status" aria-live="polite">尚未选择目录</p><div id="preview" hidden><strong id="title"></strong><pre id="body"></pre><div id="images"></div><p id="warnings"></p></div>
|
||||||
|
<button id="import" class="primary" type="button" disabled>上传配图并填写</button>
|
||||||
|
<button id="diagnostics" type="button">导出诊断</button><pre id="log" hidden></pre>
|
||||||
|
</section></details>`;
|
||||||
|
doc.body.appendChild(host);
|
||||||
|
const $ = id => shadow.getElementById(id);
|
||||||
|
const summary = () => ({ version: VERSION, busy, ready: !!article, title: article?.title || null,
|
||||||
|
markdownPath: article?.mdPath || null, imageCount: article?.images.size || 0,
|
||||||
|
warnings: article?.warnings || [], result: lastResult });
|
||||||
|
function notify(phase, message, extra = {}) {
|
||||||
|
$('status').textContent = message;
|
||||||
|
$('status').className = phase === 'error' ? 'error' : phase === 'complete' ? 'success' : '';
|
||||||
|
logs.push({ time: new Date().toISOString(), phase, message, ...extra });
|
||||||
|
$('log').hidden = false;
|
||||||
|
$('log').textContent = logs.slice(-8).map(l => l.message).join('\n');
|
||||||
|
win.dispatchEvent(new CustomEvent('dww11-importer:progress', { detail: { phase, message, ...extra } }));
|
||||||
|
}
|
||||||
|
function releasePreviews() { while (urls.length) URL.revokeObjectURL(urls.pop()); }
|
||||||
|
function clearStage() {
|
||||||
|
article = null; lastResult = null; candidates = null; releasePreviews();
|
||||||
|
$('preview').hidden = true; $('md').hidden = true; $('import').disabled = true;
|
||||||
|
}
|
||||||
|
function showArticle(prepared) {
|
||||||
|
article = prepared; lastResult = null; releasePreviews();
|
||||||
|
$('preview').hidden = false;
|
||||||
|
$('title').textContent = article.title;
|
||||||
|
$('body').textContent = article.body;
|
||||||
|
$('warnings').textContent = article.warnings.join(';');
|
||||||
|
$('images').replaceChildren();
|
||||||
|
for (const [path, blob] of article.images) {
|
||||||
|
const img = doc.createElement('img');
|
||||||
|
img.src = URL.createObjectURL(blob); urls.push(img.src); img.alt = path; img.title = path;
|
||||||
|
$('images').appendChild(img);
|
||||||
|
}
|
||||||
|
$('import').disabled = false;
|
||||||
|
notify('ready', `已检查:${article.images.size} 张图片。点击“上传配图并填写”开始导入。`);
|
||||||
|
return summary();
|
||||||
|
}
|
||||||
|
async function stageFiles(files, options = {}) {
|
||||||
|
if (busy) throw new Error('正在导入,请稍候。');
|
||||||
|
clearStage();
|
||||||
|
const revision = ++stageRevision;
|
||||||
|
const list = Array.from(files);
|
||||||
|
const prepared = await prepareFiles(list, options.markdownPath);
|
||||||
|
if (revision !== stageRevision) return { cancelled: true };
|
||||||
|
if (prepared.candidates) {
|
||||||
|
candidates = prepared.files;
|
||||||
|
$('md').replaceChildren(new Option('请选择目录中的 Markdown 文件', ''));
|
||||||
|
for (const path of prepared.candidates) $('md').appendChild(new Option(path, path));
|
||||||
|
$('md').hidden = false;
|
||||||
|
notify('choose-markdown', '目录中有多个 Markdown 文件,请选择一篇。');
|
||||||
|
return { candidates: prepared.candidates };
|
||||||
|
}
|
||||||
|
return showArticle(prepared);
|
||||||
|
}
|
||||||
|
async function stagePackage(data) {
|
||||||
|
if (busy) throw new Error('正在导入,请稍候。');
|
||||||
|
clearStage();
|
||||||
|
const revision = ++stageRevision;
|
||||||
|
const prepared = await preparePackage(data);
|
||||||
|
if (revision !== stageRevision) return { cancelled: true };
|
||||||
|
return showArticle(prepared);
|
||||||
|
}
|
||||||
|
const snapshot = () => ({ title: doc.querySelector('#subject').value,
|
||||||
|
mode: Number(win.wysiwyg), text: win.getEditorContents() });
|
||||||
|
function assertEditor() {
|
||||||
|
for (const name of ['getEditorContents', 'switchEditor']) {
|
||||||
|
if (typeof win[name] !== 'function') throw new Error('未识别到网站编辑器:' + name + '。请刷新页面后重试。');
|
||||||
|
}
|
||||||
|
if (!win.imgUpload?.settings || typeof win.updateImageList !== 'function') throw new Error('网站图片上传组件尚未就绪,请刷新后重试。');
|
||||||
|
if (Number(win.allowbbcode) === 0 || Number(win.allowimgcode) === 0) throw new Error('当前版块或账号不支持正文格式/图片。');
|
||||||
|
}
|
||||||
|
async function uploadImage(blob, path, settings, signal) {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
const url = new URL(settings.upload_url, win.location.href);
|
||||||
|
if (url.origin !== win.location.origin || url.pathname !== '/misc.php' || url.searchParams.get('operation') !== 'upload') {
|
||||||
|
throw new Error('上传地址与预期不符,已停止。');
|
||||||
|
}
|
||||||
|
const hash = Array.from(new Uint8Array(await win.crypto.subtle.digest('SHA-256', await blob.arrayBuffer())), n => n.toString(16).padStart(2, '0')).join('');
|
||||||
|
const key = url.href + ':' + settings.post_params.uid + ':' + hash;
|
||||||
|
if (cache.has(key)) { notify('upload', '复用已上传图片:' + path); return cache.get(key); }
|
||||||
|
const data = new FormData();
|
||||||
|
for (const [key, value] of Object.entries(settings.post_params)) data.append(key, String(value));
|
||||||
|
data.set('type', 'image');
|
||||||
|
data.set('filetype', blob.type || (/\.png$/i.test(path) ? 'image/png' : /\.gif$/i.test(path) ? 'image/gif' : 'image/jpeg'));
|
||||||
|
data.append(settings.file_post_name || 'Filedata', blob, path.split('/').pop());
|
||||||
|
const control = new AbortController(), timer = setTimeout(() => control.abort(), 90000);
|
||||||
|
try {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
signal?.addEventListener('abort', () => control.abort(), { once: true });
|
||||||
|
const response = await win.fetch(url.href, { method: 'POST', body: data, credentials: 'same-origin', redirect: 'error', signal: control.signal });
|
||||||
|
if (!response.ok) throw new Error('图片上传请求失败:HTTP ' + response.status);
|
||||||
|
const aid = parseAid(await response.text());
|
||||||
|
cache.set(key, aid);
|
||||||
|
return aid;
|
||||||
|
} catch (error) {
|
||||||
|
if (error.name === 'AbortError') throw new Error('上传超时;该图片可能已到达服务器,请检查网站图片列表后再重试。');
|
||||||
|
throw error;
|
||||||
|
} finally { clearTimeout(timer); }
|
||||||
|
}
|
||||||
|
async function registerImages(aids, signal) {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
if (!aids.length) return;
|
||||||
|
const ids = new Set(aids);
|
||||||
|
doc.querySelectorAll('#imgattachlist [id^="image_td_"],#imgattachlist [id^="image_"]').forEach(el => {
|
||||||
|
const id = el.id.match(/^(?:image_td_|image_)(\d+)$/)?.[1]; if (id) ids.add(id);
|
||||||
|
});
|
||||||
|
const ready = () => aids.every(id => doc.getElementById('image_' + id));
|
||||||
|
if (ready()) return;
|
||||||
|
// Use the site's own list loader, which registers attachment form fields too.
|
||||||
|
win.updateImageList(1, Array.from(ids).join('|'));
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
const timer = setTimeout(() => { observer.disconnect(); reject(new Error('图片已上传,但网站图片列表加载超时。重试会复用本页已上传图片。')); }, 25000);
|
||||||
|
signal?.addEventListener('abort', () => { clearTimeout(timer); observer.disconnect(); reject(new Error('流程已终止')); }, { once: true });
|
||||||
|
const check = () => { if (ready()) { clearTimeout(timer); observer.disconnect(); resolve(); } };
|
||||||
|
const observer = new MutationObserver(check);
|
||||||
|
observer.observe(doc.body, { childList: true, subtree: true }); check();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function setEditor(title, bbcode) {
|
||||||
|
win.switchEditor(0);
|
||||||
|
if (Number(win.wysiwyg) !== 0) throw new Error('无法切换编辑器为纯文本模式。');
|
||||||
|
const textarea = doc.querySelector('#e_textarea');
|
||||||
|
textarea.value = bbcode;
|
||||||
|
doc.querySelector('#subject').value = title;
|
||||||
|
for (const el of [textarea, doc.querySelector('#subject')]) {
|
||||||
|
el.dispatchEvent(new Event('input', { bubbles: true })); el.dispatchEvent(new Event('change', { bubbles: true }));
|
||||||
|
}
|
||||||
|
// Let Discuz generate the WYSIWYG content with native attachment metadata.
|
||||||
|
win.switchEditor(1);
|
||||||
|
const switcher = doc.querySelector('#e_switchercheck');
|
||||||
|
if (switcher) switcher.checked = Number(win.wysiwyg) === 0;
|
||||||
|
}
|
||||||
|
async function importPrepared(options = {}) {
|
||||||
|
const { signal, automatic = false } = options;
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
if (busy) throw new Error('正在导入,请勿重复点击。');
|
||||||
|
if (!article) throw new Error('请先选择文章目录或文章包。');
|
||||||
|
assertEditor();
|
||||||
|
for (const id of ['bbcodeoff', 'htmlon']) {
|
||||||
|
if (doc.getElementById(id)?.checked) throw new Error('请先关闭“禁用 Discuz! 代码”和“HTML 代码”,再导入。');
|
||||||
|
}
|
||||||
|
const before = snapshot();
|
||||||
|
if ((before.title.trim() || before.text.replace(/<[^>]*>/g, '').trim() || /<img\b/i.test(before.text)) &&
|
||||||
|
(automatic ? (() => { throw new Error('编辑器已有内容,请人工处理旧稿后重试'); })() : !win.confirm('当前编辑器已有内容。导入将替换标题和正文,是否继续?'))) return { cancelled: true };
|
||||||
|
const selected = article;
|
||||||
|
busy = true;
|
||||||
|
const controls = ['choose', 'package', 'import', 'md'].map($);
|
||||||
|
controls.forEach(el => { el.disabled = true; });
|
||||||
|
// Prevent accidental submission of a partly imported article.
|
||||||
|
const blockSubmit = event => { event.preventDefault(); event.stopImmediatePropagation(); notify('busy', '正在导入,请等待完成后再操作发布。'); };
|
||||||
|
const form = doc.querySelector('#postform');
|
||||||
|
form.addEventListener('submit', blockSubmit, true);
|
||||||
|
const submitButtons = Array.from(form.querySelectorAll('button[type="submit"],button[name],input[type="submit"]'));
|
||||||
|
const previousDisabled = submitButtons.map(el => el.disabled);
|
||||||
|
submitButtons.forEach(el => { el.disabled = true; });
|
||||||
|
let committed = false;
|
||||||
|
try {
|
||||||
|
const aids = new Map();
|
||||||
|
const settings = win.imgUpload.settings;
|
||||||
|
const max = Math.min(MAX_IMAGE_SIZE, Number(settings.file_size_limit) * 1024 || MAX_IMAGE_SIZE);
|
||||||
|
for (const [path, blob] of selected.images) if (blob.size >= max) throw new Error('配图超过当前网站限制:' + path);
|
||||||
|
let index = 0;
|
||||||
|
for (const [path, blob] of selected.images) {
|
||||||
|
notify('upload', `上传 ${++index}/${selected.images.size}:${path}`, { completed: index - 1, total: selected.images.size });
|
||||||
|
aids.set(path, await uploadImage(blob, path, settings, signal));
|
||||||
|
}
|
||||||
|
notify('register', '正在加载网站图片列表…');
|
||||||
|
await registerImages(Array.from(new Set(aids.values())), signal);
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
const now = snapshot();
|
||||||
|
if ((now.title !== before.title || now.text !== before.text) && (automatic ? (() => { throw new Error('上传期间编辑器被修改,已保留当前内容'); })() : !win.confirm('上传期间你修改了编辑器。是否用导入文章替换最新内容?'))) {
|
||||||
|
notify('cancelled', '已保留当前正文;图片已上传,可稍后再次导入。'); return { cancelled: true };
|
||||||
|
}
|
||||||
|
const backup = snapshot();
|
||||||
|
const bbcode = selected.render(aids);
|
||||||
|
// Preserve a local, downloadable snapshot before changing title or body.
|
||||||
|
savedBackup = { title: backup.title, content: backup.text, mode: backup.mode, createdAt: new Date().toISOString() };
|
||||||
|
try {
|
||||||
|
for (const id of ['bbcodeoff', 'htmlon']) {
|
||||||
|
const el = doc.getElementById(id);
|
||||||
|
if (el && id === 'bbcodeoff' && el.checked) throw new Error('请先取消“禁用 Discuz! 代码”,再重试。');
|
||||||
|
if (el && id === 'htmlon' && el.checked) throw new Error('请先关闭“HTML 代码”,再重试。');
|
||||||
|
}
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
setEditor(selected.title, bbcode);
|
||||||
|
const raw = win.getEditorContents();
|
||||||
|
if (Number(win.wysiwyg) === 1) {
|
||||||
|
for (const id of new Set(aids.values())) if (!raw.includes('attachimg_' + id)) throw new Error('富文本图片转换未通过检查。');
|
||||||
|
} else if (raw !== bbcode) throw new Error('正文填写校验失败。');
|
||||||
|
if (doc.querySelector('#subject').value !== selected.title) throw new Error('标题填写校验失败。');
|
||||||
|
committed = true;
|
||||||
|
} catch (error) {
|
||||||
|
// Restore the exact editor representation saved immediately before filling.
|
||||||
|
win.switchEditor(backup.mode);
|
||||||
|
if (backup.mode) win.editdoc.body.innerHTML = backup.text;
|
||||||
|
else doc.querySelector('#e_textarea').value = backup.text;
|
||||||
|
doc.querySelector('#subject').value = backup.title;
|
||||||
|
const switcher = doc.querySelector('#e_switchercheck'); if (switcher) switcher.checked = !backup.mode;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
lastResult = { title: selected.title, images: Object.fromEntries(aids), completedAt: new Date().toISOString() };
|
||||||
|
notify('complete', `已填写标题、正文和 ${aids.size} 张配图。请检查排版,随后手动发表。`);
|
||||||
|
return lastResult;
|
||||||
|
} catch (error) {
|
||||||
|
notify('error', error.message + (committed ? '' : ' 标题和正文未替换;已上传的图片会保留。'));
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
busy = false; form.removeEventListener('submit', blockSubmit, true);
|
||||||
|
submitButtons.forEach((el, index) => { el.disabled = previousDisabled[index]; });
|
||||||
|
controls.forEach(el => { el.disabled = false; });
|
||||||
|
$('import').disabled = !article;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function downloadJSON(value, filename) {
|
||||||
|
const url = URL.createObjectURL(new Blob([JSON.stringify(value, null, 2)], { type: 'application/json' }));
|
||||||
|
const link = doc.createElement('a'); link.href = url; link.download = filename; link.click();
|
||||||
|
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||||
|
}
|
||||||
|
// Public v1 interface. Staging is read-only; importPrepared is the explicit upload/fill step.
|
||||||
|
// No postMessage listener, localhost service, publishing endpoint or saved credentials.
|
||||||
|
const api = Object.freeze({ version: VERSION, schemaVersion: 1, stageFiles, stagePackage,
|
||||||
|
getState: summary, openPicker: () => { if (!busy) $('folder').click(); }, importPrepared,
|
||||||
|
getBackup: () => savedBackup });
|
||||||
|
win.DWW11Importer = api;
|
||||||
|
$('choose').onclick = () => { $('folder').value = ''; $('folder').click(); };
|
||||||
|
$('package').onclick = () => { $('json').value = ''; $('json').click(); };
|
||||||
|
const handleError = error => notify('error', error.message || String(error));
|
||||||
|
$('folder').onchange = () => stageFiles($('folder').files).catch(handleError);
|
||||||
|
$('json').onchange = async () => {
|
||||||
|
try { const file = $('json').files[0]; if (file) await stagePackage(JSON.parse(await file.text())); }
|
||||||
|
catch (error) { handleError(error); }
|
||||||
|
};
|
||||||
|
$('md').onchange = () => {
|
||||||
|
const files = candidates, path = $('md').value;
|
||||||
|
if (files && path) stageFiles(files, { markdownPath: path }).catch(handleError);
|
||||||
|
};
|
||||||
|
$('import').onclick = () => importPrepared().catch(handleError);
|
||||||
|
$('diagnostics').onclick = () => downloadJSON({ ...summary(), logs }, 'dww11-import-diagnostics.json');
|
||||||
|
win.addEventListener('pagehide', releasePreviews, { once: true });
|
||||||
|
}
|
||||||
|
return { VERSION, normalizePath, imagePath, readLink, markdownToBBCode, compile, prepareFiles, preparePackage, parseAid, mount };
|
||||||
|
});
|
||||||
|
|
||||||
|
// Extracted from dww11-wechat-oneclick.user.js v1.0.9; package source is authoritative.
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
const cleanText = value => String(value || '').replace(/\s+/g, ' ').trim();
|
||||||
|
function flattenParagraphs(body) {
|
||||||
|
const selector = 'p,div,section,article,figure,figcaption,h1,h2,h3,h4,h5,h6';
|
||||||
|
const marker = () => {
|
||||||
|
const br = body.ownerDocument.createElement('br');
|
||||||
|
br.dataset.dww11Boundary = '1'; return br;
|
||||||
|
};
|
||||||
|
for (const block of Array.from(body.querySelectorAll(selector)).reverse()) {
|
||||||
|
if (block.closest('table,ul,ol,blockquote')) continue;
|
||||||
|
block.replaceWith(marker(), ...block.childNodes, marker());
|
||||||
|
}
|
||||||
|
let previous = false, pending = [];
|
||||||
|
function finish() {
|
||||||
|
const actual = pending.filter(br => !br.hasAttribute('data-dww11-boundary'));
|
||||||
|
let retained = false;
|
||||||
|
for (const br of pending) {
|
||||||
|
if (!br.hasAttribute('data-dww11-boundary')) continue;
|
||||||
|
if (previous && !actual.length && !retained) { br.removeAttribute('data-dww11-boundary'); retained = true; }
|
||||||
|
else br.remove();
|
||||||
|
}
|
||||||
|
pending = [];
|
||||||
|
}
|
||||||
|
const walker = body.ownerDocument.createTreeWalker(body, NodeFilter.SHOW_ELEMENT | NodeFilter.SHOW_TEXT);
|
||||||
|
const nodes = []; while (walker.nextNode()) nodes.push(walker.currentNode);
|
||||||
|
for (const node of nodes) {
|
||||||
|
if (node.nodeType === 1 && node.tagName === 'BR') pending.push(node);
|
||||||
|
else if ((node.nodeType === 3 && node.textContent.trim()) || (node.nodeType === 1 && ['IMG', 'VIDEO', 'HR', 'IFRAME'].includes(node.tagName))) {
|
||||||
|
finish(); previous = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 末尾不保留由容器边界制造的额外换行。
|
||||||
|
for (const br of pending) if (br.hasAttribute('data-dww11-boundary')) br.remove();
|
||||||
|
}
|
||||||
|
function spaceImages(body) {
|
||||||
|
let previous = null, gapNodes = [], blankLines = 0;
|
||||||
|
const changes = [];
|
||||||
|
function content(kind, node) {
|
||||||
|
if (previous && previous.kind !== kind) {
|
||||||
|
let anchor = node;
|
||||||
|
while (anchor.parentElement && anchor.parentElement !== body &&
|
||||||
|
!['TD', 'TH', 'LI', 'BLOCKQUOTE'].includes(anchor.parentElement.tagName) && !anchor.parentElement.contains(previous.node)) anchor = anchor.parentElement;
|
||||||
|
changes.push({ anchor, nodes: gapNodes.slice(), count: Math.max(2, blankLines) });
|
||||||
|
}
|
||||||
|
previous = { kind, node }; gapNodes = []; blankLines = 0;
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
if (node.textContent.trim()) content('text', node); else gapNodes.push(node);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { gapNodes.push(node); blankLines++; return; }
|
||||||
|
if (node.tagName === 'IMG') { content('image', node); return; }
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
for (const { anchor, nodes, count } of changes) {
|
||||||
|
for (const node of nodes) node.remove();
|
||||||
|
const fragment = body.ownerDocument.createDocumentFragment();
|
||||||
|
for (let i = 0; i < count; i++) fragment.append(body.ownerDocument.createElement('br'));
|
||||||
|
anchor.parentNode.insertBefore(fragment, anchor);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function formatBody(body, author, sourceText = null) {
|
||||||
|
const doc = body.ownerDocument;
|
||||||
|
body.querySelectorAll('[data-dww11-wx-source="1"]').forEach(el => el.remove());
|
||||||
|
// Discuz 会把内联 color 值直接转为 [color=…]。
|
||||||
|
// !important 不能出现在该值中;颜色和字号只用原生 FONT 属性。
|
||||||
|
for (const el of [body, ...body.querySelectorAll('*')]) {
|
||||||
|
if (['SCRIPT', 'STYLE'].includes(el.tagName)) continue;
|
||||||
|
for (const property of ['font', 'font-size', 'color', '-webkit-text-fill-color']) {
|
||||||
|
el.style.removeProperty(property);
|
||||||
|
}
|
||||||
|
// 其他排版属性也不保留 CSS 优先级,避免它泄漏进论坛标签。
|
||||||
|
for (const property of Array.from(el.style)) {
|
||||||
|
if (el.style.getPropertyPriority(property)) {
|
||||||
|
el.style.setProperty(property, el.style.getPropertyValue(property));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!el.style.cssText) el.removeAttribute('style');
|
||||||
|
}
|
||||||
|
// 去掉采集器或旧版脚本生成的嵌套 FONT,保留其中的内容和图片。
|
||||||
|
for (const font of Array.from(body.querySelectorAll('font'))) {
|
||||||
|
font.replaceWith(...font.childNodes);
|
||||||
|
}
|
||||||
|
flattenParagraphs(body);
|
||||||
|
spaceImages(body);
|
||||||
|
// 所有正文内容和换行放在同一 FONT 中。不能使用只有 BR 的空 FONT:
|
||||||
|
// 论坛转换器可能清除空格式标签,导致空行退回默认1号或直接丢失。
|
||||||
|
const articleFont = doc.createElement('font');
|
||||||
|
articleFont.size = '4'; articleFont.color = '#000000';
|
||||||
|
articleFont.dataset.dww11WxBody = '1';
|
||||||
|
articleFont.append(...body.childNodes);
|
||||||
|
body.append(articleFont);
|
||||||
|
// A 自身的浏览器默认蓝色会覆盖外层继承色,链接文字另设黑色。
|
||||||
|
for (const link of articleFont.querySelectorAll('a')) {
|
||||||
|
const linkFont = doc.createElement('font');
|
||||||
|
linkFont.size = '4'; linkFont.color = '#000000';
|
||||||
|
linkFont.append(...link.childNodes); link.append(linkFont);
|
||||||
|
}
|
||||||
|
// 只用于编辑器图片适配;帖子展示仍交给网站原有图片/附件规则。
|
||||||
|
for (const img of articleFont.querySelectorAll('img')) {
|
||||||
|
img.style.maxWidth = '100%'; img.style.height = 'auto';
|
||||||
|
}
|
||||||
|
const source = doc.createElement('p');
|
||||||
|
source.dataset.dww11WxSource = '1';
|
||||||
|
const font = doc.createElement('font');
|
||||||
|
font.size = '1'; font.color = '#c0c0c0';
|
||||||
|
const lines = (sourceText ?? `来源:${author},如有侵权,请联系删除。`).split('\n');
|
||||||
|
lines.forEach((line, index) => {
|
||||||
|
if (index) font.append(doc.createElement('br'));
|
||||||
|
font.append(doc.createTextNode(line));
|
||||||
|
});
|
||||||
|
source.append(font); body.append(source);
|
||||||
|
}
|
||||||
|
function existingSource(body) {
|
||||||
|
// 按段落和 BR 读取末尾逻辑行,兼容嵌套 FONT、纯文本及末尾空段落。
|
||||||
|
const lines = [];
|
||||||
|
let parts = [];
|
||||||
|
function flush() {
|
||||||
|
const text = cleanText(parts.map(part => part.node.textContent.slice(part.start, part.end)).join(''));
|
||||||
|
if (text) lines.push({ text, parts });
|
||||||
|
parts = [];
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
// 保留文本中的显式换行,并记录精确位置,避免删除同容器中的正文。
|
||||||
|
const value = node.textContent;
|
||||||
|
let start = 0;
|
||||||
|
for (let i = 0; i <= value.length; i++) {
|
||||||
|
if (i !== value.length && value[i] !== '\n') continue;
|
||||||
|
// HTML 缩进换行通常只是空白;只有后续是来源或版权行才分行。
|
||||||
|
if (i < value.length && !/^\s*(?:(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]|如有侵权|若有侵权|侵权请|转载请|版权归)/.test(value.slice(i + 1))) continue;
|
||||||
|
if (i > start) parts.push({ node, start, end: i });
|
||||||
|
if (i < value.length) flush();
|
||||||
|
start = i + 1;
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { flush(); return; }
|
||||||
|
if (['IMG', 'VIDEO', 'AUDIO', 'IFRAME', 'HR'].includes(node.tagName)) {
|
||||||
|
flush(); lines.push({ text: '\ufffc', parts: [] }); return;
|
||||||
|
}
|
||||||
|
const block = /^(P|DIV|SECTION|ARTICLE|FIGURE|FIGCAPTION|H[1-6]|LI|TR|BLOCKQUOTE|PRE)$/.test(node.tagName);
|
||||||
|
if (block) flush();
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
if (block) flush();
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
flush();
|
||||||
|
let first = -1;
|
||||||
|
for (let i = lines.length - 1; i >= 0; i--) {
|
||||||
|
if (/^(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]\s*\S/.test(lines[i].text)) {
|
||||||
|
first = i;
|
||||||
|
} else if (!/^(?:如有侵权|若有侵权|侵权请|转载请|版权归)/.test(lines[i].text)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (first < 0) return null;
|
||||||
|
const start = lines[first].parts[0];
|
||||||
|
const range = body.ownerDocument.createRange();
|
||||||
|
range.setStart(start.node, start.start);
|
||||||
|
range.setEnd(body, body.childNodes.length);
|
||||||
|
return {
|
||||||
|
text: lines.slice(first).map(line => line.text).join('\n'),
|
||||||
|
remove: () => range.deleteContents()
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
window.DWW11Formatter = Object.freeze({
|
||||||
|
format(expectedSource) {
|
||||||
|
if (Number(window.wysiwyg) !== 1) throw new Error('请切换到可视化编辑器后排版');
|
||||||
|
const body = document.querySelector('#e_iframe')?.contentDocument?.body;
|
||||||
|
if (!body) throw new Error('编辑器未就绪');
|
||||||
|
const footer = existingSource(body);
|
||||||
|
const expected = `来源:${expectedSource},如有侵权,请联系删除。`;
|
||||||
|
if (!footer || footer.text !== expected) throw new Error('文末来源与文章包不一致,请人工检查');
|
||||||
|
const images = [...body.querySelectorAll('img')].map(x => x.id + ':' + x.getAttribute('src'));
|
||||||
|
const backup = body.innerHTML;
|
||||||
|
try {
|
||||||
|
footer.remove();
|
||||||
|
formatBody(body, expectedSource, expected);
|
||||||
|
const after = [...body.querySelectorAll('img')].map(x => x.id + ':' + x.getAttribute('src'));
|
||||||
|
if (JSON.stringify(images) !== JSON.stringify(after)) throw new Error('排版后图片标识或顺序改变');
|
||||||
|
if (body.querySelectorAll('[data-dww11-wx-source="1"]').length !== 1) throw new Error('来源尾注数量异常');
|
||||||
|
body.dispatchEvent(new Event('input', {bubbles:true}));
|
||||||
|
return {images: after, source: expected};
|
||||||
|
} catch (error) { body.innerHTML = backup; throw error; }
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})();
|
||||||
|
|
||||||
|
/* Page-side orchestrator. This file never clicks or submits the publication form. */
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
if (window.DWW11NewsRunner) return;
|
||||||
|
window.DWW11NewsRunner = true;
|
||||||
|
const SLOT='auto-news-current';
|
||||||
|
const owner=sessionStorage.getItem('auto-news-owner') || crypto.randomUUID();
|
||||||
|
sessionStorage.setItem('auto-news-owner',owner);
|
||||||
|
let run=null, working=false, halted=true, aborter=null, active=null, tickBusy=false;
|
||||||
|
const pending=new Map();
|
||||||
|
function rpc(route,body={}) {
|
||||||
|
return new Promise((resolve,reject)=>{
|
||||||
|
const id=crypto.randomUUID();
|
||||||
|
const timer=setTimeout(()=>{pending.delete(id);reject(new Error('本地连接无响应'));},65000);
|
||||||
|
pending.set(id,{resolve,reject,timer});
|
||||||
|
document.dispatchEvent(new CustomEvent('dww11:local-request',{detail:JSON.stringify({id,route,body:{owner,...body}})}));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
document.addEventListener('dww11:local-response',event=>{
|
||||||
|
let data;try{data=JSON.parse(event.detail);}catch{return;}
|
||||||
|
const waiter=pending.get(data.id);if(!waiter)return;
|
||||||
|
clearTimeout(waiter.timer);pending.delete(data.id);
|
||||||
|
data.error?waiter.reject(new Error(data.error)):waiter.resolve(data.value);
|
||||||
|
});
|
||||||
|
const status=message=>document.dispatchEvent(new CustomEvent('dww11:connection-status',{detail:message}));
|
||||||
|
document.addEventListener('dww11:auto-stop',()=>{halted=true;aborter?.abort(new Error('流程已终止'));});
|
||||||
|
function account() {
|
||||||
|
const element=document.querySelector('#mumucms_username');
|
||||||
|
if(!element)return null;
|
||||||
|
const name=[...element.childNodes].filter(n=>n.nodeType===Node.TEXT_NODE).map(n=>n.textContent).join('').trim();
|
||||||
|
const links=[element,...element.querySelectorAll('a')].filter(e=>e.href);
|
||||||
|
let uid=null;
|
||||||
|
for(const link of links) {
|
||||||
|
const href=link.href;
|
||||||
|
uid=new URL(href,location.href).searchParams.get('uid') || href.match(/space-uid-(\d+)/)?.[1];
|
||||||
|
if(uid)break;
|
||||||
|
}
|
||||||
|
return uid&&name?{uid,name}:null;
|
||||||
|
}
|
||||||
|
function extension() {
|
||||||
|
try{return JSON.parse(document.querySelector('#dww11-extension-panel')?.getAttribute('data-auto-news-state') || '{}');}catch{return {};}
|
||||||
|
}
|
||||||
|
function assertLive(epoch) {
|
||||||
|
if(halted||!run?.running||run.epoch!==epoch)throw new Error('流程已暂停或终止');
|
||||||
|
if(active?.account?.uid!==account()?.uid)throw new Error('当前账号与任务不一致');
|
||||||
|
}
|
||||||
|
function editorSnapshot() {
|
||||||
|
const body=document.querySelector('#e_iframe')?.contentDocument?.body;
|
||||||
|
if(!body)throw new Error('找不到可视化编辑器');
|
||||||
|
return {title:document.querySelector('#subject')?.value || '',
|
||||||
|
images:[...body.querySelectorAll('img')].map(x=>({id:x.id,src:x.getAttribute('src')})),
|
||||||
|
// Snapshot only final submitted content; never includes hidden tokens or the full form.
|
||||||
|
html:body.innerHTML.slice(0,500000), text:body.innerText.slice(0,100000)};
|
||||||
|
}
|
||||||
|
function saveLocal(value) {sessionStorage.setItem(SLOT,JSON.stringify(value));}
|
||||||
|
function loadLocal(){try{return JSON.parse(sessionStorage.getItem(SLOT)||'null');}catch{return null;}}
|
||||||
|
async function event(state,extra={}) {return rpc('event',{id:active.id,state,epoch:run.epoch,...extra});}
|
||||||
|
async function fail(error) {
|
||||||
|
status(error.message);
|
||||||
|
if(active)try{await event('needs_attention',{error:error.message});}catch{}
|
||||||
|
halted=true;aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));
|
||||||
|
}
|
||||||
|
function editorPage(job) {
|
||||||
|
const u=new URL(location.href);
|
||||||
|
return u.searchParams.get('mod')==='post'&&u.searchParams.get('action')==='newthread'&&u.searchParams.get('fid')===String(job.fid);
|
||||||
|
}
|
||||||
|
async function prepare(job) {
|
||||||
|
working=true;active=job;aborter=new AbortController();const epoch=run.epoch;
|
||||||
|
try {
|
||||||
|
assertLive(epoch);
|
||||||
|
if(!editorPage(job)) {
|
||||||
|
saveLocal({id:job.id});
|
||||||
|
location.assign(`https://www.dww11.com/forum.php?mod=post&action=newthread&fid=${job.fid}`);return;
|
||||||
|
}
|
||||||
|
if(!window.DWW11Importer||!window.DWW11Formatter)throw new Error('导入或排版模块尚未就绪,请检查安装');
|
||||||
|
status('正在检查文章包…');
|
||||||
|
const pack=await rpc('package',{id:job.id});assertLive(epoch);
|
||||||
|
const prepared=await window.DWW11Importer.stagePackage(pack);assertLive(epoch);
|
||||||
|
if(prepared.warnings?.length)throw new Error('文章需要人工处理:'+prepared.warnings.join(';'));
|
||||||
|
await event('importing');assertLive(epoch);status('上传图片并填写文章…');
|
||||||
|
const result=await window.DWW11Importer.importPrepared({automatic:true,signal:aborter.signal});assertLive(epoch);
|
||||||
|
if(result?.cancelled)throw new Error('导入已取消');
|
||||||
|
await event('formatting');assertLive(epoch);
|
||||||
|
window.DWW11Formatter.format(job.source);assertLive(epoch);
|
||||||
|
const snapshot=editorSnapshot();
|
||||||
|
if(snapshot.title!==job.title)throw new Error('标题校验失败');
|
||||||
|
const aids=new Set(Object.values(result.images));
|
||||||
|
for(const aid of aids) if(!snapshot.images.some(x=>x.id==='attachimg_'+aid))throw new Error('配图附件缺失:'+aid);
|
||||||
|
await event('awaiting_user_review',{snapshot});assertLive(epoch);
|
||||||
|
active={...job,state:'awaiting_user_review'};saveLocal({id:job.id});
|
||||||
|
status('已完成排版,请核对内容并亲手点击“发表帖子”。');
|
||||||
|
}catch(error){await fail(error);}finally{working=false;}
|
||||||
|
}
|
||||||
|
// Record only trusted user activation. Never replay a click, call submit(), or send a post request.
|
||||||
|
document.addEventListener('click',event=>{
|
||||||
|
if(!event.isTrusted||!event.target.closest?.('#postsubmit')||active?.state!=='awaiting_user_review')return;
|
||||||
|
try {
|
||||||
|
const snapshot=editorSnapshot();
|
||||||
|
const local={id:active.id,snapshot,clickedAt:Date.now(),observed:false};saveLocal(local);
|
||||||
|
rpc('event',{id:active.id,state:'user_submitting',epoch:run?.epoch,snapshot})
|
||||||
|
.then(()=>{const latest=loadLocal();if(latest?.clickedAt===local.clickedAt)saveLocal({...local,observed:true});})
|
||||||
|
.catch(error=>status('提交记录待恢复:'+error.message));
|
||||||
|
active.state='user_submitting';status('已观察到人工点击,等待结果页核对。');
|
||||||
|
}catch(error){status('提交快照记录失败:'+error.message);}
|
||||||
|
},true);
|
||||||
|
async function observeResult(job) {
|
||||||
|
const local=loadLocal();
|
||||||
|
if(local?.id===job.id&&local.snapshot&&job.state==='awaiting_user_review') {
|
||||||
|
await rpc('event',{id:job.id,state:'user_submitting',epoch:run.epoch,snapshot:local.snapshot});
|
||||||
|
job.state='user_submitting';
|
||||||
|
}
|
||||||
|
if(job.state==='user_submitting') {
|
||||||
|
// Result selectors intentionally await user-provided HTML; URL alone is not proof.
|
||||||
|
if(!editorPage(job) || (local?.clickedAt && Date.now()-local.clickedAt>45000)) {
|
||||||
|
await rpc('event',{id:job.id,state:'submit_unknown',error:'等待结果页 HTML 适配;候选页面:'+location.href});
|
||||||
|
await rpc('stop');
|
||||||
|
status('已记录候选结果页;尚不能自动核对。流程已暂停,请补充结果页 HTML。');
|
||||||
|
}
|
||||||
|
}else if(job.state==='submit_unknown')status('发布结果待核对;不会重发、计数或换号。');
|
||||||
|
}
|
||||||
|
async function switchAccount() {
|
||||||
|
if(working||halted)return;
|
||||||
|
if(location.origin!=='https://www.dww11.com'||location.pathname!=='/') {
|
||||||
|
location.assign('https://www.dww11.com/');return;
|
||||||
|
}
|
||||||
|
const ext=extension(),button=document.querySelector('#dww11-extension-next');
|
||||||
|
if(!ext.ready||!ext.idle||!button||button.disabled) {status('请安装并重新加载带联动接口的换号插件');return;}
|
||||||
|
await rpc('switch-begin');
|
||||||
|
if(halted||!run?.running)return;
|
||||||
|
button.click();status('正在换号;等待插件完成并核对新账号');
|
||||||
|
}
|
||||||
|
async function tick() {
|
||||||
|
if(tickBusy)return;tickBusy=true;
|
||||||
|
try {
|
||||||
|
run=await rpc('heartbeat',{account:account(),url:location.href,extension:extension()});
|
||||||
|
halted=!run.running;
|
||||||
|
if(!run.running) {aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));}
|
||||||
|
if(working)return;
|
||||||
|
// Passive result recording is permitted after Stop, but no new business actions.
|
||||||
|
if(run.active) {
|
||||||
|
const local=loadLocal();
|
||||||
|
if(local?.id===run.active&&local.snapshot&&!run.running)status('流程已暂停;提交结果仍需核对');
|
||||||
|
}
|
||||||
|
if(!run.running) {
|
||||||
|
if(run.active && run.owner===owner) {const saved=await rpc('current');if(saved.job) {active=saved.job;await observeResult(saved.job);}}
|
||||||
|
status(run.reason || '流程已暂停');return;
|
||||||
|
}
|
||||||
|
const next=await rpc('next',{account:account(),extension:extension()});run=next.run;
|
||||||
|
if(next.job)active=next.job;
|
||||||
|
if(next.action==='prepare'||(next.action==='resume'&&next.job.state==='preparing'))void prepare(next.job);
|
||||||
|
else if(next.action==='resume') {
|
||||||
|
await observeResult(next.job);
|
||||||
|
if(next.job.state==='awaiting_user_review') {
|
||||||
|
if(!editorPage(next.job)||!document.querySelector('#subject')?.value) await fail(new Error('待审稿页面已丢失,请人工核对旧稿,不自动重新导入'));
|
||||||
|
else status('等待你亲手发表;程序不再修改编辑器');
|
||||||
|
}
|
||||||
|
else if(['importing','formatting'].includes(next.job.state))await fail(new Error('导入阶段发生中断,请检查编辑器和已上传图片后处理任务'));
|
||||||
|
else if(next.job.state==='needs_attention')status(next.job.error);
|
||||||
|
}else if(next.action==='switch')await switchAccount();
|
||||||
|
else if(next.action==='switch_wait')status('等待换号结果;若插件已暂停,请人工处理后继续');
|
||||||
|
else if(next.action==='login')status('未能读取当前账号 UID,请登录并确认账号元素');
|
||||||
|
else if(next.action==='other_tab')status('另一个标签持有任务;如需接管请先暂停');
|
||||||
|
else if(next.action==='interval')status('等待设定的处理间隔');
|
||||||
|
else status(run.reason || '等待任务');
|
||||||
|
}catch(error){status(error.message);halted=true;aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));}
|
||||||
|
finally{tickBusy=false;}
|
||||||
|
}
|
||||||
|
const takeover=document.createElement('button');takeover.type='button';takeover.textContent='接管已暂停任务';
|
||||||
|
takeover.style.cssText='position:fixed;left:16px;top:16px;z-index:2147483646;padding:8px;background:#fff;border:1px solid #aab;border-radius:6px';
|
||||||
|
takeover.onclick=async()=>{try{await rpc('takeover');status('已接管;核对当前页面后在工作台继续');}catch(error){status(error.message);}};
|
||||||
|
document.body.append(takeover);
|
||||||
|
setInterval(tick,2000);tick();
|
||||||
|
})();
|
||||||
@@ -0,0 +1,397 @@
|
|||||||
|
// ==UserScript==
|
||||||
|
// @name 广谈大湾网 · 微信文章一键导入并排版
|
||||||
|
// @namespace local.dww11.wechat-oneclick
|
||||||
|
// @version 1.0.9
|
||||||
|
// @description 使用网站现有微信采集器,修正标题,统一正文4号黑色,追加1号银色来源;不点击保存或发布。
|
||||||
|
// @match https://www.dww11.com/forum.php*
|
||||||
|
// @match https://dww11.com/forum.php*
|
||||||
|
// @grant GM_xmlhttpRequest
|
||||||
|
// @grant unsafeWindow
|
||||||
|
// @connect mp.weixin.qq.com
|
||||||
|
// @run-at document-idle
|
||||||
|
// @noframes
|
||||||
|
// ==/UserScript==
|
||||||
|
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
const ID = 'dww11-wx-oneclick';
|
||||||
|
const FORMAT_ID = 'dww11-format-oneclick';
|
||||||
|
let busy = false;
|
||||||
|
function setBusy(value) {
|
||||||
|
busy = value;
|
||||||
|
for (const id of [ID, FORMAT_ID]) {
|
||||||
|
const button = document.getElementById(id);
|
||||||
|
if (button) button.disabled = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||||
|
const visible = el => el && el.getClientRects().length && getComputedStyle(el).display !== 'none';
|
||||||
|
async function waitFor(fn, timeout = 10000) {
|
||||||
|
const deadline = Date.now() + timeout;
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
const result = fn();
|
||||||
|
if (result) return result;
|
||||||
|
await sleep(100);
|
||||||
|
}
|
||||||
|
throw new Error('等待编辑器或采集窗口超时,请关闭遮挡窗口后重试。');
|
||||||
|
}
|
||||||
|
function normalizeURL(raw) {
|
||||||
|
const url = new URL(raw.trim().replace(/[\u200b-\u200d\ufeff]/g, ''));
|
||||||
|
if (url.protocol !== 'https:' || url.hostname !== 'mp.weixin.qq.com' || url.port || url.username || url.password || !/^\/(s(?:\/|$)|s$)/.test(url.pathname)) {
|
||||||
|
throw new Error('请输入 https://mp.weixin.qq.com/s/… 或 /s?… 微信文章地址。');
|
||||||
|
}
|
||||||
|
url.hash = '';
|
||||||
|
return url.href;
|
||||||
|
}
|
||||||
|
function cleanText(value) {
|
||||||
|
return String(value || '').replace(/\s+/g, ' ').trim();
|
||||||
|
}
|
||||||
|
function metadata(html) {
|
||||||
|
const doc = new DOMParser().parseFromString(html, 'text/html');
|
||||||
|
const text = selector => cleanText(doc.querySelector(selector)?.textContent);
|
||||||
|
const meta = selector => cleanText(doc.querySelector(selector)?.getAttribute('content'));
|
||||||
|
// 与下载组件一致,先确认是真正的文章,避免把验证页标题当成文章。
|
||||||
|
if (!doc.querySelector('#js_content')) throw new Error('未找到微信文章正文,可能需要验证、登录,或文章已失效。');
|
||||||
|
const title = text('#activity-name .js_title_inner') || text('#activity-name') || meta('meta[property="og:title"]') || meta('meta[name="twitter:title"]') || text('h1') || text('title');
|
||||||
|
// 不读取 rich_media_meta_text 容器:它可能包含重复的隐藏作者,或实际上是日期/IP。
|
||||||
|
// 来源使用蓝色链接 #js_name 的公众号名称,不使用灰色署名作者。
|
||||||
|
const author = text('#js_name');
|
||||||
|
if (!title || !author) throw new Error('微信未返回完整标题和公众号名称,可能需要验证或登录。');
|
||||||
|
return { title, author };
|
||||||
|
}
|
||||||
|
function fetchMetadata(url) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
GM_xmlhttpRequest({
|
||||||
|
method: 'GET', url, timeout: 25000, anonymous: true,
|
||||||
|
onload(response) {
|
||||||
|
try {
|
||||||
|
if (response.status !== 200) throw new Error('微信返回状态 ' + response.status);
|
||||||
|
if (response.finalUrl && new URL(response.finalUrl).hostname !== 'mp.weixin.qq.com') throw new Error('微信地址发生了非预期跳转。');
|
||||||
|
resolve(metadata(response.responseText));
|
||||||
|
} catch (error) { reject(error); }
|
||||||
|
},
|
||||||
|
onerror: () => reject(new Error('无法读取微信文章。')),
|
||||||
|
ontimeout: () => reject(new Error('读取微信文章超时。'))
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
async function getMetadata(url) {
|
||||||
|
try { return await fetchMetadata(url); }
|
||||||
|
catch (error) {
|
||||||
|
const title = prompt(error.message + '\n请从源文章复制完整标题(取消则停止导入):');
|
||||||
|
if (title === null) return null;
|
||||||
|
const author = prompt('请填写源文章蓝色链接处的公众号名称(取消则停止导入):');
|
||||||
|
if (author === null) return null;
|
||||||
|
if (!cleanText(title) || !cleanText(author)) throw new Error('标题和作者不能为空。');
|
||||||
|
return { title: cleanText(title), author: cleanText(author) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function editorBody() {
|
||||||
|
return document.querySelector('#e_iframe')?.contentDocument?.body;
|
||||||
|
}
|
||||||
|
async function ensureVisualEditor() {
|
||||||
|
const checkbox = document.querySelector('#e_switchercheck');
|
||||||
|
if (checkbox?.checked) checkbox.click();
|
||||||
|
await waitFor(() => visible(document.querySelector('#e_iframe')) && editorBody());
|
||||||
|
}
|
||||||
|
// 只监听本次现有采集器的响应,不重写其请求或图片附件处理。
|
||||||
|
function watchImport(jq, url) {
|
||||||
|
let timer, resolveResult, rejectResult;
|
||||||
|
const result = new Promise((resolve, reject) => { resolveResult = resolve; rejectResult = reject; });
|
||||||
|
const events = jq(document);
|
||||||
|
const cleanup = () => { clearTimeout(timer); events.off('.dww11wx'); };
|
||||||
|
const finish = error => { cleanup(); error ? rejectResult(error) : resolveResult(); };
|
||||||
|
events.on('ajaxComplete.dww11wx', (_event, xhr, settings) => {
|
||||||
|
let requestURL, sentURL;
|
||||||
|
try {
|
||||||
|
requestURL = new URL(settings.url, location.href);
|
||||||
|
sentURL = typeof settings.data === 'string' ? new URLSearchParams(settings.data).get('url') : settings.data?.url;
|
||||||
|
} catch { return; }
|
||||||
|
if (requestURL.origin !== location.origin || requestURL.searchParams.get('id') !== 'wxcj:cj' || sentURL !== url) return;
|
||||||
|
const parts = String(xhr.responseText || '').split('----wxcj----');
|
||||||
|
if (xhr.status < 200 || xhr.status >= 300 || parts.length < 4 || !parts[2]?.trim()) {
|
||||||
|
finish(new Error('网站采集失败或返回内容不完整。未执行排版,请检查采集提示。'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
finish();
|
||||||
|
});
|
||||||
|
timer = setTimeout(() => finish(new Error('网站采集超过90秒,未执行排版。采集器仍可能在后台返回,请勿连续重复导入。')), 90000);
|
||||||
|
return { result, cleanup };
|
||||||
|
}
|
||||||
|
// 将微信 P/SECTION 的隐式换行改成真实 BR,避免论坛忽略 SECTION 边界。
|
||||||
|
function flattenParagraphs(body) {
|
||||||
|
const selector = 'p,div,section,article,figure,figcaption,h1,h2,h3,h4,h5,h6';
|
||||||
|
const marker = () => {
|
||||||
|
const br = body.ownerDocument.createElement('br');
|
||||||
|
br.dataset.dww11Boundary = '1'; return br;
|
||||||
|
};
|
||||||
|
for (const block of Array.from(body.querySelectorAll(selector)).reverse()) {
|
||||||
|
if (block.closest('table,ul,ol,blockquote')) continue;
|
||||||
|
block.replaceWith(marker(), ...block.childNodes, marker());
|
||||||
|
}
|
||||||
|
let previous = false, pending = [];
|
||||||
|
function finish() {
|
||||||
|
const actual = pending.filter(br => !br.hasAttribute('data-dww11-boundary'));
|
||||||
|
let retained = false;
|
||||||
|
for (const br of pending) {
|
||||||
|
if (!br.hasAttribute('data-dww11-boundary')) continue;
|
||||||
|
if (previous && !actual.length && !retained) { br.removeAttribute('data-dww11-boundary'); retained = true; }
|
||||||
|
else br.remove();
|
||||||
|
}
|
||||||
|
pending = [];
|
||||||
|
}
|
||||||
|
const walker = body.ownerDocument.createTreeWalker(body, NodeFilter.SHOW_ELEMENT | NodeFilter.SHOW_TEXT);
|
||||||
|
const nodes = []; while (walker.nextNode()) nodes.push(walker.currentNode);
|
||||||
|
for (const node of nodes) {
|
||||||
|
if (node.nodeType === 1 && node.tagName === 'BR') pending.push(node);
|
||||||
|
else if ((node.nodeType === 3 && node.textContent.trim()) || (node.nodeType === 1 && ['IMG', 'VIDEO', 'HR', 'IFRAME'].includes(node.tagName))) {
|
||||||
|
finish(); previous = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 末尾不保留由容器边界制造的额外换行。
|
||||||
|
for (const br of pending) if (br.hasAttribute('data-dww11-boundary')) br.remove();
|
||||||
|
}
|
||||||
|
function spaceImages(body) {
|
||||||
|
let previous = null, gapNodes = [], blankLines = 0;
|
||||||
|
const changes = [];
|
||||||
|
function content(kind, node) {
|
||||||
|
if (previous && previous.kind !== kind) {
|
||||||
|
let anchor = node;
|
||||||
|
while (anchor.parentElement && anchor.parentElement !== body &&
|
||||||
|
!['TD', 'TH', 'LI', 'BLOCKQUOTE'].includes(anchor.parentElement.tagName) && !anchor.parentElement.contains(previous.node)) anchor = anchor.parentElement;
|
||||||
|
changes.push({ anchor, nodes: gapNodes.slice(), count: Math.max(2, blankLines) });
|
||||||
|
}
|
||||||
|
previous = { kind, node }; gapNodes = []; blankLines = 0;
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
if (node.textContent.trim()) content('text', node); else gapNodes.push(node);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { gapNodes.push(node); blankLines++; return; }
|
||||||
|
if (node.tagName === 'IMG') { content('image', node); return; }
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
for (const { anchor, nodes, count } of changes) {
|
||||||
|
for (const node of nodes) node.remove();
|
||||||
|
const fragment = body.ownerDocument.createDocumentFragment();
|
||||||
|
for (let i = 0; i < count; i++) fragment.append(body.ownerDocument.createElement('br'));
|
||||||
|
anchor.parentNode.insertBefore(fragment, anchor);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function formatBody(body, author, sourceText = null) {
|
||||||
|
const doc = body.ownerDocument;
|
||||||
|
body.querySelectorAll('[data-dww11-wx-source="1"]').forEach(el => el.remove());
|
||||||
|
// Discuz 会把内联 color 值直接转为 [color=…]。
|
||||||
|
// !important 不能出现在该值中;颜色和字号只用原生 FONT 属性。
|
||||||
|
for (const el of [body, ...body.querySelectorAll('*')]) {
|
||||||
|
if (['SCRIPT', 'STYLE'].includes(el.tagName)) continue;
|
||||||
|
for (const property of ['font', 'font-size', 'color', '-webkit-text-fill-color']) {
|
||||||
|
el.style.removeProperty(property);
|
||||||
|
}
|
||||||
|
// 其他排版属性也不保留 CSS 优先级,避免它泄漏进论坛标签。
|
||||||
|
for (const property of Array.from(el.style)) {
|
||||||
|
if (el.style.getPropertyPriority(property)) {
|
||||||
|
el.style.setProperty(property, el.style.getPropertyValue(property));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!el.style.cssText) el.removeAttribute('style');
|
||||||
|
}
|
||||||
|
// 去掉采集器或旧版脚本生成的嵌套 FONT,保留其中的内容和图片。
|
||||||
|
for (const font of Array.from(body.querySelectorAll('font'))) {
|
||||||
|
font.replaceWith(...font.childNodes);
|
||||||
|
}
|
||||||
|
flattenParagraphs(body);
|
||||||
|
spaceImages(body);
|
||||||
|
// 所有正文内容和换行放在同一 FONT 中。不能使用只有 BR 的空 FONT:
|
||||||
|
// 论坛转换器可能清除空格式标签,导致空行退回默认1号或直接丢失。
|
||||||
|
const articleFont = doc.createElement('font');
|
||||||
|
articleFont.size = '4'; articleFont.color = '#000000';
|
||||||
|
articleFont.dataset.dww11WxBody = '1';
|
||||||
|
articleFont.append(...body.childNodes);
|
||||||
|
body.append(articleFont);
|
||||||
|
// A 自身的浏览器默认蓝色会覆盖外层继承色,链接文字另设黑色。
|
||||||
|
for (const link of articleFont.querySelectorAll('a')) {
|
||||||
|
const linkFont = doc.createElement('font');
|
||||||
|
linkFont.size = '4'; linkFont.color = '#000000';
|
||||||
|
linkFont.append(...link.childNodes); link.append(linkFont);
|
||||||
|
}
|
||||||
|
// 只用于编辑器图片适配;帖子展示仍交给网站原有图片/附件规则。
|
||||||
|
for (const img of articleFont.querySelectorAll('img')) {
|
||||||
|
img.style.maxWidth = '100%'; img.style.height = 'auto';
|
||||||
|
}
|
||||||
|
const source = doc.createElement('p');
|
||||||
|
source.dataset.dww11WxSource = '1';
|
||||||
|
const font = doc.createElement('font');
|
||||||
|
font.size = '1'; font.color = '#c0c0c0';
|
||||||
|
const lines = (sourceText ?? `来源:${author},如有侵权,请联系删除。`).split('\n');
|
||||||
|
lines.forEach((line, index) => {
|
||||||
|
if (index) font.append(doc.createElement('br'));
|
||||||
|
font.append(doc.createTextNode(line));
|
||||||
|
});
|
||||||
|
source.append(font); body.append(source);
|
||||||
|
}
|
||||||
|
function existingSource(body) {
|
||||||
|
// 按段落和 BR 读取末尾逻辑行,兼容嵌套 FONT、纯文本及末尾空段落。
|
||||||
|
const lines = [];
|
||||||
|
let parts = [];
|
||||||
|
function flush() {
|
||||||
|
const text = cleanText(parts.map(part => part.node.textContent.slice(part.start, part.end)).join(''));
|
||||||
|
if (text) lines.push({ text, parts });
|
||||||
|
parts = [];
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
// 保留文本中的显式换行,并记录精确位置,避免删除同容器中的正文。
|
||||||
|
const value = node.textContent;
|
||||||
|
let start = 0;
|
||||||
|
for (let i = 0; i <= value.length; i++) {
|
||||||
|
if (i !== value.length && value[i] !== '\n') continue;
|
||||||
|
// HTML 缩进换行通常只是空白;只有后续是来源或版权行才分行。
|
||||||
|
if (i < value.length && !/^\s*(?:(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]|如有侵权|若有侵权|侵权请|转载请|版权归)/.test(value.slice(i + 1))) continue;
|
||||||
|
if (i > start) parts.push({ node, start, end: i });
|
||||||
|
if (i < value.length) flush();
|
||||||
|
start = i + 1;
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { flush(); return; }
|
||||||
|
if (['IMG', 'VIDEO', 'AUDIO', 'IFRAME', 'HR'].includes(node.tagName)) {
|
||||||
|
flush(); lines.push({ text: '\ufffc', parts: [] }); return;
|
||||||
|
}
|
||||||
|
const block = /^(P|DIV|SECTION|ARTICLE|FIGURE|FIGCAPTION|H[1-6]|LI|TR|BLOCKQUOTE|PRE)$/.test(node.tagName);
|
||||||
|
if (block) flush();
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
if (block) flush();
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
flush();
|
||||||
|
let first = -1;
|
||||||
|
for (let i = lines.length - 1; i >= 0; i--) {
|
||||||
|
if (/^(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]\s*\S/.test(lines[i].text)) {
|
||||||
|
first = i;
|
||||||
|
} else if (!/^(?:如有侵权|若有侵权|侵权请|转载请|版权归)/.test(lines[i].text)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (first < 0) return null;
|
||||||
|
const start = lines[first].parts[0];
|
||||||
|
const range = body.ownerDocument.createRange();
|
||||||
|
range.setStart(start.node, start.start);
|
||||||
|
range.setEnd(body, body.childNodes.length);
|
||||||
|
return {
|
||||||
|
text: lines.slice(first).map(line => line.text).join('\n'),
|
||||||
|
remove: () => range.deleteContents()
|
||||||
|
};
|
||||||
|
}
|
||||||
|
async function runFormat(button) {
|
||||||
|
if (busy) return;
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
await ensureVisualEditor();
|
||||||
|
const body = editorBody();
|
||||||
|
const source = existingSource(body);
|
||||||
|
const hasText = cleanText(body.textContent).replace(source ? cleanText(source.text) : '', '').trim();
|
||||||
|
if (!hasText && !body.querySelector('img,video,audio,iframe,hr,table')) {
|
||||||
|
throw new Error('正文为空,请先输入或粘贴文章。');
|
||||||
|
}
|
||||||
|
const raw = source ? '' : prompt('未在文末检测到来源/作者信息,请补充来源公众号名称(取消则停止排版):');
|
||||||
|
if (raw === null) return;
|
||||||
|
const author = cleanText(raw);
|
||||||
|
if (!source && !author) throw new Error('来源公众号名称不能为空。');
|
||||||
|
button.textContent = '正在排版…';
|
||||||
|
// 已有来源保留原文,仅调整字号颜色;没有时才生成来源行。
|
||||||
|
source?.remove();
|
||||||
|
formatBody(body, author, source?.text ?? null);
|
||||||
|
alert(`排版完成。\n正文:4号黑色;来源:1号银色。\n${source ? source.text : `来源:${author}`}\n请人工检查后自行决定是否保存或发布。`);
|
||||||
|
} catch (error) { alert('排版未完成:' + error.message); }
|
||||||
|
finally {
|
||||||
|
setBusy(false);
|
||||||
|
button.textContent = '一键排版';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async function run(button) {
|
||||||
|
if (busy) return;
|
||||||
|
const raw = prompt('粘贴微信公众号文章 URL:');
|
||||||
|
if (raw === null || !raw.trim()) return;
|
||||||
|
setBusy(true);
|
||||||
|
let watcher;
|
||||||
|
try {
|
||||||
|
const url = normalizeURL(raw);
|
||||||
|
const jq = unsafeWindow.myJQ;
|
||||||
|
if (typeof jq !== 'function' || !jq.fn?.on) throw new Error('没有找到网站微信采集器的 myJQ,请刷新页面后重试。');
|
||||||
|
const plugin = document.querySelector('#e_controls .wxcj');
|
||||||
|
if (!plugin) throw new Error('当前页面没有微信采集组件。');
|
||||||
|
button.textContent = '读取标题和作者…';
|
||||||
|
const info = await getMetadata(url);
|
||||||
|
if (!info) return;
|
||||||
|
const subject = document.querySelector('#subject');
|
||||||
|
if (!subject) throw new Error('没有找到标题框。');
|
||||||
|
if (subject.maxLength > 0 && info.title.length > subject.maxLength) throw new Error('完整标题超过标题框长度限制,请先处理标题长度。');
|
||||||
|
await ensureVisualEditor();
|
||||||
|
button.textContent = '采集文章中…';
|
||||||
|
plugin.click();
|
||||||
|
const input = await waitFor(() => Array.from(document.querySelectorAll('.wxjc-dialog input')).find(visible));
|
||||||
|
const submit = input.closest('.wxjc-dialog').querySelector('.wxjc-subimt');
|
||||||
|
if (!submit) throw new Error('没有找到采集按钮。');
|
||||||
|
input.value = url;
|
||||||
|
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||||
|
input.dispatchEvent(new Event('change', { bubbles: true }));
|
||||||
|
watcher = watchImport(jq, url);
|
||||||
|
// 先登记失败处理,避免点击失败时留下未处理的 Promise。
|
||||||
|
watcher.result.catch(() => {});
|
||||||
|
submit.click();
|
||||||
|
await watcher.result;
|
||||||
|
button.textContent = '正在排版…';
|
||||||
|
const body = await waitFor(() => editorBody()?.innerHTML.trim() && editorBody());
|
||||||
|
formatBody(body, info.author);
|
||||||
|
subject.value = info.title;
|
||||||
|
subject.dispatchEvent(new Event('input', { bubbles: true }));
|
||||||
|
subject.dispatchEvent(new Event('change', { bubbles: true }));
|
||||||
|
subject.dispatchEvent(new KeyboardEvent('keyup', { bubbles: true }));
|
||||||
|
// 正文保持在可视化编辑器,由网站正常提交流程转成 BBCode。
|
||||||
|
// 不把 HTML 写入纯文本 message,也不调用任何保存/发布入口。
|
||||||
|
alert(`导入并排版完成。\n标题:${info.title}\n作者:${info.author}\n正文:4号黑色;来源:1号银色。\n请人工检查后自行决定是否保存或发布。`);
|
||||||
|
} catch (error) { alert('导入未完成:' + error.message); }
|
||||||
|
finally {
|
||||||
|
watcher?.cleanup();
|
||||||
|
setBusy(false);
|
||||||
|
button.textContent = '一键导入微信公众号并排版';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function mount() {
|
||||||
|
const toolbar = document.querySelector('#e_controls');
|
||||||
|
if (!toolbar || !document.querySelector('#subject')) return;
|
||||||
|
const style = 'position:relative;z-index:2;padding:0 12px;height:32px;line-height:30px;border:1px solid #159447;border-radius:4px;background:#159447;color:white;font-size:13px;cursor:pointer;';
|
||||||
|
// 右浮动的同一组内使用 flex,保证排版按钮始终位于导入按钮左侧。
|
||||||
|
let group = document.getElementById('dww11-wx-actions');
|
||||||
|
if (!group) {
|
||||||
|
group = document.createElement('span');
|
||||||
|
group.id = 'dww11-wx-actions';
|
||||||
|
group.style.cssText = 'float:right;display:inline-flex;align-items:center;gap:8px;margin:6px 8px;';
|
||||||
|
toolbar.append(group);
|
||||||
|
}
|
||||||
|
if (!document.getElementById(FORMAT_ID)) {
|
||||||
|
const button = document.createElement('button');
|
||||||
|
button.id = FORMAT_ID; button.type = 'button';
|
||||||
|
button.textContent = '一键排版'; button.style.cssText = style;
|
||||||
|
button.disabled = busy;
|
||||||
|
button.addEventListener('click', () => runFormat(button));
|
||||||
|
group.prepend(button);
|
||||||
|
}
|
||||||
|
if (toolbar.querySelector('.wxcj') && !document.getElementById(ID)) {
|
||||||
|
const button = document.createElement('button');
|
||||||
|
button.id = ID; button.type = 'button';
|
||||||
|
button.textContent = '一键导入微信公众号并排版'; button.style.cssText = style;
|
||||||
|
button.disabled = busy;
|
||||||
|
button.addEventListener('click', () => run(button));
|
||||||
|
group.append(button);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mount();
|
||||||
|
const observer = new MutationObserver(mount);
|
||||||
|
observer.observe(document.documentElement, { childList: true, subtree: true });
|
||||||
|
})();
|
||||||
@@ -0,0 +1,195 @@
|
|||||||
|
// Extracted from dww11-wechat-oneclick.user.js v1.0.9; package source is authoritative.
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
const cleanText = value => String(value || '').replace(/\s+/g, ' ').trim();
|
||||||
|
function flattenParagraphs(body) {
|
||||||
|
const selector = 'p,div,section,article,figure,figcaption,h1,h2,h3,h4,h5,h6';
|
||||||
|
const marker = () => {
|
||||||
|
const br = body.ownerDocument.createElement('br');
|
||||||
|
br.dataset.dww11Boundary = '1'; return br;
|
||||||
|
};
|
||||||
|
for (const block of Array.from(body.querySelectorAll(selector)).reverse()) {
|
||||||
|
if (block.closest('table,ul,ol,blockquote')) continue;
|
||||||
|
block.replaceWith(marker(), ...block.childNodes, marker());
|
||||||
|
}
|
||||||
|
let previous = false, pending = [];
|
||||||
|
function finish() {
|
||||||
|
const actual = pending.filter(br => !br.hasAttribute('data-dww11-boundary'));
|
||||||
|
let retained = false;
|
||||||
|
for (const br of pending) {
|
||||||
|
if (!br.hasAttribute('data-dww11-boundary')) continue;
|
||||||
|
if (previous && !actual.length && !retained) { br.removeAttribute('data-dww11-boundary'); retained = true; }
|
||||||
|
else br.remove();
|
||||||
|
}
|
||||||
|
pending = [];
|
||||||
|
}
|
||||||
|
const walker = body.ownerDocument.createTreeWalker(body, NodeFilter.SHOW_ELEMENT | NodeFilter.SHOW_TEXT);
|
||||||
|
const nodes = []; while (walker.nextNode()) nodes.push(walker.currentNode);
|
||||||
|
for (const node of nodes) {
|
||||||
|
if (node.nodeType === 1 && node.tagName === 'BR') pending.push(node);
|
||||||
|
else if ((node.nodeType === 3 && node.textContent.trim()) || (node.nodeType === 1 && ['IMG', 'VIDEO', 'HR', 'IFRAME'].includes(node.tagName))) {
|
||||||
|
finish(); previous = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 末尾不保留由容器边界制造的额外换行。
|
||||||
|
for (const br of pending) if (br.hasAttribute('data-dww11-boundary')) br.remove();
|
||||||
|
}
|
||||||
|
function spaceImages(body) {
|
||||||
|
let previous = null, gapNodes = [], blankLines = 0;
|
||||||
|
const changes = [];
|
||||||
|
function content(kind, node) {
|
||||||
|
if (previous && previous.kind !== kind) {
|
||||||
|
let anchor = node;
|
||||||
|
while (anchor.parentElement && anchor.parentElement !== body &&
|
||||||
|
!['TD', 'TH', 'LI', 'BLOCKQUOTE'].includes(anchor.parentElement.tagName) && !anchor.parentElement.contains(previous.node)) anchor = anchor.parentElement;
|
||||||
|
changes.push({ anchor, nodes: gapNodes.slice(), count: Math.max(2, blankLines) });
|
||||||
|
}
|
||||||
|
previous = { kind, node }; gapNodes = []; blankLines = 0;
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
if (node.textContent.trim()) content('text', node); else gapNodes.push(node);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { gapNodes.push(node); blankLines++; return; }
|
||||||
|
if (node.tagName === 'IMG') { content('image', node); return; }
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
for (const { anchor, nodes, count } of changes) {
|
||||||
|
for (const node of nodes) node.remove();
|
||||||
|
const fragment = body.ownerDocument.createDocumentFragment();
|
||||||
|
for (let i = 0; i < count; i++) fragment.append(body.ownerDocument.createElement('br'));
|
||||||
|
anchor.parentNode.insertBefore(fragment, anchor);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function formatBody(body, author, sourceText = null) {
|
||||||
|
const doc = body.ownerDocument;
|
||||||
|
body.querySelectorAll('[data-dww11-wx-source="1"]').forEach(el => el.remove());
|
||||||
|
// Discuz 会把内联 color 值直接转为 [color=…]。
|
||||||
|
// !important 不能出现在该值中;颜色和字号只用原生 FONT 属性。
|
||||||
|
for (const el of [body, ...body.querySelectorAll('*')]) {
|
||||||
|
if (['SCRIPT', 'STYLE'].includes(el.tagName)) continue;
|
||||||
|
for (const property of ['font', 'font-size', 'color', '-webkit-text-fill-color']) {
|
||||||
|
el.style.removeProperty(property);
|
||||||
|
}
|
||||||
|
// 其他排版属性也不保留 CSS 优先级,避免它泄漏进论坛标签。
|
||||||
|
for (const property of Array.from(el.style)) {
|
||||||
|
if (el.style.getPropertyPriority(property)) {
|
||||||
|
el.style.setProperty(property, el.style.getPropertyValue(property));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!el.style.cssText) el.removeAttribute('style');
|
||||||
|
}
|
||||||
|
// 去掉采集器或旧版脚本生成的嵌套 FONT,保留其中的内容和图片。
|
||||||
|
for (const font of Array.from(body.querySelectorAll('font'))) {
|
||||||
|
font.replaceWith(...font.childNodes);
|
||||||
|
}
|
||||||
|
flattenParagraphs(body);
|
||||||
|
spaceImages(body);
|
||||||
|
// 所有正文内容和换行放在同一 FONT 中。不能使用只有 BR 的空 FONT:
|
||||||
|
// 论坛转换器可能清除空格式标签,导致空行退回默认1号或直接丢失。
|
||||||
|
const articleFont = doc.createElement('font');
|
||||||
|
articleFont.size = '4'; articleFont.color = '#000000';
|
||||||
|
articleFont.dataset.dww11WxBody = '1';
|
||||||
|
articleFont.append(...body.childNodes);
|
||||||
|
body.append(articleFont);
|
||||||
|
// A 自身的浏览器默认蓝色会覆盖外层继承色,链接文字另设黑色。
|
||||||
|
for (const link of articleFont.querySelectorAll('a')) {
|
||||||
|
const linkFont = doc.createElement('font');
|
||||||
|
linkFont.size = '4'; linkFont.color = '#000000';
|
||||||
|
linkFont.append(...link.childNodes); link.append(linkFont);
|
||||||
|
}
|
||||||
|
// 只用于编辑器图片适配;帖子展示仍交给网站原有图片/附件规则。
|
||||||
|
for (const img of articleFont.querySelectorAll('img')) {
|
||||||
|
img.style.maxWidth = '100%'; img.style.height = 'auto';
|
||||||
|
}
|
||||||
|
const source = doc.createElement('p');
|
||||||
|
source.dataset.dww11WxSource = '1';
|
||||||
|
const font = doc.createElement('font');
|
||||||
|
font.size = '1'; font.color = '#c0c0c0';
|
||||||
|
const lines = (sourceText ?? `来源:${author},如有侵权,请联系删除。`).split('\n');
|
||||||
|
lines.forEach((line, index) => {
|
||||||
|
if (index) font.append(doc.createElement('br'));
|
||||||
|
font.append(doc.createTextNode(line));
|
||||||
|
});
|
||||||
|
source.append(font); body.append(source);
|
||||||
|
}
|
||||||
|
function existingSource(body) {
|
||||||
|
// 按段落和 BR 读取末尾逻辑行,兼容嵌套 FONT、纯文本及末尾空段落。
|
||||||
|
const lines = [];
|
||||||
|
let parts = [];
|
||||||
|
function flush() {
|
||||||
|
const text = cleanText(parts.map(part => part.node.textContent.slice(part.start, part.end)).join(''));
|
||||||
|
if (text) lines.push({ text, parts });
|
||||||
|
parts = [];
|
||||||
|
}
|
||||||
|
function visit(node) {
|
||||||
|
if (node.nodeType === 3) {
|
||||||
|
// 保留文本中的显式换行,并记录精确位置,避免删除同容器中的正文。
|
||||||
|
const value = node.textContent;
|
||||||
|
let start = 0;
|
||||||
|
for (let i = 0; i <= value.length; i++) {
|
||||||
|
if (i !== value.length && value[i] !== '\n') continue;
|
||||||
|
// HTML 缩进换行通常只是空白;只有后续是来源或版权行才分行。
|
||||||
|
if (i < value.length && !/^\s*(?:(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]|如有侵权|若有侵权|侵权请|转载请|版权归)/.test(value.slice(i + 1))) continue;
|
||||||
|
if (i > start) parts.push({ node, start, end: i });
|
||||||
|
if (i < value.length) flush();
|
||||||
|
start = i + 1;
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (node.nodeType !== 1 || ['SCRIPT', 'STYLE', 'NOSCRIPT'].includes(node.tagName)) return;
|
||||||
|
if (node.tagName === 'BR') { flush(); return; }
|
||||||
|
if (['IMG', 'VIDEO', 'AUDIO', 'IFRAME', 'HR'].includes(node.tagName)) {
|
||||||
|
flush(); lines.push({ text: '\ufffc', parts: [] }); return;
|
||||||
|
}
|
||||||
|
const block = /^(P|DIV|SECTION|ARTICLE|FIGURE|FIGCAPTION|H[1-6]|LI|TR|BLOCKQUOTE|PRE)$/.test(node.tagName);
|
||||||
|
if (block) flush();
|
||||||
|
for (const child of node.childNodes) visit(child);
|
||||||
|
if (block) flush();
|
||||||
|
}
|
||||||
|
for (const child of body.childNodes) visit(child);
|
||||||
|
flush();
|
||||||
|
let first = -1;
|
||||||
|
for (let i = lines.length - 1; i >= 0; i--) {
|
||||||
|
if (/^(?:来源|文章来源|本文来源|图文来源|文字来源|作者|作者信息|来源作者|转自|转载自|出处)\s*[::]\s*\S/.test(lines[i].text)) {
|
||||||
|
first = i;
|
||||||
|
} else if (!/^(?:如有侵权|若有侵权|侵权请|转载请|版权归)/.test(lines[i].text)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (first < 0) return null;
|
||||||
|
const start = lines[first].parts[0];
|
||||||
|
const range = body.ownerDocument.createRange();
|
||||||
|
range.setStart(start.node, start.start);
|
||||||
|
range.setEnd(body, body.childNodes.length);
|
||||||
|
return {
|
||||||
|
text: lines.slice(first).map(line => line.text).join('\n'),
|
||||||
|
remove: () => range.deleteContents()
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
window.DWW11Formatter = Object.freeze({
|
||||||
|
format(expectedSource) {
|
||||||
|
if (Number(window.wysiwyg) !== 1) throw new Error('请切换到可视化编辑器后排版');
|
||||||
|
const body = document.querySelector('#e_iframe')?.contentDocument?.body;
|
||||||
|
if (!body) throw new Error('编辑器未就绪');
|
||||||
|
const footer = existingSource(body);
|
||||||
|
const expected = `来源:${expectedSource},如有侵权,请联系删除。`;
|
||||||
|
if (!footer || footer.text !== expected) throw new Error('文末来源与文章包不一致,请人工检查');
|
||||||
|
const images = [...body.querySelectorAll('img')].map(x => x.id + ':' + x.getAttribute('src'));
|
||||||
|
const backup = body.innerHTML;
|
||||||
|
try {
|
||||||
|
footer.remove();
|
||||||
|
formatBody(body, expectedSource, expected);
|
||||||
|
const after = [...body.querySelectorAll('img')].map(x => x.id + ':' + x.getAttribute('src'));
|
||||||
|
if (JSON.stringify(images) !== JSON.stringify(after)) throw new Error('排版后图片标识或顺序改变');
|
||||||
|
if (body.querySelectorAll('[data-dww11-wx-source="1"]').length !== 1) throw new Error('来源尾注数量异常');
|
||||||
|
body.dispatchEvent(new Event('input', {bubbles:true}));
|
||||||
|
return {images: after, source: expected};
|
||||||
|
} catch (error) { body.innerHTML = backup; throw error; }
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})();
|
||||||
@@ -0,0 +1,480 @@
|
|||||||
|
// ==UserScript==
|
||||||
|
// @name 广谈大湾网 · 本地文章目录导入
|
||||||
|
// @namespace local.dww11.article-importer
|
||||||
|
// @version 1.0.0
|
||||||
|
// @description 选择整个文章目录,导入 Markdown 标题、正文与本地配图;保留人工发布与程序接口。
|
||||||
|
// @match https://www.dww11.com/forum.php*
|
||||||
|
// @match https://dww11.com/forum.php*
|
||||||
|
// @run-at document-idle
|
||||||
|
// @grant none
|
||||||
|
// @noframes
|
||||||
|
// @license MIT
|
||||||
|
// ==/UserScript==
|
||||||
|
|
||||||
|
(function (root, factory) {
|
||||||
|
'use strict';
|
||||||
|
const core = factory();
|
||||||
|
if (typeof module === 'object' && module.exports) module.exports = core;
|
||||||
|
else core.mount(root);
|
||||||
|
})(typeof window === 'undefined' ? null : window, function () {
|
||||||
|
'use strict';
|
||||||
|
const VERSION = '1.0.0';
|
||||||
|
const MAX_IMAGE_SIZE = 20 * 1024 * 1024;
|
||||||
|
const IMAGE_EXT = /\.(?:jpe?g|png|gif)$/i;
|
||||||
|
|
||||||
|
function normalizePath(path) {
|
||||||
|
const parts = [];
|
||||||
|
const value = String(path).replace(/\\/g, '/').normalize('NFC');
|
||||||
|
if (/^(?:\/|[a-z]:|[a-z]+:)/i.test(value)) throw new Error('请使用目录内的相对路径:' + path);
|
||||||
|
for (const part of value.split('/')) {
|
||||||
|
if (!part || part === '.') continue;
|
||||||
|
if (part === '..') {
|
||||||
|
if (!parts.length) throw new Error('图片路径超出所选目录:' + path);
|
||||||
|
parts.pop();
|
||||||
|
} else parts.push(part);
|
||||||
|
}
|
||||||
|
return parts.join('/');
|
||||||
|
}
|
||||||
|
|
||||||
|
function imagePath(mdPath, target) {
|
||||||
|
if (/^(?:[a-z][\w+.-]*:|\/\/|\/)/i.test(target)) throw new Error('图片必须在所选目录内,请先下载:' + target);
|
||||||
|
let decoded;
|
||||||
|
try { decoded = decodeURIComponent(target); } catch { throw new Error('图片路径编码无效:' + target); }
|
||||||
|
return normalizePath(mdPath.slice(0, mdPath.lastIndexOf('/') + 1) + decoded);
|
||||||
|
}
|
||||||
|
|
||||||
|
// A small, intentionally bounded Markdown reader. Raw HTML is never executed.
|
||||||
|
// Links/images support balanced parentheses, angle destinations and optional titles.
|
||||||
|
function readLink(text, start) {
|
||||||
|
const image = text.startsWith('![', start);
|
||||||
|
if (!image && text[start] !== '[') return null;
|
||||||
|
const left = start + (image ? 2 : 1);
|
||||||
|
let end = left;
|
||||||
|
for (; end < text.length; end++) {
|
||||||
|
if (text[end] === '\\') { end++; continue; }
|
||||||
|
if (text[end] === ']') break;
|
||||||
|
}
|
||||||
|
if (text.slice(end, end + 2) !== '](') return null;
|
||||||
|
let pos = end + 2, depth = 1, quoted = '', angle = false;
|
||||||
|
for (; pos < text.length; pos++) {
|
||||||
|
const ch = text[pos];
|
||||||
|
if (ch === '\\') { pos++; continue; }
|
||||||
|
if (quoted) { if (ch === quoted) quoted = ''; continue; }
|
||||||
|
if (!angle && depth === 1 && /\s/.test(text[pos - 1]) && (ch === '"' || ch === "'")) { quoted = ch; continue; }
|
||||||
|
if (ch === '<') { angle = true; continue; }
|
||||||
|
if (ch === '>') { angle = false; continue; }
|
||||||
|
if (!angle && ch === '(') depth++;
|
||||||
|
if (!angle && ch === ')' && --depth === 0) break;
|
||||||
|
}
|
||||||
|
if (depth) return null;
|
||||||
|
let target = text.slice(end + 2, pos).trim();
|
||||||
|
const destination = target.match(/^<([^>]+)>\s*(?:["'][\s\S]*["'])?$/);
|
||||||
|
target = destination ? destination[1] : target.replace(/\s+["'][\s\S]*["']\s*$/, '');
|
||||||
|
target = target.replace(/\\([()\\ ])/g, '$1');
|
||||||
|
return { image, label: text.slice(left, end), target, end: pos + 1 };
|
||||||
|
}
|
||||||
|
|
||||||
|
function inline(text, onImage) {
|
||||||
|
let out = '';
|
||||||
|
for (let pos = 0; pos < text.length;) {
|
||||||
|
if (text[pos] === '\\' && pos + 1 < text.length) { out += text[pos + 1]; pos += 2; continue; }
|
||||||
|
const link = readLink(text, pos);
|
||||||
|
if (link) {
|
||||||
|
if (link.image) out += onImage(link.target, link.label);
|
||||||
|
else if (/^(https?:\/\/|mailto:)/i.test(link.target) && !/[\[\]\r\n"']/.test(link.target)) {
|
||||||
|
out += '[url=' + link.target + ']' + inline(link.label, onImage) + '[/url]';
|
||||||
|
} else out += link.label + ' (' + link.target + ')';
|
||||||
|
pos = link.end;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (text[pos] === '`') {
|
||||||
|
const end = text.indexOf('`', pos + 1);
|
||||||
|
if (end > pos + 1) { out += text.slice(pos + 1, end); pos = end + 1; continue; }
|
||||||
|
}
|
||||||
|
let matched = false;
|
||||||
|
for (const [mark, tag] of [['**', 'b'], ['__', 'b'], ['~~', 's'], ['*', 'i'], ['_', 'i']]) {
|
||||||
|
if (!text.startsWith(mark, pos) || (mark === '_' && /\w/.test(text[pos - 1] || ''))) continue;
|
||||||
|
const end = text.indexOf(mark, pos + mark.length);
|
||||||
|
if (end <= pos + mark.length) continue;
|
||||||
|
out += '[' + tag + ']' + inline(text.slice(pos + mark.length, end), onImage) + '[/' + tag + ']';
|
||||||
|
pos = end + mark.length; matched = true; break;
|
||||||
|
}
|
||||||
|
if (!matched) out += text[pos++];
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function markdownToBBCode(markdown, onImage) {
|
||||||
|
const lines = markdown.replace(/\r\n?/g, '\n').split('\n');
|
||||||
|
const output = [];
|
||||||
|
let list = '', quote = false, code = false, fence = '';
|
||||||
|
const closeList = () => { if (list) output.push('[/list]'); list = ''; };
|
||||||
|
for (let index = 0; index < lines.length; index++) {
|
||||||
|
const line = lines[index];
|
||||||
|
const codeMark = line.match(/^\s*(`{3,}|~{3,})/);
|
||||||
|
if (codeMark && (!code || codeMark[1][0] === fence)) {
|
||||||
|
closeList();
|
||||||
|
output.push(code ? '[/code]' : '[code]');
|
||||||
|
code = !code; fence = codeMark[1][0]; continue;
|
||||||
|
}
|
||||||
|
if (code) { output.push(line.replace(/\[\/code\]/gi, '[/code]')); continue; }
|
||||||
|
const quoted = /^\s*>\s?/.test(line);
|
||||||
|
if (quoted !== quote) { closeList(); output.push(quoted ? '[quote]' : '[/quote]'); quote = quoted; }
|
||||||
|
const content = quoted ? line.replace(/^\s*>\s?/, '') : line;
|
||||||
|
const item = content.match(/^\s*(?:([-+*])|\d+[.)])\s+(.+)$/);
|
||||||
|
if (item) {
|
||||||
|
const type = item[1] ? '[list]' : '[list=1]';
|
||||||
|
if (list !== type) { closeList(); output.push(type); list = type; }
|
||||||
|
output.push('[*]' + inline(item[2], onImage)); continue;
|
||||||
|
}
|
||||||
|
closeList();
|
||||||
|
const heading = content.match(/^\s{0,3}(#{1,6})\s+(.+?)\s*#*$/);
|
||||||
|
if (heading) output.push('[b]' + inline(heading[2], onImage) + '[/b]');
|
||||||
|
else if (/^\s*(?:---+|\*\*\*+|___+)\s*$/.test(content)) output.push('[hr]');
|
||||||
|
else output.push(inline(content, onImage));
|
||||||
|
}
|
||||||
|
closeList();
|
||||||
|
if (quote) output.push('[/quote]');
|
||||||
|
if (code) output.push('[/code]');
|
||||||
|
return output.join('\n').trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function compile(markdown, mdPath, fileMap, titleOverride) {
|
||||||
|
let body = String(markdown).replace(/^\uFEFF/, '').replace(/\r\n?/g, '\n');
|
||||||
|
const firstHeading = body.match(/^\s*#\s+([^\n]+)\n?/);
|
||||||
|
const title = String(titleOverride || firstHeading?.[1] || mdPath.split('/').pop().replace(/\.md$/i, '')).trim();
|
||||||
|
if (firstHeading) body = body.slice(firstHeading[0].length).trim();
|
||||||
|
if (!title || Array.from(title).length > 80) throw new Error('标题为空或超过 80 字,请修改 Markdown 标题后重试。');
|
||||||
|
if (!body.trim()) throw new Error('正文为空。');
|
||||||
|
const warnings = [];
|
||||||
|
if (/<\/?[a-z][^>]*>/i.test(body)) warnings.push('存在 HTML,将作为文字保留;请检查排版。');
|
||||||
|
if (/^\s*\|.*\|\s*$/m.test(body)) warnings.push('表格暂按原文保留,需手动检查。');
|
||||||
|
if (/!\[[^\]]*\]\s*\[[^\]]*\]/.test(body) || /^\s*\[[^\]]+\]:/m.test(body)) {
|
||||||
|
throw new Error('暂不支持引用式 Markdown 图片/链接,请改成  或 [文字](网址)。');
|
||||||
|
}
|
||||||
|
const images = new Map();
|
||||||
|
// Validate every image before any network request; repeated references upload once.
|
||||||
|
markdownToBBCode(body, (target) => {
|
||||||
|
const path = imagePath(mdPath, target);
|
||||||
|
const file = fileMap.get(path);
|
||||||
|
if (!file) throw new Error('找不到配图:' + path);
|
||||||
|
if (!IMAGE_EXT.test(path)) throw new Error('网站仅支持 JPG、JPEG、PNG、GIF:' + path);
|
||||||
|
if (!file.size || file.size >= MAX_IMAGE_SIZE) throw new Error('配图为空或达到 20 MB:' + path);
|
||||||
|
images.set(path, file);
|
||||||
|
return '[图片:' + path + ']';
|
||||||
|
});
|
||||||
|
return { title, body, mdPath, images, warnings,
|
||||||
|
render(aids) {
|
||||||
|
return markdownToBBCode(body, target => {
|
||||||
|
const path = imagePath(mdPath, target);
|
||||||
|
const aid = aids.get(path);
|
||||||
|
if (!/^[1-9]\d*$/.test(String(aid))) throw new Error('配图尚未上传:' + path);
|
||||||
|
return '[attachimg]' + aid + '[/attachimg]';
|
||||||
|
});
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function prepareFiles(files, markdownPath) {
|
||||||
|
const fileMap = new Map();
|
||||||
|
for (const file of Array.from(files)) {
|
||||||
|
const path = normalizePath(file.webkitRelativePath || file.name);
|
||||||
|
if (fileMap.has(path)) throw new Error('文件路径重复:' + path);
|
||||||
|
fileMap.set(path, file);
|
||||||
|
}
|
||||||
|
const candidates = Array.from(fileMap.keys()).filter(p => /\.md$/i.test(p)).sort();
|
||||||
|
if (!candidates.length) throw new Error('所选目录中没有 .md 文件。');
|
||||||
|
if (!markdownPath && candidates.length > 1) return { candidates, files: Array.from(files) };
|
||||||
|
const mdPath = markdownPath || candidates[0];
|
||||||
|
const md = fileMap.get(mdPath);
|
||||||
|
if (!md) throw new Error('指定的 Markdown 文件不存在。');
|
||||||
|
return compile(await md.text(), mdPath, fileMap);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function preparePackage(data) {
|
||||||
|
if (!data || data.schemaVersion !== 1 || typeof data.markdown !== 'string' || !Array.isArray(data.images)) {
|
||||||
|
throw new Error('文章包格式错误,需要 schemaVersion: 1、markdown 和 images。');
|
||||||
|
}
|
||||||
|
const map = new Map();
|
||||||
|
for (const item of data.images) {
|
||||||
|
const path = normalizePath(item.path);
|
||||||
|
if (map.has(path)) throw new Error('文章包图片路径重复:' + path);
|
||||||
|
let blob = item.blob;
|
||||||
|
if (!blob && typeof item.base64 === 'string') {
|
||||||
|
const binary = atob(item.base64);
|
||||||
|
blob = new Blob([Uint8Array.from(binary, ch => ch.charCodeAt(0))], { type: item.mime || 'application/octet-stream' });
|
||||||
|
}
|
||||||
|
if (!blob || typeof blob.arrayBuffer !== 'function') throw new Error('文章包缺少图片数据:' + path);
|
||||||
|
map.set(path, blob);
|
||||||
|
}
|
||||||
|
return compile(data.markdown, normalizePath(data.markdownPath || 'article.md'), map, data.title);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseAid(text) {
|
||||||
|
const value = String(text).trim();
|
||||||
|
// The site's actual WebUploader adapter uses a positive integer response.
|
||||||
|
if (!/^(?:[1-9]\d*|"[1-9]\d*")$/.test(value)) throw new Error('服务器未返回有效附件编号(可能登录过期、超出权限或上传失败)。');
|
||||||
|
return value.replace(/"/g, '');
|
||||||
|
}
|
||||||
|
|
||||||
|
function mount(win) {
|
||||||
|
const doc = win.document;
|
||||||
|
const params = new URLSearchParams(win.location.search);
|
||||||
|
if (params.get('mod') !== 'post' || params.get('action') !== 'newthread' || !doc.querySelector('#subject') || !doc.querySelector('#e_textarea')) return;
|
||||||
|
if (win.DWW11Importer) return;
|
||||||
|
let article = null, busy = false, candidates = null, lastResult = null, savedBackup = null, stageRevision = 0;
|
||||||
|
const cache = new Map(), urls = [], logs = [];
|
||||||
|
const host = doc.createElement('div');
|
||||||
|
host.id = 'dww11-importer';
|
||||||
|
host.style.cssText = 'position:fixed;right:18px;bottom:18px;z-index:2147483000';
|
||||||
|
const shadow = host.attachShadow({ mode: 'open' });
|
||||||
|
shadow.innerHTML = `<style>
|
||||||
|
:host{all:initial}*{box-sizing:border-box}details{font:14px/1.6 system-ui,sans-serif;color:#203044;background:#fff;border:1px solid #d4dee8;border-radius:12px;box-shadow:0 6px 28px #0003;width:360px;max-width:92vw}summary{cursor:pointer;padding:12px 16px;font-weight:650}section{padding:0 16px 16px;max-height:70vh;overflow:auto}button,select{font:inherit;padding:7px 10px;border-radius:6px;border:1px solid #b8c9da;background:#f4f8fc;color:#203044;cursor:pointer;margin:4px 4px 4px 0}button.primary{background:#1767b3;color:white;border-color:#1767b3}button:disabled{opacity:.5;cursor:default}p{margin:8px 0}small{color:#617183}pre{font:12px/1.6 system-ui;white-space:pre-wrap;overflow-wrap:anywhere;max-height:170px;overflow:auto;background:#f4f6f9;padding:8px}img{max-width:90px;max-height:80px;margin:4px;object-fit:contain}.error{color:#b32424}.success{color:#18713e}select{max-width:100%}[hidden]{display:none!important}
|
||||||
|
</style><details open><summary>本地文章导入 · 1.0</summary><section>
|
||||||
|
<small>选择整个文章文件夹,检查后上传配图并填写。发布由你操作。网站自带的自动保存仍可能运行。</small>
|
||||||
|
<p><button id="choose" type="button">选择文章目录</button><button id="package" type="button">导入文章包</button></p>
|
||||||
|
<input id="folder" type="file" webkitdirectory multiple hidden><input id="json" type="file" accept=".json,application/json" hidden>
|
||||||
|
<select id="md" hidden aria-label="选择 Markdown"></select>
|
||||||
|
<p id="status" role="status" aria-live="polite">尚未选择目录</p><div id="preview" hidden><strong id="title"></strong><pre id="body"></pre><div id="images"></div><p id="warnings"></p></div>
|
||||||
|
<button id="import" class="primary" type="button" disabled>上传配图并填写</button>
|
||||||
|
<button id="diagnostics" type="button">导出诊断</button><pre id="log" hidden></pre>
|
||||||
|
</section></details>`;
|
||||||
|
doc.body.appendChild(host);
|
||||||
|
const $ = id => shadow.getElementById(id);
|
||||||
|
const summary = () => ({ version: VERSION, busy, ready: !!article, title: article?.title || null,
|
||||||
|
markdownPath: article?.mdPath || null, imageCount: article?.images.size || 0,
|
||||||
|
warnings: article?.warnings || [], result: lastResult });
|
||||||
|
function notify(phase, message, extra = {}) {
|
||||||
|
$('status').textContent = message;
|
||||||
|
$('status').className = phase === 'error' ? 'error' : phase === 'complete' ? 'success' : '';
|
||||||
|
logs.push({ time: new Date().toISOString(), phase, message, ...extra });
|
||||||
|
$('log').hidden = false;
|
||||||
|
$('log').textContent = logs.slice(-8).map(l => l.message).join('\n');
|
||||||
|
win.dispatchEvent(new CustomEvent('dww11-importer:progress', { detail: { phase, message, ...extra } }));
|
||||||
|
}
|
||||||
|
function releasePreviews() { while (urls.length) URL.revokeObjectURL(urls.pop()); }
|
||||||
|
function clearStage() {
|
||||||
|
article = null; lastResult = null; candidates = null; releasePreviews();
|
||||||
|
$('preview').hidden = true; $('md').hidden = true; $('import').disabled = true;
|
||||||
|
}
|
||||||
|
function showArticle(prepared) {
|
||||||
|
article = prepared; lastResult = null; releasePreviews();
|
||||||
|
$('preview').hidden = false;
|
||||||
|
$('title').textContent = article.title;
|
||||||
|
$('body').textContent = article.body;
|
||||||
|
$('warnings').textContent = article.warnings.join(';');
|
||||||
|
$('images').replaceChildren();
|
||||||
|
for (const [path, blob] of article.images) {
|
||||||
|
const img = doc.createElement('img');
|
||||||
|
img.src = URL.createObjectURL(blob); urls.push(img.src); img.alt = path; img.title = path;
|
||||||
|
$('images').appendChild(img);
|
||||||
|
}
|
||||||
|
$('import').disabled = false;
|
||||||
|
notify('ready', `已检查:${article.images.size} 张图片。点击“上传配图并填写”开始导入。`);
|
||||||
|
return summary();
|
||||||
|
}
|
||||||
|
async function stageFiles(files, options = {}) {
|
||||||
|
if (busy) throw new Error('正在导入,请稍候。');
|
||||||
|
clearStage();
|
||||||
|
const revision = ++stageRevision;
|
||||||
|
const list = Array.from(files);
|
||||||
|
const prepared = await prepareFiles(list, options.markdownPath);
|
||||||
|
if (revision !== stageRevision) return { cancelled: true };
|
||||||
|
if (prepared.candidates) {
|
||||||
|
candidates = prepared.files;
|
||||||
|
$('md').replaceChildren(new Option('请选择目录中的 Markdown 文件', ''));
|
||||||
|
for (const path of prepared.candidates) $('md').appendChild(new Option(path, path));
|
||||||
|
$('md').hidden = false;
|
||||||
|
notify('choose-markdown', '目录中有多个 Markdown 文件,请选择一篇。');
|
||||||
|
return { candidates: prepared.candidates };
|
||||||
|
}
|
||||||
|
return showArticle(prepared);
|
||||||
|
}
|
||||||
|
async function stagePackage(data) {
|
||||||
|
if (busy) throw new Error('正在导入,请稍候。');
|
||||||
|
clearStage();
|
||||||
|
const revision = ++stageRevision;
|
||||||
|
const prepared = await preparePackage(data);
|
||||||
|
if (revision !== stageRevision) return { cancelled: true };
|
||||||
|
return showArticle(prepared);
|
||||||
|
}
|
||||||
|
const snapshot = () => ({ title: doc.querySelector('#subject').value,
|
||||||
|
mode: Number(win.wysiwyg), text: win.getEditorContents() });
|
||||||
|
function assertEditor() {
|
||||||
|
for (const name of ['getEditorContents', 'switchEditor']) {
|
||||||
|
if (typeof win[name] !== 'function') throw new Error('未识别到网站编辑器:' + name + '。请刷新页面后重试。');
|
||||||
|
}
|
||||||
|
if (!win.imgUpload?.settings || typeof win.updateImageList !== 'function') throw new Error('网站图片上传组件尚未就绪,请刷新后重试。');
|
||||||
|
if (Number(win.allowbbcode) === 0 || Number(win.allowimgcode) === 0) throw new Error('当前版块或账号不支持正文格式/图片。');
|
||||||
|
}
|
||||||
|
async function uploadImage(blob, path, settings, signal) {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
const url = new URL(settings.upload_url, win.location.href);
|
||||||
|
if (url.origin !== win.location.origin || url.pathname !== '/misc.php' || url.searchParams.get('operation') !== 'upload') {
|
||||||
|
throw new Error('上传地址与预期不符,已停止。');
|
||||||
|
}
|
||||||
|
const hash = Array.from(new Uint8Array(await win.crypto.subtle.digest('SHA-256', await blob.arrayBuffer())), n => n.toString(16).padStart(2, '0')).join('');
|
||||||
|
const key = url.href + ':' + settings.post_params.uid + ':' + hash;
|
||||||
|
if (cache.has(key)) { notify('upload', '复用已上传图片:' + path); return cache.get(key); }
|
||||||
|
const data = new FormData();
|
||||||
|
for (const [key, value] of Object.entries(settings.post_params)) data.append(key, String(value));
|
||||||
|
data.set('type', 'image');
|
||||||
|
data.set('filetype', blob.type || (/\.png$/i.test(path) ? 'image/png' : /\.gif$/i.test(path) ? 'image/gif' : 'image/jpeg'));
|
||||||
|
data.append(settings.file_post_name || 'Filedata', blob, path.split('/').pop());
|
||||||
|
const control = new AbortController(), timer = setTimeout(() => control.abort(), 90000);
|
||||||
|
try {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
signal?.addEventListener('abort', () => control.abort(), { once: true });
|
||||||
|
const response = await win.fetch(url.href, { method: 'POST', body: data, credentials: 'same-origin', redirect: 'error', signal: control.signal });
|
||||||
|
if (!response.ok) throw new Error('图片上传请求失败:HTTP ' + response.status);
|
||||||
|
const aid = parseAid(await response.text());
|
||||||
|
cache.set(key, aid);
|
||||||
|
return aid;
|
||||||
|
} catch (error) {
|
||||||
|
if (error.name === 'AbortError') throw new Error('上传超时;该图片可能已到达服务器,请检查网站图片列表后再重试。');
|
||||||
|
throw error;
|
||||||
|
} finally { clearTimeout(timer); }
|
||||||
|
}
|
||||||
|
async function registerImages(aids, signal) {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
if (!aids.length) return;
|
||||||
|
const ids = new Set(aids);
|
||||||
|
doc.querySelectorAll('#imgattachlist [id^="image_td_"],#imgattachlist [id^="image_"]').forEach(el => {
|
||||||
|
const id = el.id.match(/^(?:image_td_|image_)(\d+)$/)?.[1]; if (id) ids.add(id);
|
||||||
|
});
|
||||||
|
const ready = () => aids.every(id => doc.getElementById('image_' + id));
|
||||||
|
if (ready()) return;
|
||||||
|
// Use the site's own list loader, which registers attachment form fields too.
|
||||||
|
win.updateImageList(1, Array.from(ids).join('|'));
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
const timer = setTimeout(() => { observer.disconnect(); reject(new Error('图片已上传,但网站图片列表加载超时。重试会复用本页已上传图片。')); }, 25000);
|
||||||
|
signal?.addEventListener('abort', () => { clearTimeout(timer); observer.disconnect(); reject(new Error('流程已终止')); }, { once: true });
|
||||||
|
const check = () => { if (ready()) { clearTimeout(timer); observer.disconnect(); resolve(); } };
|
||||||
|
const observer = new MutationObserver(check);
|
||||||
|
observer.observe(doc.body, { childList: true, subtree: true }); check();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function setEditor(title, bbcode) {
|
||||||
|
win.switchEditor(0);
|
||||||
|
if (Number(win.wysiwyg) !== 0) throw new Error('无法切换编辑器为纯文本模式。');
|
||||||
|
const textarea = doc.querySelector('#e_textarea');
|
||||||
|
textarea.value = bbcode;
|
||||||
|
doc.querySelector('#subject').value = title;
|
||||||
|
for (const el of [textarea, doc.querySelector('#subject')]) {
|
||||||
|
el.dispatchEvent(new Event('input', { bubbles: true })); el.dispatchEvent(new Event('change', { bubbles: true }));
|
||||||
|
}
|
||||||
|
// Let Discuz generate the WYSIWYG content with native attachment metadata.
|
||||||
|
win.switchEditor(1);
|
||||||
|
const switcher = doc.querySelector('#e_switchercheck');
|
||||||
|
if (switcher) switcher.checked = Number(win.wysiwyg) === 0;
|
||||||
|
}
|
||||||
|
async function importPrepared(options = {}) {
|
||||||
|
const { signal, automatic = false } = options;
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
if (busy) throw new Error('正在导入,请勿重复点击。');
|
||||||
|
if (!article) throw new Error('请先选择文章目录或文章包。');
|
||||||
|
assertEditor();
|
||||||
|
for (const id of ['bbcodeoff', 'htmlon']) {
|
||||||
|
if (doc.getElementById(id)?.checked) throw new Error('请先关闭“禁用 Discuz! 代码”和“HTML 代码”,再导入。');
|
||||||
|
}
|
||||||
|
const before = snapshot();
|
||||||
|
if ((before.title.trim() || before.text.replace(/<[^>]*>/g, '').trim() || /<img\b/i.test(before.text)) &&
|
||||||
|
(automatic ? (() => { throw new Error('编辑器已有内容,请人工处理旧稿后重试'); })() : !win.confirm('当前编辑器已有内容。导入将替换标题和正文,是否继续?'))) return { cancelled: true };
|
||||||
|
const selected = article;
|
||||||
|
busy = true;
|
||||||
|
const controls = ['choose', 'package', 'import', 'md'].map($);
|
||||||
|
controls.forEach(el => { el.disabled = true; });
|
||||||
|
// Prevent accidental submission of a partly imported article.
|
||||||
|
const blockSubmit = event => { event.preventDefault(); event.stopImmediatePropagation(); notify('busy', '正在导入,请等待完成后再操作发布。'); };
|
||||||
|
const form = doc.querySelector('#postform');
|
||||||
|
form.addEventListener('submit', blockSubmit, true);
|
||||||
|
const submitButtons = Array.from(form.querySelectorAll('button[type="submit"],button[name],input[type="submit"]'));
|
||||||
|
const previousDisabled = submitButtons.map(el => el.disabled);
|
||||||
|
submitButtons.forEach(el => { el.disabled = true; });
|
||||||
|
let committed = false;
|
||||||
|
try {
|
||||||
|
const aids = new Map();
|
||||||
|
const settings = win.imgUpload.settings;
|
||||||
|
const max = Math.min(MAX_IMAGE_SIZE, Number(settings.file_size_limit) * 1024 || MAX_IMAGE_SIZE);
|
||||||
|
for (const [path, blob] of selected.images) if (blob.size >= max) throw new Error('配图超过当前网站限制:' + path);
|
||||||
|
let index = 0;
|
||||||
|
for (const [path, blob] of selected.images) {
|
||||||
|
notify('upload', `上传 ${++index}/${selected.images.size}:${path}`, { completed: index - 1, total: selected.images.size });
|
||||||
|
aids.set(path, await uploadImage(blob, path, settings, signal));
|
||||||
|
}
|
||||||
|
notify('register', '正在加载网站图片列表…');
|
||||||
|
await registerImages(Array.from(new Set(aids.values())), signal);
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
const now = snapshot();
|
||||||
|
if ((now.title !== before.title || now.text !== before.text) && (automatic ? (() => { throw new Error('上传期间编辑器被修改,已保留当前内容'); })() : !win.confirm('上传期间你修改了编辑器。是否用导入文章替换最新内容?'))) {
|
||||||
|
notify('cancelled', '已保留当前正文;图片已上传,可稍后再次导入。'); return { cancelled: true };
|
||||||
|
}
|
||||||
|
const backup = snapshot();
|
||||||
|
const bbcode = selected.render(aids);
|
||||||
|
// Preserve a local, downloadable snapshot before changing title or body.
|
||||||
|
savedBackup = { title: backup.title, content: backup.text, mode: backup.mode, createdAt: new Date().toISOString() };
|
||||||
|
try {
|
||||||
|
for (const id of ['bbcodeoff', 'htmlon']) {
|
||||||
|
const el = doc.getElementById(id);
|
||||||
|
if (el && id === 'bbcodeoff' && el.checked) throw new Error('请先取消“禁用 Discuz! 代码”,再重试。');
|
||||||
|
if (el && id === 'htmlon' && el.checked) throw new Error('请先关闭“HTML 代码”,再重试。');
|
||||||
|
}
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
setEditor(selected.title, bbcode);
|
||||||
|
const raw = win.getEditorContents();
|
||||||
|
if (Number(win.wysiwyg) === 1) {
|
||||||
|
for (const id of new Set(aids.values())) if (!raw.includes('attachimg_' + id)) throw new Error('富文本图片转换未通过检查。');
|
||||||
|
} else if (raw !== bbcode) throw new Error('正文填写校验失败。');
|
||||||
|
if (doc.querySelector('#subject').value !== selected.title) throw new Error('标题填写校验失败。');
|
||||||
|
committed = true;
|
||||||
|
} catch (error) {
|
||||||
|
// Restore the exact editor representation saved immediately before filling.
|
||||||
|
win.switchEditor(backup.mode);
|
||||||
|
if (backup.mode) win.editdoc.body.innerHTML = backup.text;
|
||||||
|
else doc.querySelector('#e_textarea').value = backup.text;
|
||||||
|
doc.querySelector('#subject').value = backup.title;
|
||||||
|
const switcher = doc.querySelector('#e_switchercheck'); if (switcher) switcher.checked = !backup.mode;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
lastResult = { title: selected.title, images: Object.fromEntries(aids), completedAt: new Date().toISOString() };
|
||||||
|
notify('complete', `已填写标题、正文和 ${aids.size} 张配图。请检查排版,随后手动发表。`);
|
||||||
|
return lastResult;
|
||||||
|
} catch (error) {
|
||||||
|
notify('error', error.message + (committed ? '' : ' 标题和正文未替换;已上传的图片会保留。'));
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
busy = false; form.removeEventListener('submit', blockSubmit, true);
|
||||||
|
submitButtons.forEach((el, index) => { el.disabled = previousDisabled[index]; });
|
||||||
|
controls.forEach(el => { el.disabled = false; });
|
||||||
|
$('import').disabled = !article;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function downloadJSON(value, filename) {
|
||||||
|
const url = URL.createObjectURL(new Blob([JSON.stringify(value, null, 2)], { type: 'application/json' }));
|
||||||
|
const link = doc.createElement('a'); link.href = url; link.download = filename; link.click();
|
||||||
|
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||||
|
}
|
||||||
|
// Public v1 interface. Staging is read-only; importPrepared is the explicit upload/fill step.
|
||||||
|
// No postMessage listener, localhost service, publishing endpoint or saved credentials.
|
||||||
|
const api = Object.freeze({ version: VERSION, schemaVersion: 1, stageFiles, stagePackage,
|
||||||
|
getState: summary, openPicker: () => { if (!busy) $('folder').click(); }, importPrepared,
|
||||||
|
getBackup: () => savedBackup });
|
||||||
|
win.DWW11Importer = api;
|
||||||
|
$('choose').onclick = () => { $('folder').value = ''; $('folder').click(); };
|
||||||
|
$('package').onclick = () => { $('json').value = ''; $('json').click(); };
|
||||||
|
const handleError = error => notify('error', error.message || String(error));
|
||||||
|
$('folder').onchange = () => stageFiles($('folder').files).catch(handleError);
|
||||||
|
$('json').onchange = async () => {
|
||||||
|
try { const file = $('json').files[0]; if (file) await stagePackage(JSON.parse(await file.text())); }
|
||||||
|
catch (error) { handleError(error); }
|
||||||
|
};
|
||||||
|
$('md').onchange = () => {
|
||||||
|
const files = candidates, path = $('md').value;
|
||||||
|
if (files && path) stageFiles(files, { markdownPath: path }).catch(handleError);
|
||||||
|
};
|
||||||
|
$('import').onclick = () => importPrepared().catch(handleError);
|
||||||
|
$('diagnostics').onclick = () => downloadJSON({ ...summary(), logs }, 'dww11-import-diagnostics.json');
|
||||||
|
win.addEventListener('pagehide', releasePreviews, { once: true });
|
||||||
|
}
|
||||||
|
return { VERSION, normalizePath, imagePath, readLink, markdownToBBCode, compile, prepareFiles, preparePackage, parseAid, mount };
|
||||||
|
});
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
// ==UserScript==
|
||||||
|
// @name 新闻工作台 · 本地连接
|
||||||
|
// @namespace local.dww11.auto-news
|
||||||
|
// @version 0.1.0
|
||||||
|
// @match https://www.dww11.com/*
|
||||||
|
// @match https://dww11.com/*
|
||||||
|
// @match https://house.dww11.com/*
|
||||||
|
// @grant GM_xmlhttpRequest
|
||||||
|
// @grant GM_getValue
|
||||||
|
// @grant GM_setValue
|
||||||
|
// @grant GM_addValueChangeListener
|
||||||
|
// @connect 127.0.0.1
|
||||||
|
// @run-at document-start
|
||||||
|
// @noframes
|
||||||
|
// ==/UserScript==
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
const allowed = new Set(['heartbeat','next','package','event','accept','switch-begin','takeover','stop','current']);
|
||||||
|
let token = GM_getValue('pairToken',''), port = GM_getValue('pairPort',8770);
|
||||||
|
let stopRevision = 0;
|
||||||
|
const signalStop = () => { stopRevision++; document.dispatchEvent(new CustomEvent('dww11:auto-stop')); };
|
||||||
|
GM_addValueChangeListener('halted', (_key, _old, value) => { if (value) signalStop(); });
|
||||||
|
function request(route, body) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
if (!token) return reject(new Error('请在左下角填入本地工作台的连接码'));
|
||||||
|
GM_xmlhttpRequest({method:'POST',url:`http://127.0.0.1:${port}/bridge/${route}`,
|
||||||
|
headers:{'Content-Type':'application/json','X-Session-Token':token},data:JSON.stringify(body),timeout:60000,
|
||||||
|
onload(response) { try { const value=JSON.parse(response.responseText); if(response.status!==200) throw new Error(value.error || '本地连接失败'); resolve(value); } catch(error) { reject(error); } },
|
||||||
|
onerror:()=>reject(new Error('本地服务未连接,请检查服务与脚本权限')),
|
||||||
|
ontimeout:()=>reject(new Error('本地服务超时,任务已暂停'))});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
document.addEventListener('dww11:local-request', async event => {
|
||||||
|
let message;
|
||||||
|
try {message=JSON.parse(event.detail);} catch {return;}
|
||||||
|
if (!allowed.has(message.route) || typeof message.id !== 'string') return;
|
||||||
|
try {
|
||||||
|
if (message.route==='stop') { GM_setValue('halted',true); signalStop(); }
|
||||||
|
const revision = stopRevision;
|
||||||
|
let value=await request(message.route,message.body || {});
|
||||||
|
if (message.route==='heartbeat') {
|
||||||
|
if (revision !== stopRevision) value = {...value, running:false, reason:'终止后忽略旧连接响应'};
|
||||||
|
GM_setValue('halted',!value.running);
|
||||||
|
if (!value.running) signalStop();
|
||||||
|
}
|
||||||
|
document.dispatchEvent(new CustomEvent('dww11:local-response',{detail:JSON.stringify({id:message.id,value})}));
|
||||||
|
} catch(error) {
|
||||||
|
if (message.route==='heartbeat') {GM_setValue('halted',true); signalStop();}
|
||||||
|
document.dispatchEvent(new CustomEvent('dww11:local-response',{detail:JSON.stringify({id:message.id,error:error.message})}));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
function mount() {
|
||||||
|
if (!document.body || document.getElementById('auto-news-connection')) return;
|
||||||
|
const host=document.createElement('div');host.id='auto-news-connection';
|
||||||
|
host.style.cssText='position:fixed;left:16px;bottom:16px;z-index:2147483647';
|
||||||
|
const shadow=host.attachShadow({mode:'closed'});
|
||||||
|
shadow.innerHTML='<style>section{font:13px/1.6 system-ui;background:white;color:#24364a;border:1px solid #ccd7e0;border-radius:10px;padding:12px;width:250px;box-shadow:0 3px 20px #0002}button,input{font:inherit;box-sizing:border-box;padding:7px;margin:4px 0;width:100%}button{cursor:pointer}#halt{background:#ad302d;color:white;border:0;border-radius:6px}small{display:block}</style><section><strong>新闻工作台 · 本地连接</strong><button id="halt">终止流程</button><details><summary>连接设置</summary><label>本地端口<input id="port" type="number" min="1" max="65535"></label><label>连接码<input id="token" type="password" autocomplete="off"></label><button id="save">保存连接设置</button></details><small id="status">请启动本地工作台并配对</small></section>';
|
||||||
|
const $=id=>shadow.getElementById(id);$('port').value=port;
|
||||||
|
$('save').onclick=()=>{
|
||||||
|
const value=Number($('port').value);if(!Number.isInteger(value)||value<1||value>65535){$('status').textContent='端口无效';return;}
|
||||||
|
if($('token').value.trim()) token=$('token').value.trim();port=value;
|
||||||
|
GM_setValue('pairToken',token);GM_setValue('pairPort',port);$('token').value='';$('status').textContent='已保存;服务重启后需要更新连接码';
|
||||||
|
};
|
||||||
|
$('halt').onclick=async()=>{
|
||||||
|
GM_setValue('halted',true);signalStop();$('status').textContent='已终止;在途请求仍可能完成';
|
||||||
|
try {await request('stop',{});} catch(error) {$('status').textContent+='。'+error.message;}
|
||||||
|
};
|
||||||
|
document.addEventListener('dww11:connection-status',event=>{$('status').textContent=String(event.detail).slice(0,250);});
|
||||||
|
document.body.append(host);
|
||||||
|
if(GM_getValue('halted',true)) signalStop();
|
||||||
|
}
|
||||||
|
if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',mount,{once:true}); else mount();
|
||||||
|
})();
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
/* Page-side orchestrator. This file never clicks or submits the publication form. */
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
if (window.DWW11NewsRunner) return;
|
||||||
|
window.DWW11NewsRunner = true;
|
||||||
|
const SLOT='auto-news-current';
|
||||||
|
const owner=sessionStorage.getItem('auto-news-owner') || crypto.randomUUID();
|
||||||
|
sessionStorage.setItem('auto-news-owner',owner);
|
||||||
|
let run=null, working=false, halted=true, aborter=null, active=null, tickBusy=false;
|
||||||
|
const pending=new Map();
|
||||||
|
function rpc(route,body={}) {
|
||||||
|
return new Promise((resolve,reject)=>{
|
||||||
|
const id=crypto.randomUUID();
|
||||||
|
const timer=setTimeout(()=>{pending.delete(id);reject(new Error('本地连接无响应'));},65000);
|
||||||
|
pending.set(id,{resolve,reject,timer});
|
||||||
|
document.dispatchEvent(new CustomEvent('dww11:local-request',{detail:JSON.stringify({id,route,body:{owner,...body}})}));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
document.addEventListener('dww11:local-response',event=>{
|
||||||
|
let data;try{data=JSON.parse(event.detail);}catch{return;}
|
||||||
|
const waiter=pending.get(data.id);if(!waiter)return;
|
||||||
|
clearTimeout(waiter.timer);pending.delete(data.id);
|
||||||
|
data.error?waiter.reject(new Error(data.error)):waiter.resolve(data.value);
|
||||||
|
});
|
||||||
|
const status=message=>document.dispatchEvent(new CustomEvent('dww11:connection-status',{detail:message}));
|
||||||
|
document.addEventListener('dww11:auto-stop',()=>{halted=true;aborter?.abort(new Error('流程已终止'));});
|
||||||
|
function account() {
|
||||||
|
const element=document.querySelector('#mumucms_username');
|
||||||
|
if(!element)return null;
|
||||||
|
const name=[...element.childNodes].filter(n=>n.nodeType===Node.TEXT_NODE).map(n=>n.textContent).join('').trim();
|
||||||
|
const links=[element,...element.querySelectorAll('a')].filter(e=>e.href);
|
||||||
|
let uid=null;
|
||||||
|
for(const link of links) {
|
||||||
|
const href=link.href;
|
||||||
|
uid=new URL(href,location.href).searchParams.get('uid') || href.match(/space-uid-(\d+)/)?.[1];
|
||||||
|
if(uid)break;
|
||||||
|
}
|
||||||
|
return uid&&name?{uid,name}:null;
|
||||||
|
}
|
||||||
|
function extension() {
|
||||||
|
try{return JSON.parse(document.querySelector('#dww11-extension-panel')?.getAttribute('data-auto-news-state') || '{}');}catch{return {};}
|
||||||
|
}
|
||||||
|
function assertLive(epoch) {
|
||||||
|
if(halted||!run?.running||run.epoch!==epoch)throw new Error('流程已暂停或终止');
|
||||||
|
if(active?.account?.uid!==account()?.uid)throw new Error('当前账号与任务不一致');
|
||||||
|
}
|
||||||
|
function editorSnapshot() {
|
||||||
|
const body=document.querySelector('#e_iframe')?.contentDocument?.body;
|
||||||
|
if(!body)throw new Error('找不到可视化编辑器');
|
||||||
|
return {title:document.querySelector('#subject')?.value || '',
|
||||||
|
images:[...body.querySelectorAll('img')].map(x=>({id:x.id,src:x.getAttribute('src')})),
|
||||||
|
// Snapshot only final submitted content; never includes hidden tokens or the full form.
|
||||||
|
html:body.innerHTML.slice(0,500000), text:body.innerText.slice(0,100000)};
|
||||||
|
}
|
||||||
|
function saveLocal(value) {sessionStorage.setItem(SLOT,JSON.stringify(value));}
|
||||||
|
function loadLocal(){try{return JSON.parse(sessionStorage.getItem(SLOT)||'null');}catch{return null;}}
|
||||||
|
async function event(state,extra={}) {return rpc('event',{id:active.id,state,epoch:run.epoch,...extra});}
|
||||||
|
async function fail(error) {
|
||||||
|
status(error.message);
|
||||||
|
if(active)try{await event('needs_attention',{error:error.message});}catch{}
|
||||||
|
halted=true;aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));
|
||||||
|
}
|
||||||
|
function editorPage(job) {
|
||||||
|
const u=new URL(location.href);
|
||||||
|
return u.searchParams.get('mod')==='post'&&u.searchParams.get('action')==='newthread'&&u.searchParams.get('fid')===String(job.fid);
|
||||||
|
}
|
||||||
|
async function prepare(job) {
|
||||||
|
working=true;active=job;aborter=new AbortController();const epoch=run.epoch;
|
||||||
|
try {
|
||||||
|
assertLive(epoch);
|
||||||
|
if(!editorPage(job)) {
|
||||||
|
saveLocal({id:job.id});
|
||||||
|
location.assign(`https://www.dww11.com/forum.php?mod=post&action=newthread&fid=${job.fid}`);return;
|
||||||
|
}
|
||||||
|
if(!window.DWW11Importer||!window.DWW11Formatter)throw new Error('导入或排版模块尚未就绪,请检查安装');
|
||||||
|
status('正在检查文章包…');
|
||||||
|
const pack=await rpc('package',{id:job.id});assertLive(epoch);
|
||||||
|
const prepared=await window.DWW11Importer.stagePackage(pack);assertLive(epoch);
|
||||||
|
if(prepared.warnings?.length)throw new Error('文章需要人工处理:'+prepared.warnings.join(';'));
|
||||||
|
await event('importing');assertLive(epoch);status('上传图片并填写文章…');
|
||||||
|
const result=await window.DWW11Importer.importPrepared({automatic:true,signal:aborter.signal});assertLive(epoch);
|
||||||
|
if(result?.cancelled)throw new Error('导入已取消');
|
||||||
|
await event('formatting');assertLive(epoch);
|
||||||
|
window.DWW11Formatter.format(job.source);assertLive(epoch);
|
||||||
|
const snapshot=editorSnapshot();
|
||||||
|
if(snapshot.title!==job.title)throw new Error('标题校验失败');
|
||||||
|
const aids=new Set(Object.values(result.images));
|
||||||
|
for(const aid of aids) if(!snapshot.images.some(x=>x.id==='attachimg_'+aid))throw new Error('配图附件缺失:'+aid);
|
||||||
|
await event('awaiting_user_review',{snapshot});assertLive(epoch);
|
||||||
|
active={...job,state:'awaiting_user_review'};saveLocal({id:job.id});
|
||||||
|
status('已完成排版,请核对内容并亲手点击“发表帖子”。');
|
||||||
|
}catch(error){await fail(error);}finally{working=false;}
|
||||||
|
}
|
||||||
|
// Record only trusted user activation. Never replay a click, call submit(), or send a post request.
|
||||||
|
document.addEventListener('click',event=>{
|
||||||
|
if(!event.isTrusted||!event.target.closest?.('#postsubmit')||active?.state!=='awaiting_user_review')return;
|
||||||
|
try {
|
||||||
|
const snapshot=editorSnapshot();
|
||||||
|
const local={id:active.id,snapshot,clickedAt:Date.now(),observed:false};saveLocal(local);
|
||||||
|
rpc('event',{id:active.id,state:'user_submitting',epoch:run?.epoch,snapshot})
|
||||||
|
.then(()=>{const latest=loadLocal();if(latest?.clickedAt===local.clickedAt)saveLocal({...local,observed:true});})
|
||||||
|
.catch(error=>status('提交记录待恢复:'+error.message));
|
||||||
|
active.state='user_submitting';status('已观察到人工点击,等待结果页核对。');
|
||||||
|
}catch(error){status('提交快照记录失败:'+error.message);}
|
||||||
|
},true);
|
||||||
|
async function observeResult(job) {
|
||||||
|
const local=loadLocal();
|
||||||
|
if(local?.id===job.id&&local.snapshot&&job.state==='awaiting_user_review') {
|
||||||
|
await rpc('event',{id:job.id,state:'user_submitting',epoch:run.epoch,snapshot:local.snapshot});
|
||||||
|
job.state='user_submitting';
|
||||||
|
}
|
||||||
|
if(job.state==='user_submitting') {
|
||||||
|
// Result selectors intentionally await user-provided HTML; URL alone is not proof.
|
||||||
|
if(!editorPage(job) || (local?.clickedAt && Date.now()-local.clickedAt>45000)) {
|
||||||
|
await rpc('event',{id:job.id,state:'submit_unknown',error:'等待结果页 HTML 适配;候选页面:'+location.href});
|
||||||
|
await rpc('stop');
|
||||||
|
status('已记录候选结果页;尚不能自动核对。流程已暂停,请补充结果页 HTML。');
|
||||||
|
}
|
||||||
|
}else if(job.state==='submit_unknown')status('发布结果待核对;不会重发、计数或换号。');
|
||||||
|
}
|
||||||
|
async function switchAccount() {
|
||||||
|
if(working||halted)return;
|
||||||
|
if(location.origin!=='https://www.dww11.com'||location.pathname!=='/') {
|
||||||
|
location.assign('https://www.dww11.com/');return;
|
||||||
|
}
|
||||||
|
const ext=extension(),button=document.querySelector('#dww11-extension-next');
|
||||||
|
if(!ext.ready||!ext.idle||!button||button.disabled) {status('请安装并重新加载带联动接口的换号插件');return;}
|
||||||
|
await rpc('switch-begin');
|
||||||
|
if(halted||!run?.running)return;
|
||||||
|
button.click();status('正在换号;等待插件完成并核对新账号');
|
||||||
|
}
|
||||||
|
async function tick() {
|
||||||
|
if(tickBusy)return;tickBusy=true;
|
||||||
|
try {
|
||||||
|
run=await rpc('heartbeat',{account:account(),url:location.href,extension:extension()});
|
||||||
|
halted=!run.running;
|
||||||
|
if(!run.running) {aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));}
|
||||||
|
if(working)return;
|
||||||
|
// Passive result recording is permitted after Stop, but no new business actions.
|
||||||
|
if(run.active) {
|
||||||
|
const local=loadLocal();
|
||||||
|
if(local?.id===run.active&&local.snapshot&&!run.running)status('流程已暂停;提交结果仍需核对');
|
||||||
|
}
|
||||||
|
if(!run.running) {
|
||||||
|
if(run.active && run.owner===owner) {const saved=await rpc('current');if(saved.job) {active=saved.job;await observeResult(saved.job);}}
|
||||||
|
status(run.reason || '流程已暂停');return;
|
||||||
|
}
|
||||||
|
const next=await rpc('next',{account:account(),extension:extension()});run=next.run;
|
||||||
|
if(next.job)active=next.job;
|
||||||
|
if(next.action==='prepare'||(next.action==='resume'&&next.job.state==='preparing'))void prepare(next.job);
|
||||||
|
else if(next.action==='resume') {
|
||||||
|
await observeResult(next.job);
|
||||||
|
if(next.job.state==='awaiting_user_review') {
|
||||||
|
if(!editorPage(next.job)||!document.querySelector('#subject')?.value) await fail(new Error('待审稿页面已丢失,请人工核对旧稿,不自动重新导入'));
|
||||||
|
else status('等待你亲手发表;程序不再修改编辑器');
|
||||||
|
}
|
||||||
|
else if(['importing','formatting'].includes(next.job.state))await fail(new Error('导入阶段发生中断,请检查编辑器和已上传图片后处理任务'));
|
||||||
|
else if(next.job.state==='needs_attention')status(next.job.error);
|
||||||
|
}else if(next.action==='switch')await switchAccount();
|
||||||
|
else if(next.action==='switch_wait')status('等待换号结果;若插件已暂停,请人工处理后继续');
|
||||||
|
else if(next.action==='login')status('未能读取当前账号 UID,请登录并确认账号元素');
|
||||||
|
else if(next.action==='other_tab')status('另一个标签持有任务;如需接管请先暂停');
|
||||||
|
else if(next.action==='interval')status('等待设定的处理间隔');
|
||||||
|
else status(run.reason || '等待任务');
|
||||||
|
}catch(error){status(error.message);halted=true;aborter?.abort();document.dispatchEvent(new CustomEvent('dww11:auto-stop'));}
|
||||||
|
finally{tickBusy=false;}
|
||||||
|
}
|
||||||
|
const takeover=document.createElement('button');takeover.type='button';takeover.textContent='接管已暂停任务';
|
||||||
|
takeover.style.cssText='position:fixed;left:16px;top:16px;z-index:2147483646;padding:8px;background:#fff;border:1px solid #aab;border-radius:6px';
|
||||||
|
takeover.onclick=async()=>{try{await rpc('takeover');status('已接管;核对当前页面后在工作台继续');}catch(error){status(error.message);}};
|
||||||
|
document.body.append(takeover);
|
||||||
|
setInterval(tick,2000);tick();
|
||||||
|
})();
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
"""Build the page-context userscript from reviewed local components; no external assets."""
|
||||||
|
from pathlib import Path
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
header = '''// ==UserScript==
|
||||||
|
// @name 新闻工作台 · 导入排版与流程
|
||||||
|
// @namespace local.dww11.auto-news
|
||||||
|
// @version 0.1.0
|
||||||
|
// @match https://www.dww11.com/*
|
||||||
|
// @match https://dww11.com/*
|
||||||
|
// @match https://house.dww11.com/*
|
||||||
|
// @grant none
|
||||||
|
// @run-at document-idle
|
||||||
|
// @noframes
|
||||||
|
// ==/UserScript==
|
||||||
|
'''
|
||||||
|
parts = []
|
||||||
|
for name in ('importer.js', 'formatter.js', 'runner.js'):
|
||||||
|
source = (ROOT / 'browser' / name).read_text()
|
||||||
|
if '// ==/UserScript==' in source:
|
||||||
|
source = source.split('// ==/UserScript==', 1)[1]
|
||||||
|
parts.append(source)
|
||||||
|
(ROOT / 'browser/auto-news.user.js').write_text(header + '\n'.join(parts))
|
||||||
|
print('已生成 browser/auto-news.user.js')
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# 探数新闻下载器
|
||||||
|
|
||||||
|
在本机浏览器中填写 API Key、获取频道、选择数量,下载新闻正文和图片。所有项目文件与下载结果都位于本目录,不修改微信下载器。
|
||||||
|
|
||||||
|
## 启动
|
||||||
|
|
||||||
|
安装 `uv` 后,macOS 可以双击 `start.command`。也可以在本目录运行:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
启动后自动打开 http://127.0.0.1:8765 。保留启动窗口,退出时按 Ctrl+C。端口被占用时加 `--port 8766`;不自动打开浏览器时加 `--no-browser`。
|
||||||
|
|
||||||
|
1. 填写探数 API Key,点击「获取新闻频道」。频道来自真实接口,不使用写死的列表。
|
||||||
|
2. 选择频道、输入目标数量,点击「开始下载」。
|
||||||
|
3. 页面显示成功保存的数量、百分比、重复数、失败数和最近 300 条日志。日志固定高度,可滚动;向上查看时不会强制滚回底部。
|
||||||
|
4. 下载结果保存在本目录的 `downloads` 文件夹。点击「停止下载」后,等待当前请求或文章保存结束。
|
||||||
|
5. 「调试终端」默认折叠,支持频道查询以及列表查询的 `type`、`num`、`start` 参数。测试请求只显示响应,不保存文章;同样消耗 API 调用额度。输出最多保留 100000 字符。
|
||||||
|
|
||||||
|
密钥仅在页面和服务运行内存中使用,不保存到配置、历史数据库或浏览器存储。服务仅监听本机地址。刷新页面可查看当前任务进度,开始新任务需重新填写密钥。单个服务同时只运行一个下载任务,请勿同时启动多个实例操作同一下载目录。
|
||||||
|
|
||||||
|
## 保存格式
|
||||||
|
|
||||||
|
沿用 `wechat_official_acc_article_downloader` 的格式和图片处理规则:
|
||||||
|
|
||||||
|
```text
|
||||||
|
downloads/
|
||||||
|
├── .history.sqlite3
|
||||||
|
└── 新闻标题/
|
||||||
|
├── 新闻标题.md
|
||||||
|
├── image001.jpg
|
||||||
|
└── image002.png
|
||||||
|
```
|
||||||
|
|
||||||
|
Markdown 开头是一级标题,空行后直接接由接口 `content` 转换的正文,不额外添加接口 `time` 对应的发布时间或分隔线。末尾为 `来源:{src},如有侵权,请联系删除。`,来源为空时使用「未知作者」。不在开头添加原文链接或作者。正文中的图片下载到同一目录、使用相对引用、按真实文件类型命名,同一图片地址只下载一次。接口的 `pic` 封面不单独插入正文。
|
||||||
|
|
||||||
|
标题中的不合法字符会替换;不同文章碰巧同名时添加 ` (2)` 等后缀,目录和 Markdown 文件保持同名。任何正文图片下载失败,该篇不计入完成数,并清理半成品,再继续其他新闻;下次任务可以重试。
|
||||||
|
|
||||||
|
## 去重与分页
|
||||||
|
|
||||||
|
- 下载图片前,检查原文 URL / PC URL(移除常见跟踪参数和片段)及规范化正文指纹。任一匹配即跳过,标题相同但正文和网址不同不会单凭标题去重。
|
||||||
|
- SQLite 保存成功文章的指纹,跨任务、跨频道和服务重启后继续去重。被删除的文章目录不会阻止重新下载。不要删除 `.history.sqlite3`,否则会失去对应历史记录。
|
||||||
|
- 每次任务从最新位置 `start=0` 开始,按返回数量推进,每页最多 40 条;重复和失败不计入成功数量,并继续后面的新闻。
|
||||||
|
- 接口限制 `start ≤ 400`,一次任务最多覆盖约 440 条,具体还受返回数量和实时更新影响。达到分页边界或收到空列表时停止,未达目标就显示「未达到目标」,不会伪装成 100%。
|
||||||
|
- 列表接口不自动重试,防止隐蔽消耗计费额度。API 业务错误、网络错误或响应格式错误会中断任务,已成功保存的文章保留。图片请求沿用原下载器的重试策略。
|
||||||
|
- 新闻列表会实时变化,无法保证服务器在不同请求中不返回同一新闻;本工具在保存前去重,尽量避免重复图片下载与文件保存。
|
||||||
|
|
||||||
|
接口依据:[探数官方新闻 API 文档](https://www.tanshuapi.com/market/detail-85)。
|
||||||
|
|
||||||
|
## 验证
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python -m unittest discover -s tests -v
|
||||||
|
```
|
||||||
|
|
||||||
|
测试使用模拟新闻响应与临时目录,不消耗真实 API 额度。覆盖格式与图片引用、图片失败清理、历史去重、分页边界、失败重试、取消、日志上限和本地 HTTP 接口。真实账号权限、余额、返回内容与远程图片可达性,需要在页面填入有效 Key 后验证。
|
||||||
@@ -0,0 +1,299 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""仅在本机运行的探数新闻下载器。"""
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from collections import deque
|
||||||
|
from datetime import datetime
|
||||||
|
from hashlib import sha256
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import secrets
|
||||||
|
import sqlite3
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import unicodedata
|
||||||
|
import webbrowser
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from article_writer import save_article
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
API = 'https://api.tanshuapi.com/api/toutiao/v1/'
|
||||||
|
|
||||||
|
|
||||||
|
def api_request(key, endpoint, params=None):
|
||||||
|
if endpoint not in ('channel', 'index'):
|
||||||
|
raise ValueError('未知接口')
|
||||||
|
try:
|
||||||
|
# 不自动重试计费接口,避免一次操作产生不可见的重复调用。
|
||||||
|
response = requests.get(API + endpoint, params={'key': key, **(params or {})}, timeout=(10, 40))
|
||||||
|
response.raise_for_status()
|
||||||
|
payload = response.json()
|
||||||
|
except requests.RequestException:
|
||||||
|
raise ValueError('接口网络请求失败或超时,请在调试终端检查后重试') from None
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError('接口未返回有效 JSON') from None
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise ValueError('接口响应格式错误')
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def result_list(payload):
|
||||||
|
if payload.get('code') != 1:
|
||||||
|
raise ValueError(f"探数接口错误 {payload.get('code')}:{payload.get('msg', '未知错误')}")
|
||||||
|
data = payload.get('data')
|
||||||
|
if not isinstance(data, dict) or not isinstance(data.get('list'), list):
|
||||||
|
raise ValueError('接口响应缺少 data.list')
|
||||||
|
return data['list']
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprints(article):
|
||||||
|
values = []
|
||||||
|
for field in ('url', 'weburl'):
|
||||||
|
value = str(article.get(field) or '').strip()
|
||||||
|
if value:
|
||||||
|
u = urlsplit(value)
|
||||||
|
query = [(k, v) for k, v in parse_qsl(u.query) if not k.lower().startswith('utm_') and k.lower() not in ('spm', 'from', 'pos', 'vt')]
|
||||||
|
values.append('url:' + urlunsplit(('', u.netloc.lower(), u.path, urlencode(sorted(query)), '')))
|
||||||
|
body = BeautifulSoup(str(article.get('content') or ''), 'html.parser').get_text(' ', strip=True)
|
||||||
|
body = ''.join(unicodedata.normalize('NFKC', body).split())
|
||||||
|
if body:
|
||||||
|
values.append('body:' + body)
|
||||||
|
return [sha256(value.encode()).hexdigest() for value in values]
|
||||||
|
|
||||||
|
|
||||||
|
class Downloader:
|
||||||
|
def __init__(self, root=ROOT / 'downloads', requester=api_request, writer=save_article, on_saved=None):
|
||||||
|
self.root = Path(root).resolve()
|
||||||
|
self.root.mkdir(parents=True, exist_ok=True)
|
||||||
|
self.requester, self.writer = requester, writer
|
||||||
|
self.on_saved = on_saved
|
||||||
|
self.lock = threading.RLock()
|
||||||
|
self.stop = threading.Event()
|
||||||
|
self.thread = None
|
||||||
|
self.logs = deque(maxlen=300)
|
||||||
|
self.seq = 0
|
||||||
|
self.state = dict(status='idle', saved=0, target=0, skipped=0, failed=0, scanned=0, requests=0, channel='', message='等待开始')
|
||||||
|
with self.db() as db:
|
||||||
|
db.execute('CREATE TABLE IF NOT EXISTS seen (fingerprint TEXT PRIMARY KEY, directory TEXT NOT NULL)')
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def db(self):
|
||||||
|
connection = sqlite3.connect(self.root / '.history.sqlite3')
|
||||||
|
try:
|
||||||
|
with connection:
|
||||||
|
yield connection
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
def log(self, message):
|
||||||
|
with self.lock:
|
||||||
|
self.seq += 1
|
||||||
|
self.logs.append(dict(id=self.seq, time=datetime.now().strftime('%H:%M:%S'), message=message))
|
||||||
|
|
||||||
|
def update(self, **values):
|
||||||
|
with self.lock:
|
||||||
|
self.state.update(values)
|
||||||
|
|
||||||
|
def snapshot(self):
|
||||||
|
with self.lock:
|
||||||
|
return {**self.state, 'logs': list(self.logs), 'output': str(self.root)}
|
||||||
|
|
||||||
|
def start(self, key, channel, target):
|
||||||
|
with self.lock:
|
||||||
|
if self.thread and self.thread.is_alive():
|
||||||
|
raise ValueError('已有任务正在运行')
|
||||||
|
self.stop.clear()
|
||||||
|
self.logs.clear()
|
||||||
|
self.state = dict(status='running', saved=0, target=target, skipped=0, failed=0, scanned=0, requests=0, channel=channel, message='正在下载')
|
||||||
|
self.thread = threading.Thread(target=self.run, args=(key, channel, target), daemon=True)
|
||||||
|
self.thread.start()
|
||||||
|
|
||||||
|
def run(self, key, channel, target):
|
||||||
|
saved = skipped = failed = scanned = calls = 0
|
||||||
|
seen_in_run = set()
|
||||||
|
self.log(f'开始下载频道「{channel}」,目标 {target} 条;历史重复新闻将跳过')
|
||||||
|
try:
|
||||||
|
with self.db() as db:
|
||||||
|
offset = 0
|
||||||
|
while offset <= 400 and saved < target and not self.stop.is_set():
|
||||||
|
count = min(40, target - saved)
|
||||||
|
calls += 1
|
||||||
|
self.update(requests=calls)
|
||||||
|
self.log(f'请求第 {calls} 页:start={offset},num={count}')
|
||||||
|
items = result_list(self.requester(key, 'index', {'type': channel, 'num': count, 'start': offset}))
|
||||||
|
if not items:
|
||||||
|
self.log('接口返回空列表,已停止')
|
||||||
|
break
|
||||||
|
for article in items:
|
||||||
|
if saved >= target or self.stop.is_set():
|
||||||
|
break
|
||||||
|
scanned += 1
|
||||||
|
try:
|
||||||
|
if not isinstance(article, dict):
|
||||||
|
raise ValueError('新闻条目格式错误')
|
||||||
|
keys = fingerprints(article)
|
||||||
|
existing = any(k in seen_in_run for k in keys)
|
||||||
|
for k in keys:
|
||||||
|
row = db.execute('SELECT directory FROM seen WHERE fingerprint=?', (k,)).fetchone()
|
||||||
|
if row and Path(row[0]).is_dir():
|
||||||
|
existing = True
|
||||||
|
seen_in_run.update(keys)
|
||||||
|
if existing:
|
||||||
|
skipped += 1
|
||||||
|
self.log('跳过重复新闻:' + str(article.get('title', '无标题'))[:160])
|
||||||
|
continue
|
||||||
|
result = self.writer(article, self.root)
|
||||||
|
try:
|
||||||
|
db.executemany('INSERT OR REPLACE INTO seen VALUES (?, ?)', [(k, result.directory) for k in keys])
|
||||||
|
db.commit()
|
||||||
|
except Exception:
|
||||||
|
# 保存目录尚未登记成功,回滚本次产物,避免下次重复保存。
|
||||||
|
import shutil
|
||||||
|
shutil.rmtree(result.directory, ignore_errors=True)
|
||||||
|
raise
|
||||||
|
saved += 1
|
||||||
|
if self.on_saved:
|
||||||
|
try:
|
||||||
|
self.on_saved(article, result, keys)
|
||||||
|
except Exception as error:
|
||||||
|
self.log('文章已保存,入队需处理:' + str(error).replace(key, '[已隐藏]')[:300])
|
||||||
|
self.log(f'下载新闻 {saved}/{target} 条:{result.title}')
|
||||||
|
except Exception as exc:
|
||||||
|
failed += 1
|
||||||
|
self.log(f'保存失败:{str(exc).replace(key, "[已隐藏]")[:400]}')
|
||||||
|
finally:
|
||||||
|
self.update(saved=saved, skipped=skipped, failed=failed, scanned=scanned)
|
||||||
|
# 避免越过 start=400;已消费的条目决定下一起始位置。
|
||||||
|
offset += len(items)
|
||||||
|
if offset > 400:
|
||||||
|
self.log('已到达接口允许的分页范围')
|
||||||
|
if saved < target and offset <= 400:
|
||||||
|
self.stop.wait(0.3)
|
||||||
|
status = 'completed' if saved >= target else 'cancelled' if self.stop.is_set() else 'partial'
|
||||||
|
message = f'已保存 {saved}/{target} 条,跳过重复 {skipped} 条,失败 {failed} 条'
|
||||||
|
if status == 'partial':
|
||||||
|
message += ';当前可获取新闻不足,未达到目标数量'
|
||||||
|
if status == 'cancelled':
|
||||||
|
message += ';任务已停止'
|
||||||
|
self.update(status=status, message=message)
|
||||||
|
self.log(message)
|
||||||
|
except Exception as exc:
|
||||||
|
message = str(exc).replace(key, '[已隐藏]')[:500]
|
||||||
|
self.update(status='error', message=message)
|
||||||
|
self.log('任务中断:' + message)
|
||||||
|
|
||||||
|
|
||||||
|
def make_handler(manager, token):
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
def log_message(self, *args):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def send(self, status, payload, mime='application/json; charset=utf-8'):
|
||||||
|
data = json.dumps(payload, ensure_ascii=False).encode() if isinstance(payload, dict) else payload
|
||||||
|
self.send_response(status)
|
||||||
|
self.send_header('Content-Type', mime)
|
||||||
|
self.send_header('Content-Length', str(len(data)))
|
||||||
|
self.send_header('Cache-Control', 'no-store')
|
||||||
|
self.send_header('X-Content-Type-Options', 'nosniff')
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(data)
|
||||||
|
|
||||||
|
def local(self):
|
||||||
|
return self.headers.get('Host') in (f'127.0.0.1:{self.server.server_port}', f'localhost:{self.server.server_port}')
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
if not self.local():
|
||||||
|
return self.send(403, {'error': '仅允许本机访问'})
|
||||||
|
path = urlsplit(self.path).path
|
||||||
|
if path == '/api/state':
|
||||||
|
return self.send(200, manager.snapshot())
|
||||||
|
if path == '/api/session':
|
||||||
|
return self.send(200, {'token': token})
|
||||||
|
files = {'/': ('index.html', 'text/html; charset=utf-8'), '/app.js': ('app.js', 'text/javascript; charset=utf-8'), '/style.css': ('style.css', 'text/css; charset=utf-8')}
|
||||||
|
if path not in files:
|
||||||
|
return self.send(404, {'error': '未找到页面'})
|
||||||
|
name, mime = files[path]
|
||||||
|
self.send(200, (ROOT / 'static' / name).read_bytes(), mime)
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
if not self.local() or self.headers.get('X-Session-Token') != token:
|
||||||
|
return self.send(403, {'error': '页面会话失效,请刷新页面'})
|
||||||
|
key = ''
|
||||||
|
try:
|
||||||
|
length = int(self.headers.get('Content-Length', 0))
|
||||||
|
if not 0 < length <= 16384:
|
||||||
|
raise ValueError('请求大小无效')
|
||||||
|
body = json.loads(self.rfile.read(length))
|
||||||
|
if not isinstance(body, dict):
|
||||||
|
raise ValueError('请求必须是 JSON 对象')
|
||||||
|
if self.path == '/api/stop':
|
||||||
|
manager.stop.set()
|
||||||
|
return self.send(200, {'ok': True})
|
||||||
|
key = str(body.get('key') or '').strip()
|
||||||
|
if not key or len(key) > 512:
|
||||||
|
raise ValueError('请填写有效 API Key')
|
||||||
|
if self.path == '/api/channels':
|
||||||
|
channels = result_list(manager.requester(key, 'channel'))
|
||||||
|
if not channels or not all(isinstance(c, str) and c for c in channels):
|
||||||
|
raise ValueError('频道列表为空或格式错误')
|
||||||
|
return self.send(200, {'channels': channels})
|
||||||
|
if self.path == '/api/start':
|
||||||
|
target = body.get('target')
|
||||||
|
channel = body.get('channel')
|
||||||
|
if type(target) is not int or not 1 <= target <= 100000:
|
||||||
|
raise ValueError('下载数量须为 1 至 100000 的整数')
|
||||||
|
if not isinstance(channel, str) or not channel.strip() or len(channel) > 100:
|
||||||
|
raise ValueError('请选择新闻频道')
|
||||||
|
manager.start(key, channel, target)
|
||||||
|
return self.send(200, {'ok': True})
|
||||||
|
if self.path == '/api/debug':
|
||||||
|
endpoint = body.get('endpoint')
|
||||||
|
params = {}
|
||||||
|
if endpoint == 'index':
|
||||||
|
for field, low, high in [('num', 1, 40), ('start', 0, 400)]:
|
||||||
|
value = body.get(field)
|
||||||
|
if type(value) is not int or not low <= value <= high:
|
||||||
|
raise ValueError(f'{field} 必须在 {low}–{high} 之间')
|
||||||
|
params[field] = value
|
||||||
|
params['type'] = str(body.get('channel') or '')
|
||||||
|
before = time.monotonic()
|
||||||
|
payload = manager.requester(key, endpoint, params)
|
||||||
|
# 服务端响应同样隐藏 key,且只作为文本展示,不渲染上游 HTML。
|
||||||
|
clean = json.loads(json.dumps(payload, ensure_ascii=False).replace(key, '[已隐藏]'))
|
||||||
|
return self.send(200, {'elapsed': round(time.monotonic() - before, 2), 'request': API + str(endpoint) + '?' + urlencode({'key': '[已隐藏]', **params}), 'response': clean})
|
||||||
|
self.send(404, {'error': '未知操作'})
|
||||||
|
except Exception as exc:
|
||||||
|
message = str(exc)
|
||||||
|
if key:
|
||||||
|
message = message.replace(key, '[已隐藏]')
|
||||||
|
self.send(400, {'error': message[:500]})
|
||||||
|
return Handler
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument('--port', type=int, default=8765)
|
||||||
|
parser.add_argument('--no-browser', action='store_true')
|
||||||
|
args = parser.parse_args()
|
||||||
|
manager = Downloader()
|
||||||
|
server = ThreadingHTTPServer(('127.0.0.1', args.port), make_handler(manager, secrets.token_urlsafe(32)))
|
||||||
|
url = f'http://127.0.0.1:{server.server_port}'
|
||||||
|
print(f'新闻下载器:{url}\n保存目录:{manager.root}\n按 Ctrl+C 退出', flush=True)
|
||||||
|
if not args.no_browser:
|
||||||
|
webbrowser.open(url)
|
||||||
|
try:
|
||||||
|
server.serve_forever()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
manager.stop.set()
|
||||||
|
if manager.thread:
|
||||||
|
manager.thread.join(timeout=45)
|
||||||
|
finally:
|
||||||
|
server.server_close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,212 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""将探数新闻正文保存为与微信下载器相同的 Markdown 目录格式。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import tempfile
|
||||||
|
from urllib.parse import unquote, urljoin, urlsplit
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from markdownify import markdownify
|
||||||
|
from requests.adapters import HTTPAdapter
|
||||||
|
from urllib3.util.retry import Retry
|
||||||
|
|
||||||
|
|
||||||
|
class ArticleDownloadError(RuntimeError):
|
||||||
|
"""页面不是文章、图片下载失败等可向调用方报告的错误。"""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ArticleResult:
|
||||||
|
title: str
|
||||||
|
source_url: str
|
||||||
|
directory: str
|
||||||
|
markdown: str
|
||||||
|
images: list[str]
|
||||||
|
warnings: list[str]
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
def safe_name(title: str) -> str:
|
||||||
|
"""跨平台目录/文件名;目录名与其中 Markdown 的文件名保持一致。"""
|
||||||
|
name = re.sub(r'[<>:"/\\|?*\x00-\x1f\x7f]', '_', title)
|
||||||
|
name = re.sub(r'\s+', ' ', name).strip(' .')
|
||||||
|
# 同时限制 UTF-8 字节数,给 .md 和重名后缀保留空间。
|
||||||
|
name = name.encode('utf-8')[:180].decode('utf-8', errors='ignore').rstrip(' .')
|
||||||
|
if not name:
|
||||||
|
name = '未命名文章'
|
||||||
|
if re.fullmatch(r'(?i:CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(?:\..*)?', name):
|
||||||
|
name = '_' + name
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def _session() -> requests.Session:
|
||||||
|
session = requests.Session()
|
||||||
|
session.headers.update({
|
||||||
|
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
|
||||||
|
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',
|
||||||
|
'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
|
||||||
|
})
|
||||||
|
retry = Retry(total=3, backoff_factor=0.6,
|
||||||
|
status_forcelist=[429, 500, 502, 503, 504], allowed_methods=['GET'])
|
||||||
|
session.mount('https://', HTTPAdapter(max_retries=retry))
|
||||||
|
session.mount('http://', HTTPAdapter(max_retries=retry))
|
||||||
|
return session
|
||||||
|
|
||||||
|
|
||||||
|
def _get(session: requests.Session, url: str, timeout: float, limit: int,
|
||||||
|
referer: str | None = None) -> tuple[bytes, str, str]:
|
||||||
|
if urlsplit(url).scheme not in ('http', 'https'):
|
||||||
|
raise ArticleDownloadError(f'不支持的下载地址:{url[:200]}')
|
||||||
|
headers = {'Referer': referer} if referer else {}
|
||||||
|
with session.get(url, timeout=(min(timeout, 15), timeout),
|
||||||
|
headers=headers, stream=True) as response:
|
||||||
|
response.raise_for_status()
|
||||||
|
chunks, size = [], 0
|
||||||
|
for chunk in response.iter_content(65536):
|
||||||
|
size += len(chunk)
|
||||||
|
if size > limit:
|
||||||
|
raise ArticleDownloadError(f'文件超过大小限制({limit // 1024 // 1024} MB):{url[:200]}')
|
||||||
|
chunks.append(chunk)
|
||||||
|
return b''.join(chunks), response.headers.get('Content-Type', ''), response.url
|
||||||
|
|
||||||
|
|
||||||
|
def _image_ext(data: bytes, mime: str) -> str:
|
||||||
|
if data.startswith(b'\xff\xd8\xff'):
|
||||||
|
return '.jpg'
|
||||||
|
if data.startswith(b'\x89PNG\r\n\x1a\n'):
|
||||||
|
return '.png'
|
||||||
|
if data.startswith((b'GIF87a', b'GIF89a')):
|
||||||
|
return '.gif'
|
||||||
|
if data[:4] == b'RIFF' and data[8:12] == b'WEBP':
|
||||||
|
return '.webp'
|
||||||
|
if len(data) >= 12 and data[4:8] == b'ftyp' and data[8:12] in (b'avif', b'avis'):
|
||||||
|
return '.avif'
|
||||||
|
if b'<svg' in data[:1024].lower() and mime.split(';')[0] == 'image/svg+xml':
|
||||||
|
return '.svg'
|
||||||
|
raise ArticleDownloadError(f'返回内容不是支持的图片(Content-Type: {mime or "未知"})')
|
||||||
|
|
||||||
|
|
||||||
|
def _image_source(img) -> str:
|
||||||
|
for attr in ('data-src', 'data-original', 'data-actualsrc', 'data-lazy-src', 'src'):
|
||||||
|
value = img.get(attr)
|
||||||
|
if value and str(value).strip():
|
||||||
|
return str(value).strip()
|
||||||
|
srcset = img.get('data-srcset') or img.get('srcset')
|
||||||
|
if srcset:
|
||||||
|
return str(srcset).split(',')[-1].strip().split()[0]
|
||||||
|
return ''
|
||||||
|
|
||||||
|
|
||||||
|
def save_article(article: dict, output_root: str | Path = '.', *, timeout: float = 30,
|
||||||
|
allow_missing_images: bool = False) -> ArticleResult:
|
||||||
|
"""从 API 正文保存文章;图片失败时清理半成品,不覆盖已有同名目录。"""
|
||||||
|
url = str(article.get('weburl') or article.get('url') or '')
|
||||||
|
title = str(article.get('title') or '').strip()
|
||||||
|
raw_content = str(article.get('content') or '').strip()
|
||||||
|
if not title or not raw_content:
|
||||||
|
raise ArticleDownloadError('新闻标题或正文为空')
|
||||||
|
if timeout <= 0:
|
||||||
|
raise ValueError('timeout 必须大于 0')
|
||||||
|
with _session() as session:
|
||||||
|
final_url = url
|
||||||
|
soup = BeautifulSoup('<article>' + raw_content + '</article>', 'html.parser')
|
||||||
|
content = soup.article
|
||||||
|
for tag in content.select('script, style, noscript, iframe, form, button'):
|
||||||
|
tag.decompose()
|
||||||
|
# 部分微信排版把插图放在行内 CSS 背景中,也转为普通图片参与下载。
|
||||||
|
for tag in list(content.find_all(style=True)):
|
||||||
|
for match in re.finditer(r'url\(\s*[\'\"]?(.*?)[\'\"]?\s*\)', str(tag['style'])):
|
||||||
|
source = match.group(1).strip()
|
||||||
|
if source and not source.startswith('#'):
|
||||||
|
background = soup.new_tag('img', src=source, alt='')
|
||||||
|
tag.insert(0, background)
|
||||||
|
if not content.get_text(strip=True) and not content.find('img'):
|
||||||
|
raise ArticleDownloadError('文章正文为空,可能需要浏览器执行 JavaScript')
|
||||||
|
for link in content.find_all('a', href=True):
|
||||||
|
href = str(link['href']).strip()
|
||||||
|
if href and not href.startswith('#'):
|
||||||
|
absolute = urljoin(final_url, href)
|
||||||
|
if urlsplit(absolute).scheme in ('http', 'https', 'mailto', 'tel'):
|
||||||
|
link['href'] = absolute
|
||||||
|
else:
|
||||||
|
del link['href']
|
||||||
|
|
||||||
|
root = Path(output_root).expanduser().resolve()
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
stage = Path(tempfile.mkdtemp(prefix='.article-', dir=root))
|
||||||
|
images, warnings, downloaded = [], [], {}
|
||||||
|
try:
|
||||||
|
for img in list(content.find_all('img')):
|
||||||
|
source = _image_source(img)
|
||||||
|
if not source:
|
||||||
|
img.replace_with(soup.new_string('[图片缺少下载地址]'))
|
||||||
|
if not allow_missing_images:
|
||||||
|
raise ArticleDownloadError('正文中存在没有下载地址的图片')
|
||||||
|
warnings.append('正文中存在没有下载地址的图片')
|
||||||
|
continue
|
||||||
|
image_url = urljoin(final_url, source)
|
||||||
|
try:
|
||||||
|
if image_url not in downloaded:
|
||||||
|
if source.startswith('data:image/'):
|
||||||
|
header, payload = source.split(',', 1)
|
||||||
|
data = base64.b64decode(payload, validate=True) if ';base64' in header else unquote(payload).encode()
|
||||||
|
mime = header[5:].split(';')[0]
|
||||||
|
if len(data) > 50 * 1024 * 1024:
|
||||||
|
raise ArticleDownloadError('内嵌图片超过 50 MB')
|
||||||
|
else:
|
||||||
|
data, mime, _ = _get(session, image_url, timeout, 50 * 1024 * 1024, final_url)
|
||||||
|
ext = _image_ext(data, mime)
|
||||||
|
filename = f'image{len(images) + 1:03d}{ext}'
|
||||||
|
(stage / filename).write_bytes(data)
|
||||||
|
images.append(filename)
|
||||||
|
downloaded[image_url] = filename
|
||||||
|
# 清除懒加载/响应式属性,使转换器只引用本地文件。
|
||||||
|
alt = str(img.get('alt') or '')
|
||||||
|
img.attrs = {'src': downloaded[image_url], 'alt': alt}
|
||||||
|
except (requests.RequestException, ArticleDownloadError, ValueError) as exc:
|
||||||
|
if not allow_missing_images:
|
||||||
|
raise ArticleDownloadError(f'图片下载失败:{image_url[:200]};{exc}') from exc
|
||||||
|
warnings.append(f'图片下载失败:{image_url[:200]};{exc}')
|
||||||
|
img.replace_with(soup.new_string(f'[图片下载失败:{img.get("alt") or "未命名图片"}]'))
|
||||||
|
|
||||||
|
# 新闻图片后常带空的图注标题,避免生成单独的 ## 和多余空行。
|
||||||
|
for heading in content.select('h1, h2, h3, h4, h5, h6'):
|
||||||
|
if not heading.get_text(strip=True) and not heading.find('img'):
|
||||||
|
heading.decompose()
|
||||||
|
body = markdownify(str(content), heading_style='ATX', bullets='-',
|
||||||
|
strip=['picture', 'source']).strip()
|
||||||
|
# 微信富文本中各段常没有标准 p 标签;markdownify 仍会保留文字和图片顺序。
|
||||||
|
author = str(article.get('src') or '未知作者').strip() or '未知作者'
|
||||||
|
text = f'# {title}\n\n'
|
||||||
|
text += body + f'\n\n来源:{author},如有侵权,请联系删除。\n'
|
||||||
|
basename = safe_name(title)
|
||||||
|
# 原子认领新目录,支持多个进程并发保存同名文章。
|
||||||
|
index = 1
|
||||||
|
while True:
|
||||||
|
name = basename if index == 1 else f'{basename} ({index})'
|
||||||
|
destination = root / name
|
||||||
|
try:
|
||||||
|
destination.mkdir()
|
||||||
|
break
|
||||||
|
except FileExistsError:
|
||||||
|
index += 1
|
||||||
|
try:
|
||||||
|
(stage / f'{name}.md').write_text(text, encoding='utf-8')
|
||||||
|
for item in stage.iterdir():
|
||||||
|
shutil.move(str(item), destination / item.name)
|
||||||
|
except BaseException:
|
||||||
|
shutil.rmtree(destination, ignore_errors=True)
|
||||||
|
raise
|
||||||
|
return ArticleResult(title, url, str(destination), str(destination / f'{name}.md'),
|
||||||
|
[str(destination / x) for x in images], warnings)
|
||||||
|
finally:
|
||||||
|
shutil.rmtree(stage, ignore_errors=True)
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
requests>=2.32,<3
|
||||||
|
beautifulsoup4>=4.12,<5
|
||||||
|
markdownify>=1.1,<2
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
'use strict';
|
||||||
|
const $ = id => document.getElementById(id);
|
||||||
|
let token = '', running = false, starting = false, loadingChannels = false, loadedKey = '', lastLog = -1;
|
||||||
|
const labels = {idle:'等待开始',running:'下载中',completed:'已完成',partial:'未达到目标',cancelled:'已停止',error:'任务中断'};
|
||||||
|
const key = () => $('key').value.trim();
|
||||||
|
function controls() {
|
||||||
|
const busy = running || starting;
|
||||||
|
$('start').disabled = busy || !loadedKey || loadedKey !== key() || !$('channel').value;
|
||||||
|
$('stop').disabled = !running;
|
||||||
|
$('channels').disabled = busy || loadingChannels;
|
||||||
|
for (const id of ['key','channel','target']) $(id).disabled = busy || (id === 'channel' && !loadedKey);
|
||||||
|
}
|
||||||
|
async function post(path, body) {
|
||||||
|
const response = await fetch('/api/' + path, {method:'POST',headers:{'Content-Type':'application/json','X-Session-Token':token},body:JSON.stringify(body)});
|
||||||
|
const data = await response.json();
|
||||||
|
if (!response.ok) throw new Error(data.error || '请求失败');
|
||||||
|
return data;
|
||||||
|
}
|
||||||
|
$('key').addEventListener('input', () => {loadedKey=''; $('channel').replaceChildren(new Option('请重新获取频道','')); controls();});
|
||||||
|
$('channels').addEventListener('click', async () => {
|
||||||
|
loadingChannels=true; controls(); $('notice').textContent='正在获取频道…';
|
||||||
|
const submittedKey = key();
|
||||||
|
try {
|
||||||
|
const data = await post('channels',{key:submittedKey});
|
||||||
|
if (key() !== submittedKey) return;
|
||||||
|
$('channel').replaceChildren(...data.channels.map(c => new Option(c,c)));
|
||||||
|
loadedKey=submittedKey; $('notice').textContent=`已获取 ${data.channels.length} 个频道。`;
|
||||||
|
$('debug-channel').value=$('channel').value;
|
||||||
|
} catch(error) {$('notice').textContent=error.message;}
|
||||||
|
finally {loadingChannels=false; controls();}
|
||||||
|
});
|
||||||
|
$('channel').addEventListener('change', () => {$('debug-channel').value=$('channel').value; controls();});
|
||||||
|
$('download-form').addEventListener('submit', async event => {
|
||||||
|
event.preventDefault(); starting=true; controls(); $('notice').textContent='';
|
||||||
|
try {await post('start',{key:key(),channel:$('channel').value,target:Number($('target').value)}); await refresh();}
|
||||||
|
catch(error) {$('notice').textContent=error.message;}
|
||||||
|
finally {starting=false; controls();}
|
||||||
|
});
|
||||||
|
$('stop').addEventListener('click', async () => {
|
||||||
|
$('stop').disabled=true;
|
||||||
|
try {await post('stop',{}); $('notice').textContent='正在停止,等待当前请求或文章保存结束…';}
|
||||||
|
catch(error) {$('notice').textContent=error.message;}
|
||||||
|
});
|
||||||
|
$('debug-form').addEventListener('submit', async event => {
|
||||||
|
event.preventDefault(); $('debug-submit').disabled=true; $('debug-output').textContent='正在请求…';
|
||||||
|
try {
|
||||||
|
const data=await post('debug',{key:key(),endpoint:$('endpoint').value,channel:$('debug-channel').value,num:Number($('debug-num').value),start:Number($('debug-start').value)});
|
||||||
|
const output=JSON.stringify(data,null,2);
|
||||||
|
$('debug-output').textContent=output.slice(0,100000)+(output.length>100000?'\n…输出超过 100000 字符,已截断。':'');
|
||||||
|
} catch(error) {$('debug-output').textContent=error.message;}
|
||||||
|
finally {$('debug-submit').disabled=false;}
|
||||||
|
});
|
||||||
|
async function refresh() {
|
||||||
|
const response=await fetch('/api/state');
|
||||||
|
if (!response.ok) throw new Error('无法读取下载状态');
|
||||||
|
const data=await response.json();
|
||||||
|
running=data.status==='running'; controls();
|
||||||
|
const percent=data.target ? Math.floor(data.saved/data.target*100) : 0;
|
||||||
|
$('status').textContent=labels[data.status] || data.status;
|
||||||
|
$('progress-text').textContent=`下载新闻 ${data.saved}/${data.target} 条`;
|
||||||
|
$('percent').textContent=`已完成 ${percent}%`; $('progress').value=percent;
|
||||||
|
for (const id of ['saved','skipped','failed','requests','message','output']) $(id).textContent=data[id];
|
||||||
|
const latest=data.logs.length ? data.logs[data.logs.length-1].id : 0;
|
||||||
|
if (latest!==lastLog) {
|
||||||
|
const logs=$('logs'); const bottom=logs.scrollHeight-logs.scrollTop-logs.clientHeight<40;
|
||||||
|
logs.replaceChildren(...data.logs.map(log => {const row=document.createElement('div'); const stamp=document.createElement('time'); stamp.textContent=log.time; row.append(stamp,document.createTextNode(log.message)); return row;}));
|
||||||
|
if (bottom) logs.scrollTop=logs.scrollHeight;
|
||||||
|
lastLog=latest;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async function poll() {
|
||||||
|
try {await refresh();}
|
||||||
|
catch(error) {$('notice').textContent='与本地服务连接中断,请检查启动窗口。';}
|
||||||
|
setTimeout(poll,700);
|
||||||
|
}
|
||||||
|
(async () => {
|
||||||
|
try {const response=await fetch('/api/session'); token=(await response.json()).token; await poll();}
|
||||||
|
catch(error) {$('notice').textContent='初始化失败,请刷新页面。';}
|
||||||
|
})();
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>探数 · 新闻下载器</title><link rel="stylesheet" href="/style.css"><script src="/app.js" defer></script></head>
|
||||||
|
<body><main>
|
||||||
|
<header><div class="brand">TANSHU / NEWS ARCHIVE</div><h1>把新闻,保存到本地。</h1><p>选择频道,收集新闻。正文与图片一起保存,自动跳过已下载的内容。</p></header>
|
||||||
|
<section class="card"><div class="section-heading"><h2>下载设置</h2><span class="badge">本地运行</span></div>
|
||||||
|
<form id="download-form"><label for="key">API Key</label><div class="key-row"><input id="key" type="password" placeholder="填写探数 API Key" autocomplete="off" required maxlength="512"><button id="channels" type="button" class="secondary">获取新闻频道</button></div><p class="hint">密钥仅用于本次页面会话,不写入文件或浏览器存储。</p>
|
||||||
|
<div class="fields"><div><label for="channel">新闻频道</label><select id="channel" required disabled><option value="">先填写密钥并获取频道</option></select></div><div><label for="target">下载数量</label><input id="target" type="number" min="1" max="100000" step="1" value="10" required></div></div>
|
||||||
|
<p class="hint">接口每次最多 40 条,起始位置最多 400。遇到重复会继续向后查找;可用新闻不足时会说明原因。</p><div class="actions"><button id="start" disabled>开始下载</button><button id="stop" type="button" class="secondary" disabled>停止下载</button></div><p id="notice" role="alert"></p></form></section>
|
||||||
|
<section class="card"><div class="section-heading"><h2>下载进度</h2><span id="status" class="badge">等待开始</span></div><div class="progress-heading"><strong id="progress-text">下载新闻 0/0 条</strong><span id="percent">已完成 0%</span></div><progress id="progress" max="100" value="0" aria-label="下载进度"></progress><div class="stats"><span>已保存 <b id="saved">0</b></span><span>重复跳过 <b id="skipped">0</b></span><span>保存失败 <b id="failed">0</b></span><span>接口请求 <b id="requests">0</b></span></div><p id="message">等待开始</p><p class="hint path">保存位置:<span id="output">加载中…</span></p></section>
|
||||||
|
<section class="card"><div class="section-heading"><h2>运行输出</h2><span class="hint">保留最近 300 条</span></div><div id="logs" class="console" role="log" aria-label="运行输出"><div class="empty">运行情况将在这里显示。</div></div></section>
|
||||||
|
<details class="card"><summary>调试终端 <span>手动请求接口</span></summary><p class="hint">使用上方 API Key。请求会消耗接口调用额度,但不会保存新闻。</p><form id="debug-form"><div class="fields"><div><label for="endpoint">请求接口</label><select id="endpoint"><option value="channel">新闻频道查询 /channel</option><option value="index">新闻列表查询 /index</option></select></div><div><label for="debug-channel">type · 频道</label><input id="debug-channel" value="头条"></div><div><label for="debug-num">num · 数量(1–40)</label><input id="debug-num" type="number" min="1" max="40" value="1" required></div><div><label for="debug-start">start · 起始位置(0–400)</label><input id="debug-start" type="number" min="0" max="400" value="0" required></div></div><button id="debug-submit" class="secondary">发送测试请求</button></form><pre id="debug-output" class="console">尚未发送请求。</pre></details>
|
||||||
|
<footer>Markdown 正文 · 本地图片 · 历史去重</footer>
|
||||||
|
</main></body></html>
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
:root{font-family:Inter,-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif;color:#20312e;background:#f3f6f3;font-size:15px}*{box-sizing:border-box}body{margin:0}main{max-width:960px;margin:0 auto;padding:48px 24px 24px}header{margin-bottom:32px}.brand{font-size:12px;letter-spacing:2px;font-weight:700;color:#437b69}h1{font-size:36px;letter-spacing:-1px;margin:14px 0}header p{color:#687a74;line-height:1.8}h2{font-size:17px;margin:0}.card{padding:26px 28px;background:#fff;border:1px solid #dfe7e1;border-radius:16px;margin-bottom:20px;box-shadow:0 4px 16px #243c2904}.section-heading{display:flex;align-items:center;justify-content:space-between;gap:10px;margin-bottom:24px}.badge{background:#edf4ef;color:#4c7564;padding:6px 10px;border-radius:20px;font-size:12px}.key-row{display:flex;gap:12px}.key-row input{flex:1;min-width:0}label{display:block;font-weight:600;margin-bottom:9px;font-size:13px}input,select,button{font:inherit;border-radius:8px}input,select{background:#fbfcfb;border:1px solid #d7e0d9;width:100%;padding:12px;color:inherit}input:focus,select:focus,button:focus-visible,summary:focus-visible{outline:3px solid #8dcab5;outline-offset:2px}button{cursor:pointer;border:1px solid #286850;background:#286850;color:#fff;padding:12px 22px;font-weight:600;white-space:nowrap}.secondary{background:#fff;color:#345e4f;border-color:#cedbd3}button:disabled{opacity:.45;cursor:not-allowed}.fields{display:grid;grid-template-columns:1fr 1fr;gap:20px;margin:24px 0 16px}.hint{font-size:12px;color:#79867f;line-height:1.8;margin:10px 0}.actions{display:flex;gap:12px;margin-top:22px}#notice{color:#ac4c32;font-size:13px;white-space:pre-wrap}#notice:empty{display:none}.progress-heading{display:flex;justify-content:space-between;gap:16px;font-size:14px}#percent{color:#3f7b60}progress{display:block;width:100%;height:12px;border:0;border-radius:10px;overflow:hidden;margin:18px 0 20px;background:#edf2ef}progress::-webkit-progress-bar{background:#edf2ef}progress::-webkit-progress-value{background:#3c8b68;transition:width .25s}progress::-moz-progress-bar{background:#3c8b68}.stats{display:flex;gap:28px;flex-wrap:wrap;color:#7a8880;font-size:13px}.stats b{color:#304e40;margin-left:8px;font-size:17px}#message{font-size:13px;color:#52685b;line-height:1.8}.path{overflow-wrap:anywhere}.console{height:260px;overflow:auto;background:#15251f;color:#c5dacf;border-radius:10px;padding:18px;font:12px/1.9 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;overflow-wrap:anywhere}.console time{color:#7eac94;margin-right:14px}.empty{color:#7e9889}summary{cursor:pointer;font-weight:600}summary span{font-size:12px;font-weight:400;color:#79867f;margin-left:12px}#debug-output{margin-bottom:0;height:300px}footer{text-align:center;color:#89958e;font-size:12px;margin:32px 0}@media(max-width:600px){main{padding:26px 14px}h1{font-size:28px}.card{padding:22px 18px}.key-row{flex-direction:column}.fields{grid-template-columns:1fr}.stats{gap:14px}.progress-heading{font-size:12px}}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build a portable v1 article package for the DWW11 userscript (standard library only)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import mimetypes
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def make_package(directory: str | Path, markdown: str | None = None) -> dict:
|
||||||
|
"""Return JSON-serializable data; no network access or publishing side effects."""
|
||||||
|
root = Path(directory).expanduser().resolve(strict=True)
|
||||||
|
if not root.is_dir():
|
||||||
|
raise ValueError("文章目录不存在")
|
||||||
|
choices = sorted(path for path in root.rglob("*") if path.is_file() and path.suffix.lower() == ".md")
|
||||||
|
if markdown:
|
||||||
|
selected = (root / markdown).resolve(strict=True)
|
||||||
|
elif len(choices) == 1:
|
||||||
|
selected = choices[0].resolve()
|
||||||
|
else:
|
||||||
|
raise ValueError("目录必须包含一个 Markdown 文件,或通过 --markdown 指定目录内相对路径")
|
||||||
|
if not selected.is_relative_to(root) or selected.suffix.lower() != ".md" or not selected.is_file():
|
||||||
|
raise ValueError("Markdown 必须位于所选文章目录内")
|
||||||
|
images = []
|
||||||
|
# Package local supported images. The importer uploads only those referenced by Markdown.
|
||||||
|
for path in sorted(root.rglob("*")):
|
||||||
|
if not path.is_file() or path.suffix.lower() not in {".jpg", ".jpeg", ".png", ".gif"}:
|
||||||
|
continue
|
||||||
|
if not path.resolve().is_relative_to(root):
|
||||||
|
raise ValueError(f"图片符号链接指向目录外:{path.name}")
|
||||||
|
if not 0 < path.stat().st_size < 20 * 1024 * 1024:
|
||||||
|
raise ValueError(f"图片为空或达到 20 MB:{path.name}")
|
||||||
|
images.append({"path": path.relative_to(root).as_posix(),
|
||||||
|
"mime": mimetypes.guess_type(path.name)[0] or "application/octet-stream",
|
||||||
|
"base64": base64.b64encode(path.read_bytes()).decode("ascii")})
|
||||||
|
return {"schemaVersion": 1, "markdownPath": selected.relative_to(root).as_posix(),
|
||||||
|
"markdown": selected.read_text(encoding="utf-8-sig"), "images": images}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="把文章目录打包为油猴脚本可导入的 JSON;不上传或发布")
|
||||||
|
parser.add_argument("directory", help="包含 Markdown 和图片的目录")
|
||||||
|
parser.add_argument("--markdown", help="多个 Markdown 时指定目录内的相对路径")
|
||||||
|
parser.add_argument("-o", "--output", help="输出文件路径;未指定时输出到标准输出")
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
content = json.dumps(make_package(args.directory, args.markdown), ensure_ascii=False, indent=2)
|
||||||
|
if args.output:
|
||||||
|
output = Path(args.output).expanduser()
|
||||||
|
# Never silently overwrite an article or an existing package.
|
||||||
|
with output.open("x", encoding="utf-8") as handle:
|
||||||
|
handle.write(content + "\n")
|
||||||
|
print(json.dumps({"ok": True, "package": str(output.resolve())}, ensure_ascii=False))
|
||||||
|
else:
|
||||||
|
print(content)
|
||||||
|
except (OSError, ValueError) as error:
|
||||||
|
parser.exit(1, f"打包失败:{error}\n")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
requests>=2.32,<3
|
||||||
|
beautifulsoup4>=4.12,<5
|
||||||
|
markdownify>=1.1,<2
|
||||||
Executable
+8
@@ -0,0 +1,8 @@
|
|||||||
|
#!/bin/zsh
|
||||||
|
cd "${0:A:h}"
|
||||||
|
if ! command -v uv >/dev/null 2>&1; then
|
||||||
|
print '未找到 uv,请安装 uv 后重试。'
|
||||||
|
read '?按回车关闭…'
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
uv run --no-project --python 3.12 --with-requirements requirements.txt python app.py
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
'use strict';
|
||||||
|
const $=id=>document.getElementById(id);
|
||||||
|
let token='',state=null,loadedKey='',refreshing=false;
|
||||||
|
const labels={queued:'待处理',preparing:'准备中',importing:'上传与导入',formatting:'排版中',awaiting_user_review:'等待人工审核',user_submitting:'已观察到人工提交',submit_unknown:'发布结果待核对',accepted_pending_moderation:'已受理 · 审核中',published:'已确认公开',needs_attention:'需要处理',cancelled:'已移出'};
|
||||||
|
function notice(message){$('notice').hidden=!message;$('notice').textContent=message;}
|
||||||
|
async function api(route,body={}) {
|
||||||
|
const response=await fetch('/api/'+route,{method:'POST',headers:{'Content-Type':'application/json','X-Session-Token':token},body:JSON.stringify(body)});
|
||||||
|
const data=await response.json();if(!response.ok)throw new Error(data.error||'请求失败');return data;
|
||||||
|
}
|
||||||
|
function action(id,fn){$(id).addEventListener('click',async()=>{const button=$(id);button.disabled=true;try{await fn();await refresh();}catch(error){notice(error.message);}finally{button.disabled=false;}});}
|
||||||
|
function text(tag,value,className){const e=document.createElement(tag);e.textContent=value;if(className)e.className=className;return e;}
|
||||||
|
function render(data){
|
||||||
|
state=data;const {jobs,run,download}=data;
|
||||||
|
$('total').textContent=jobs.length;$('review').textContent=jobs.filter(j=>j.state==='awaiting_user_review').length;
|
||||||
|
$('accepted').textContent=jobs.filter(j=>['published','accepted_pending_moderation'].includes(j.state)).length;
|
||||||
|
$('batch').textContent=run.batch?`${run.batch.count} / ${run.batch.target}`:'—';$('account').textContent=run.batch?run.batch.account.name:'尚未绑定账号';
|
||||||
|
$('run-state').textContent=run.running?'流程运行中':'流程已暂停';$('run-reason').textContent=run.reason;
|
||||||
|
$('run-light').classList.toggle('gray',!run.running);$('empty').hidden=jobs.length>0;
|
||||||
|
$('jobs').replaceChildren(...jobs.map(job=>{
|
||||||
|
const tr=document.createElement('tr'),title=document.createElement('td');title.append(text('div',job.title),text('small',`${job.image_count} 张配图 · ${job.source}`));if(job.original_title!==job.title)title.append(text('small','已缩短标题,原题:'+job.original_title));
|
||||||
|
const status=document.createElement('td');status.append(text('span',labels[job.state]||job.state,'badge'));if(job.error)status.append(text('small',job.error));
|
||||||
|
const account=text('td',job.account?`${job.account.name} / ${job.fid}`:'待领取');const result=document.createElement('td');
|
||||||
|
if(job.post_url){const link=text('a','查看帖子');link.href=job.post_url;link.target='_blank';link.rel='noopener';result.append(link);}
|
||||||
|
if(!['published','accepted_pending_moderation'].includes(job.state)){
|
||||||
|
const retry=text('button','人工核对 / 处理');retry.className='secondary';retry.onclick=async()=>{
|
||||||
|
if(state.run.running){notice('请先暂停流程,再处理任务');return;}
|
||||||
|
if(!confirm('只有确认网站未受理这篇文章,且已清理编辑器/草稿后才能继续。\n已受理、审核中或结果不明的文章请取消。\n你是否已经确认未受理?'))return;
|
||||||
|
const again=confirm('重新加入队列?\n确定:重新排队。取消:移出队列。');
|
||||||
|
try{await api('resolve',{id:job.id,confirmed_not_accepted:true,retry:again});await refresh();}catch(error){notice(error.message);}
|
||||||
|
};result.append(retry);
|
||||||
|
}
|
||||||
|
tr.append(title,status,account,result);return tr;
|
||||||
|
}));
|
||||||
|
const dlLabels={idle:'等待开始',running:'下载中',completed:'下载完成',partial:'未达到目标',cancelled:'已停止',error:'下载出错'};
|
||||||
|
$('download-state').textContent=dlLabels[download.status]||download.status;$('download-progress').textContent=`已保存 ${download.saved}/${download.target} 篇 · 跳过 ${download.skipped} · 失败 ${download.failed}`;
|
||||||
|
$('progress').value=download.target?Math.floor(download.saved/download.target*100):0;
|
||||||
|
$('download').disabled=download.status==='running';$('logs').textContent=download.logs.map(x=>`${x.time} ${x.message}`).join('\n')||'暂无日志';
|
||||||
|
$('browser-state').textContent=data.clients.length?`${data.clients.length} 个标签在线`:'尚未连接';
|
||||||
|
$('clients').textContent=data.clients.map(c=>`${c.account?.name||'未识别账号'} · 换号插件${c.extension?.ready?'已就绪':'未连接或需更新'}`).join('\n')||'没有在线执行标签';
|
||||||
|
}
|
||||||
|
async function refresh(){if(refreshing||!token)return;refreshing=true;try{render(await api('state'));}catch(error){notice(error.message);$('run-state').textContent='本地服务连接中断';}finally{refreshing=false;}}
|
||||||
|
action('pair',async()=>{const value=await api('pair');$('pair-code').hidden=false;$('pair-code').type='text';$('pair-code').value=value.token;$('pair-code').select();});
|
||||||
|
action('channels',async()=>{const value=await api('channels',{key:$('key').value.trim()});loadedKey=$('key').value.trim();$('channel').replaceChildren(...value.channels.map(c=>new Option(c,c)));});
|
||||||
|
$('key').oninput=()=>{loadedKey='';$('channel').replaceChildren(new Option('请重新获取频道',''));};
|
||||||
|
$('download-form').onsubmit=async event=>{event.preventDefault();try{if(!loadedKey||loadedKey!==$('key').value.trim())throw new Error('请先获取频道');await api('download',{key:loadedKey,channel:$('channel').value,target:Number($('target').value)});notice('下载已开始,完整保存的文章会自动入队');await refresh();}catch(error){notice(error.message);}};
|
||||||
|
action('download-stop',()=>api('download-stop'));
|
||||||
|
action('scan',async()=>{notice('正在检查文章目录…');const value=await api('scan',{root:$('root').value.trim()});$('scan-result').textContent=value.results.map(x=>`${x.directory}:${x.status}`).join('\n')||'目录中未找到文章';notice('目录检查完成');});
|
||||||
|
$('config-form').onsubmit=async event=>{event.preventDefault();try{await api('configure',{fid:Number($('fid').value),minimum:Number($('minimum').value),maximum:Number($('maximum').value),interval:Number($('interval').value),history_checked:$('history-checked').checked});notice('处理规则已保存');await refresh();}catch(error){notice(error.message);}};
|
||||||
|
for(const name of ['start','pause','stop'])action(name,async()=>{await api(name);notice(name==='stop'?'流程已终止;浏览器将在下一次连接检查时停止后续操作。已发出的请求可能仍已完成。':'');});
|
||||||
|
$('history-file').onchange=async()=>{try{const file=$('history-file').files[0];if(!file)return;if(file.size>2*1024*1024)throw new Error('记录文件过大');const value=JSON.parse(await file.text());await api('history',{records:value.records});notice('历史记录已导入;仅使用标记为 block_repost 的记录');await refresh();}catch(error){notice(error.message);}};
|
||||||
|
(async()=>{try{const response=await fetch('/api/session');const session=await response.json();if(!response.ok)throw new Error(session.error);token=session.token;$('port').value=location.port;await refresh();if(state?.run.config?.fid){for(const id of ['fid','minimum','maximum','interval'])$(id).value=state.run.config[id];$('history-checked').checked=state.run.config.history_checked;}}catch(error){notice(error.message);}setInterval(refresh,2500);})();
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>新闻工作台</title><link rel="stylesheet" href="/style.css"></head>
|
||||||
|
<body>
|
||||||
|
<aside><div class="brand"><span class="mark">闻</span><div>新闻工作台<small>本地内容工作流</small></div></div><nav><a href="#overview" class="selected">工作概览</a><a href="#source">下载与文章</a><a href="#settings">处理规则</a><a href="#queue">文章队列</a><a href="#connection">浏览器连接</a></nav><div class="aside-note"><span class="dot"></span> 数据保存在本机<p>文章自动准备与排版<br>发表由你亲手完成</p></div></aside>
|
||||||
|
<main>
|
||||||
|
<header id="overview"><div><div class="eyebrow">CONTENT WORKSPACE</div><h1>把内容准备好,把审核留给你。</h1><p>下载、导入、排版与账号切换,在一个工作台管理。</p></div><button id="stop" class="danger">终止流程</button></header>
|
||||||
|
<div id="notice" role="status" aria-live="polite" class="notice" hidden></div>
|
||||||
|
<section class="flow"><div><span>01</span>下载新闻</div><i>→</i><div><span>02</span>导入与排版</div><i>→</i><div class="highlight"><span>03</span>人工审核与发表</div><i>→</i><div><span>04</span>结果核对与换号</div></section>
|
||||||
|
<div class="pending-note"><b>当前接入进度</b> 下载、导入、排版与换号接口已整合;结果页自动核对等待页面 HTML。人工发表后会暂停,不自动计数或继续下一篇。</div>
|
||||||
|
<section class="metrics"><article><small>队列文章</small><strong id="total">0</strong><span>完整文件检查后入队</span></article><article><small>等待你审核</small><strong id="review">0</strong><span>请在网站检查并亲手发表</span></article><article><small>明确受理</small><strong id="accepted">0</strong><span>审核中与已公开分别记录</span></article><article><small>当前账号批次</small><strong id="batch">—</strong><span id="account">尚未绑定账号</span></article></section>
|
||||||
|
<div class="workspace-grid">
|
||||||
|
<section id="source" class="card"><div class="section-head"><div><span class="eyebrow">SOURCE</span><h2>下载与文章</h2></div><span id="download-state" class="badge">等待开始</span></div>
|
||||||
|
<form id="download-form"><label>探数 API Key<input id="key" type="password" autocomplete="off" placeholder="仅在本次运行中使用"></label><div class="row"><label class="grow">新闻频道<select id="channel" required><option value="">先获取频道</option></select></label><button id="channels" type="button" class="secondary">获取频道</button></div><div class="row"><label class="grow">下载篇数<input id="target" type="number" min="1" max="100000" value="5" required></label><button id="download" type="submit">开始下载</button><button id="download-stop" type="button" class="secondary">停止下载</button></div></form>
|
||||||
|
<div class="progress"><progress id="progress" max="100" value="0"></progress><small id="download-progress">等待下载</small></div>
|
||||||
|
<details><summary>导入已有文章目录</summary><p class="muted">填写包含文章子文件夹的完整路径。留空时扫描本工具的 downloads 目录。请在文件完整保存后扫描。</p><label>文章根目录<input id="root" placeholder="/完整路径/新闻目录"></label><button id="scan" class="secondary">检查并加入队列</button><pre id="scan-result"></pre></details>
|
||||||
|
</section>
|
||||||
|
<section id="settings" class="card"><div class="section-head"><div><span class="eyebrow">WORKFLOW</span><h2>处理规则</h2></div><span class="badge">人工发表</span></div>
|
||||||
|
<form id="config-form"><label>目标版块 ID(fid)<input id="fid" type="number" min="1" placeholder="从新帖页地址中的 fid 获取" required></label><div class="row"><label class="grow">每账号最少篇数<input id="minimum" type="number" min="1" max="1000" placeholder="最小值" required></label><label class="grow">每账号最多篇数<input id="maximum" type="number" min="1" max="1000" placeholder="最大值" required></label></div><label>两篇之间间隔(秒)<input id="interval" type="number" min="0" max="86400" placeholder="请输入间隔" required></label><label class="check"><input id="history-checked" type="checkbox" required><span>我已核对待处理文章与历史发布记录,避免重复发表。</span></label><button type="submit" class="secondary">保存处理规则</button></form><p class="muted">每个账号只抽取一次目标篇数;只有明确受理的文章才计数。历史演示去重记录会自动读取。</p>
|
||||||
|
</section></div>
|
||||||
|
<section class="runbar"><div><span id="run-light" class="dot gray"></span><b id="run-state">正在连接本地服务…</b><p id="run-reason">等待状态</p></div><div class="actions"><button id="start">开始 / 继续</button><button id="pause" class="secondary">暂停</button></div></section>
|
||||||
|
<section id="queue" class="card"><div class="section-head"><div><span class="eyebrow">ARTICLE QUEUE</span><h2>文章队列</h2></div><label class="file-button">导入历史去重记录<input id="history-file" type="file" accept=".json,application/json" hidden></label></div><div class="table-wrap"><table><thead><tr><th>文章</th><th>状态</th><th>账号 / 版块</th><th>结果与操作</th></tr></thead><tbody id="jobs"></tbody></table><div id="empty" class="empty"><span>+</span><b>从第一篇文章开始</b><p>下载新闻,或检查已有文章目录。准备好的文章会出现在这里。</p></div></div></section>
|
||||||
|
<section id="connection" class="card"><div class="section-head"><div><span class="eyebrow">CONNECTION</span><h2>连接你的浏览器</h2></div><span id="browser-state" class="badge">尚未连接</span></div><div class="connection-grid"><div><ol><li>在油猴中安装以下两个脚本,并停用原导入、排版脚本。</li><li>重新加载原目录中的换号插件,然后刷新网站。</li><li>在网站左下角展开连接设置,填入本地端口和连接码。</li></ol><div class="actions"><a class="button secondary" href="/browser/local-bridge.user.js">安装本地连接脚本</a><a class="button secondary" href="/browser/auto-news.user.js">安装导入排版脚本</a></div></div><div><label>本地端口<input id="port" readonly></label><button id="pair" class="secondary">显示本次连接码</button><input id="pair-code" type="password" readonly hidden aria-label="本次连接码"><small class="muted">服务重启后需要更新连接码。仅复制到你安装的本地连接脚本。</small><p id="clients" class="muted">没有在线执行标签</p></div></div></section>
|
||||||
|
<details class="card"><summary>下载日志</summary><pre id="logs">暂无日志</pre></details><footer>新闻工作台 · 本机运行 · v0.1 / 终止后不会自动恢复,已发出的请求可能仍已完成。</footer>
|
||||||
|
</main><script src="/app.js"></script>
|
||||||
|
</body></html>
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
:root{font-family:Inter,-apple-system,BlinkMacSystemFont,"PingFang SC","Microsoft YaHei",sans-serif;color:#203b38;background:#f3f6f4;font-synthesis:none;font-size:14px}*{box-sizing:border-box}body{margin:0}aside{position:fixed;inset:0 auto 0 0;width:220px;background:#113e37;color:#f4fbf7;padding:32px 22px;display:flex;flex-direction:column}.brand{display:flex;align-items:center;gap:12px;font-size:20px;font-weight:650;letter-spacing:.5px}.mark{display:grid;place-items:center;background:#d8efb9;color:#17483e;width:42px;height:42px;border-radius:13px}.brand small{display:block;font-size:11px;font-weight:400;color:#b3cdc5;margin-top:5px}nav{display:grid;gap:9px;margin-top:54px}nav a{color:#bad1c9;text-decoration:none;padding:12px 16px;border-radius:8px}nav a:hover,nav .selected{background:#28534a;color:#fff}.aside-note{margin-top:auto;color:#bad1c9;font-size:12px;line-height:1.9}.dot{display:inline-block;width:8px;height:8px;border-radius:50%;background:#a3db80;margin-right:7px}.dot.gray{background:#a1ada7}main{max-width:1660px;margin-left:220px;padding:42px 40px}header{display:flex;align-items:center;justify-content:space-between;gap:20px}.eyebrow{font-size:10px;letter-spacing:2px;color:#6b8980;font-weight:700}h1{font-size:28px;letter-spacing:-.6px;margin:12px 0}header p{color:#73857e;margin:0;font-size:13px}button,.button,.file-button{font:inherit;font-weight:550;border:1px solid #1c6554;border-radius:7px;background:#1c6554;color:white;padding:10px 15px;cursor:pointer;white-space:nowrap;text-decoration:none;display:inline-block}button:hover,.button:hover{filter:brightness(.94)}button:disabled{opacity:.45;cursor:wait}.secondary,.file-button{background:white;color:#36574d;border-color:#d4dfd7}.danger{background:#fff8f6;border-color:#e7c5bb;color:#a34d37}.flow{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-top:34px;background:white;border:1px solid #e2e9e2;padding:22px 25px;border-radius:12px;font-size:12px}.flow span{background:#edf3ef;color:#5f7d70;padding:5px 7px;border-radius:5px;font-size:10px;margin-right:8px}.flow i{color:#b1c0b6;font-style:normal}.flow .highlight{font-weight:650;color:#2e6947}.flow .highlight span{background:#e3efcc;color:#476635}.pending-note{background:#f8f2df;border:1px solid #eae0bd;color:#7b6938;padding:12px 16px;border-radius:8px;margin:16px 0;font-size:12px;line-height:1.8}.pending-note b{margin-right:10px}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin:22px 0}.metrics article{padding:20px 23px;background:white;border:1px solid #e0e8e0;border-radius:10px}.metrics small{font-size:12px;color:#6b7f73}.metrics strong{display:block;font-size:30px;font-weight:550;margin:12px 0 8px}.metrics span{font-size:11px;color:#8a988f}.workspace-grid{display:grid;grid-template-columns:1.1fr 1fr;gap:20px}.card{background:white;border:1px solid #e0e8e0;border-radius:12px;padding:24px;margin-bottom:20px;min-width:0}.section-head{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-bottom:20px}h2{font-size:17px;margin:6px 0 0;font-weight:600}.badge{font-size:11px;border-radius:20px;background:#f0f5ef;color:#6c826d;padding:5px 10px}.row{display:flex;gap:12px;align-items:flex-end}.grow{flex:1;min-width:0}label{display:block;font-size:12px;color:#61756b;margin-bottom:14px}input,select{display:block;width:100%;font:inherit;color:#304a3f;padding:10px 12px;margin-top:7px;border:1px solid #dbe4dd;border-radius:6px;background:#fcfdfb;outline:none}input:focus,select:focus{border-color:#659981;box-shadow:0 0 0 3px #6599811a}input::placeholder{color:#a4afa8}.row button{margin-bottom:14px}.check{display:flex;align-items:flex-start;gap:8px;line-height:1.7}.check input{width:15px;height:15px;margin:2px 0 0;accent-color:#276b54}.muted{font-size:12px;line-height:1.8;color:#8b988e}.progress{margin:12px 0 20px}progress{appearance:none;width:100%;height:5px;border:0;border-radius:4px;display:block;background:#edf2ec}progress::-webkit-progress-bar{background:#edf2ec}progress::-webkit-progress-value{background:#76a77f;border-radius:4px}.progress small{display:block;color:#82917f;margin-top:8px;font-size:11px}summary{cursor:pointer;color:#617d6d;font-size:12px}details label{margin-top:12px}.runbar{display:flex;align-items:center;justify-content:space-between;border-radius:10px;background:#e5eee5;padding:19px 23px;margin-bottom:22px;gap:16px}.runbar b{font-weight:550}.runbar p{margin:7px 0 0 15px;font-size:12px;color:#758b76}.actions{display:flex;gap:10px;flex-wrap:wrap}table{border-collapse:collapse;width:100%;text-align:left;font-size:12px}th{color:#8a998f;font-weight:500;background:#f7f9f5;padding:12px}td{padding:16px 12px;border-bottom:1px solid #eef1eb;vertical-align:top;line-height:1.8}td:first-child{max-width:350px}td small{display:block;color:#91a093;font-size:11px}td button{font-size:11px;padding:4px 8px}td a{color:#1c6554}.table-wrap{overflow-x:auto}.empty{padding:44px 16px;text-align:center;color:#81927f}.empty span{display:block;foLine truncated
|
||||||
@@ -0,0 +1,294 @@
|
|||||||
|
"""Persistent serial queue. Browser submissions are always initiated by the user."""
|
||||||
|
import json
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
import secrets
|
||||||
|
import sqlite3
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from articles import inspect_article
|
||||||
|
|
||||||
|
ACCEPTED = {'published', 'accepted_pending_moderation'}
|
||||||
|
SUBMITTED = {'user_submitting', 'submit_unknown'}
|
||||||
|
|
||||||
|
|
||||||
|
class Store:
|
||||||
|
def __init__(self, path, ledger=None):
|
||||||
|
self.path = str(path)
|
||||||
|
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
self.lock = threading.RLock()
|
||||||
|
with self.db() as db:
|
||||||
|
db.executescript('''
|
||||||
|
CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, article_key TEXT UNIQUE, value TEXT NOT NULL);
|
||||||
|
CREATE TABLE IF NOT EXISTS ledger(key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||||
|
CREATE TABLE IF NOT EXISTS settings(id INTEGER PRIMARY KEY CHECK(id=1), value TEXT NOT NULL);
|
||||||
|
CREATE TABLE IF NOT EXISTS posts(id TEXT PRIMARY KEY, job TEXT NOT NULL);
|
||||||
|
''')
|
||||||
|
row = db.execute('SELECT value FROM settings WHERE id=1').fetchone()
|
||||||
|
self.run = json.loads(row[0]) if row else dict(running=False, epoch=0, config={}, batch=None, active=None, owner=None, switch=None, reason='请配置任务后开始')
|
||||||
|
self.run.update(running=False, epoch=self.run['epoch'] + 1, reason='服务已启动,请明确点击开始/继续;未决任务不会重发')
|
||||||
|
self._persist(db)
|
||||||
|
self.clients = {}
|
||||||
|
if ledger and Path(ledger).is_file():
|
||||||
|
self.import_history(json.loads(Path(ledger).read_text()).get('records', []))
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def db(self):
|
||||||
|
connection = sqlite3.connect(self.path, timeout=20)
|
||||||
|
try:
|
||||||
|
with connection:
|
||||||
|
yield connection
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
def _persist(self, db):
|
||||||
|
db.execute('INSERT OR REPLACE INTO settings VALUES(1,?)', (json.dumps(self.run, ensure_ascii=False),))
|
||||||
|
|
||||||
|
def _job(self, db, id):
|
||||||
|
row = db.execute('SELECT value FROM jobs WHERE id=?', (id,)).fetchone()
|
||||||
|
if not row:
|
||||||
|
raise ValueError('任务不存在')
|
||||||
|
return json.loads(row[0])
|
||||||
|
|
||||||
|
def _save(self, db, job):
|
||||||
|
job['updated_at'] = time.time()
|
||||||
|
db.execute('INSERT OR REPLACE INTO jobs VALUES(?,?,?)', (job['id'], job['article_key'], json.dumps(job, ensure_ascii=False)))
|
||||||
|
|
||||||
|
def import_history(self, records):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
for record in records:
|
||||||
|
digest = record.get('markdown_sha256')
|
||||||
|
if digest and record.get('block_repost'):
|
||||||
|
db.execute('INSERT OR IGNORE INTO ledger VALUES(?,?)', ('md:' + digest, json.dumps(record, ensure_ascii=False)))
|
||||||
|
# Add content identity where the original historical file is still present.
|
||||||
|
try:
|
||||||
|
article = inspect_article(Path(record['markdown_path']).parent)
|
||||||
|
db.execute('INSERT OR IGNORE INTO ledger VALUES(?,?)', ('article:' + article['article_key'], json.dumps(record, ensure_ascii=False)))
|
||||||
|
except (OSError, ValueError, KeyError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def enqueue(self, directory):
|
||||||
|
article = inspect_article(directory)
|
||||||
|
article.pop('package')
|
||||||
|
keys = ['article:' + article['article_key'], 'md:' + article['markdown_sha256']] + ['source:' + str(k) for k in article['source_keys']]
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
if any(db.execute('SELECT 1 FROM ledger WHERE key=?', (k,)).fetchone() for k in keys):
|
||||||
|
return '已受理,跳过'
|
||||||
|
if db.execute('SELECT 1 FROM jobs WHERE article_key=?', (article['article_key'],)).fetchone():
|
||||||
|
return '已在队列,跳过'
|
||||||
|
# Source URL identities prevent different renderings of one downloaded article re-entering.
|
||||||
|
for row in db.execute('SELECT value FROM jobs'):
|
||||||
|
if set(article['source_keys']) & set(json.loads(row[0]).get('source_keys', [])):
|
||||||
|
return '相同来源已在队列,跳过'
|
||||||
|
article.update(id=secrets.token_hex(12), state='queued', created_at=time.time(), error='', post_url='')
|
||||||
|
self._save(db, article)
|
||||||
|
return '已入队'
|
||||||
|
|
||||||
|
def state(self):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
jobs = [json.loads(r[0]) for r in db.execute('SELECT value FROM jobs ORDER BY rowid')]
|
||||||
|
return dict(run=json.loads(json.dumps(self.run)), jobs=jobs,
|
||||||
|
clients=[v for v in self.clients.values() if time.time() - v['seen'] < 20],
|
||||||
|
history_count=db.execute('SELECT count(*) FROM ledger').fetchone()[0])
|
||||||
|
|
||||||
|
def configure(self, config):
|
||||||
|
fid, minimum, maximum = config.get('fid'), config.get('minimum'), config.get('maximum')
|
||||||
|
interval = config.get('interval')
|
||||||
|
if any(type(x) is not int for x in (fid, minimum, maximum, interval)) or not (fid > 0 and 1 <= minimum <= maximum <= 1000 and 0 <= interval <= 86400):
|
||||||
|
raise ValueError('请填写有效版块 ID、篇数范围(1~1000)和间隔秒数')
|
||||||
|
if config.get('history_checked') is not True:
|
||||||
|
raise ValueError('请先核对待处理文章与历史发布记录,并勾选确认')
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
if self.run['active'] or self.run['switch'] or self.run['running']:
|
||||||
|
raise ValueError('当前流程未结束,不能更换配置')
|
||||||
|
self.run.update(config={k: config[k] for k in ('fid', 'minimum', 'maximum', 'interval', 'history_checked')}, batch=None)
|
||||||
|
self._persist(db)
|
||||||
|
|
||||||
|
def control(self, action):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
if action == 'start':
|
||||||
|
if not self.run['config']:
|
||||||
|
raise ValueError('请先保存配置')
|
||||||
|
self.run.update(running=True, reason='等待浏览器连接')
|
||||||
|
else:
|
||||||
|
self.run.update(running=False, reason='已终止' if action == 'stop' else '已暂停')
|
||||||
|
self.run['epoch'] += 1
|
||||||
|
self._persist(db)
|
||||||
|
return dict(self.run)
|
||||||
|
|
||||||
|
def _batch(self, account):
|
||||||
|
cfg = self.run['config']
|
||||||
|
return dict(account=account, target=cfg['minimum'] + secrets.randbelow(cfg['maximum'] - cfg['minimum'] + 1), count=0)
|
||||||
|
|
||||||
|
def _check_owner(self, owner):
|
||||||
|
if self.run['owner'] != owner:
|
||||||
|
raise ValueError('任务由另一个浏览器标签持有')
|
||||||
|
|
||||||
|
def heartbeat(self, data):
|
||||||
|
owner = str(data.get('owner', ''))[:100]
|
||||||
|
if not owner:
|
||||||
|
raise ValueError('缺少执行标签标识')
|
||||||
|
with self.lock:
|
||||||
|
self.clients[owner] = dict(owner=owner, account=data.get('account'), url=str(data.get('url', ''))[:1000],
|
||||||
|
extension=data.get('extension'), seen=time.time())
|
||||||
|
return self.state()['run']
|
||||||
|
|
||||||
|
def next(self, data):
|
||||||
|
owner, account = data.get('owner'), data.get('account')
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
if not self.run['running']:
|
||||||
|
return dict(action='idle', run=self.run)
|
||||||
|
if not account or not str(account.get('uid', '')).isdigit() or not account.get('name'):
|
||||||
|
return dict(action='login', run=self.run)
|
||||||
|
if self.run['owner'] not in (None, owner):
|
||||||
|
return dict(action='other_tab', run=self.run)
|
||||||
|
self.run['owner'] = owner
|
||||||
|
if self.run['switch']:
|
||||||
|
switch = self.run['switch']
|
||||||
|
if account['uid'] != switch['from'] and data.get('extension', {}).get('idle') is True:
|
||||||
|
self.run.update(batch=self._batch(account), switch=None)
|
||||||
|
self._persist(db)
|
||||||
|
else:
|
||||||
|
return dict(action='switch' if switch['state'] == 'pending' else 'switch_wait', run=self.run)
|
||||||
|
if self.run['batch'] is None:
|
||||||
|
self.run['batch'] = self._batch(account)
|
||||||
|
if self.run['batch']['account']['uid'] != account['uid']:
|
||||||
|
self.run.update(running=False, reason='检测到账号变化,请先核对当前任务')
|
||||||
|
self._persist(db)
|
||||||
|
return dict(action='idle', run=self.run)
|
||||||
|
if self.run['active']:
|
||||||
|
job = self._job(db, self.run['active'])
|
||||||
|
return dict(action='resume', job=job, run=self.run)
|
||||||
|
rows = [json.loads(r[0]) for r in db.execute('SELECT value FROM jobs ORDER BY rowid')]
|
||||||
|
job = next((j for j in rows if j['state'] == 'queued'), None)
|
||||||
|
if not job:
|
||||||
|
self.run.update(running=False, reason='当前队列已处理完;新文章入队后请点击继续')
|
||||||
|
self._persist(db)
|
||||||
|
return dict(action='idle', run=self.run)
|
||||||
|
if time.time() < self.run.get('not_before', 0):
|
||||||
|
return dict(action='interval', run=self.run)
|
||||||
|
if self.run['batch']['count'] >= self.run['batch']['target']:
|
||||||
|
self.run['switch'] = dict(state='pending', **{'from': account['uid']}, created_at=time.time())
|
||||||
|
self._persist(db)
|
||||||
|
return dict(action='switch', run=self.run)
|
||||||
|
job.update(state='preparing', account=account, fid=self.run['config']['fid'], error='')
|
||||||
|
self.run['active'] = job['id']
|
||||||
|
self._save(db, job)
|
||||||
|
self._persist(db)
|
||||||
|
return dict(action='prepare', job=job, run=self.run)
|
||||||
|
|
||||||
|
def current(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
self._check_owner(data['owner'])
|
||||||
|
return {'job': self._job(db, self.run['active']) if self.run['active'] else None, 'run': self.run}
|
||||||
|
|
||||||
|
def switch_begin(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
self._check_owner(data['owner'])
|
||||||
|
if not self.run['running'] or not self.run['switch'] or self.run['switch']['state'] != 'pending':
|
||||||
|
raise ValueError('换号状态已改变,禁止重复点击')
|
||||||
|
self.run['switch']['state'] = 'clicked'
|
||||||
|
self._persist(db)
|
||||||
|
return {'ok': True}
|
||||||
|
|
||||||
|
def package(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
self._check_owner(data['owner'])
|
||||||
|
if not self.run['running'] or self.run['active'] != data['id']:
|
||||||
|
raise ValueError('流程已停止或任务不匹配')
|
||||||
|
job = self._job(db, data['id'])
|
||||||
|
if job['state'] != 'preparing':
|
||||||
|
raise ValueError('当前任务不允许重新导入,请先核对已上传内容')
|
||||||
|
article = inspect_article(job['directory'])
|
||||||
|
if article['markdown_sha256'] != job['markdown_sha256'] or article['article_key'] != job['article_key']:
|
||||||
|
raise ValueError('入队后原文章已改变,请移出并重新扫描')
|
||||||
|
return article['package']
|
||||||
|
|
||||||
|
def event(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
self._check_owner(data['owner'])
|
||||||
|
job = self._job(db, data['id'])
|
||||||
|
if self.run['active'] != job['id']:
|
||||||
|
raise ValueError('不是当前任务')
|
||||||
|
target = data['state']
|
||||||
|
transitions = {'preparing': {'importing', 'needs_attention'}, 'importing': {'formatting', 'needs_attention'},
|
||||||
|
'formatting': {'awaiting_user_review', 'needs_attention'},
|
||||||
|
'awaiting_user_review': {'user_submitting', 'needs_attention'},
|
||||||
|
'user_submitting': {'submit_unknown', 'needs_attention'}, 'submit_unknown': {'needs_attention'}}
|
||||||
|
if target not in transitions.get(job['state'], set()):
|
||||||
|
raise ValueError('任务状态不允许此操作:' + job['state'] + ' → ' + target)
|
||||||
|
if target in ('importing', 'formatting', 'awaiting_user_review') and (not self.run['running'] or data.get('epoch') != self.run['epoch']):
|
||||||
|
raise ValueError('流程已暂停/终止,过期回调不能继续')
|
||||||
|
job.update(state=target, error=str(data.get('error', ''))[:1000])
|
||||||
|
if target == 'user_submitting':
|
||||||
|
job.update(submitted_at=time.time(), snapshot=data.get('snapshot', {}), attempt=secrets.token_hex(8))
|
||||||
|
if target == 'awaiting_user_review':
|
||||||
|
job['prepared'] = data.get('snapshot', {})
|
||||||
|
if target == 'needs_attention':
|
||||||
|
self.run.update(running=False, reason=job['error'] or '当前任务需要处理')
|
||||||
|
self._persist(db)
|
||||||
|
self._save(db, job)
|
||||||
|
return {'ok': True}
|
||||||
|
|
||||||
|
def accept(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
self._check_owner(data['owner'])
|
||||||
|
job = self._job(db, data['id'])
|
||||||
|
if job['state'] in ACCEPTED:
|
||||||
|
return {'ok': True, 'duplicate': True}
|
||||||
|
if self.run['active'] != job['id'] or job['state'] not in SUBMITTED:
|
||||||
|
raise ValueError('只有已观察到人工提交的任务才能登记结果')
|
||||||
|
url = urlsplit(str(data.get('post_url', '')))
|
||||||
|
import re
|
||||||
|
match = re.fullmatch(r'/thread/(\d+)\.html', url.path)
|
||||||
|
if url.scheme != 'https' or url.netloc not in ('www.dww11.com', 'dww11.com') or not match:
|
||||||
|
raise ValueError('未识别到帖子详情页')
|
||||||
|
if data.get('author_uid') != job['account']['uid'] or data.get('fid') != job['fid']:
|
||||||
|
raise ValueError('结果页作者或版块与任务不符,请人工核对')
|
||||||
|
if data.get('title') != job.get('snapshot', {}).get('title'):
|
||||||
|
raise ValueError('结果页标题与人工提交快照不一致')
|
||||||
|
if data.get('state') not in ACCEPTED:
|
||||||
|
raise ValueError('需要明确的公开或审核中状态')
|
||||||
|
if not data.get('evidence'):
|
||||||
|
raise ValueError('缺少页面结果证据')
|
||||||
|
post_id = match[1]
|
||||||
|
existing = db.execute('SELECT job FROM posts WHERE id=?', (post_id,)).fetchone()
|
||||||
|
if existing and existing[0] != job['id']:
|
||||||
|
raise ValueError('帖子已绑定其他任务,不能重复计数')
|
||||||
|
db.execute('INSERT OR IGNORE INTO posts VALUES(?,?)', (post_id, job['id']))
|
||||||
|
job.update(state=data['state'], post_id=post_id, post_url=url.geturl(), result=data['evidence'])
|
||||||
|
keys = ['article:' + job['article_key'], 'md:' + job['markdown_sha256']] + ['source:' + str(k) for k in job['source_keys']]
|
||||||
|
for key in keys:
|
||||||
|
db.execute('INSERT OR REPLACE INTO ledger VALUES(?,?)', (key, json.dumps(job, ensure_ascii=False)))
|
||||||
|
self.run['batch']['count'] += 1
|
||||||
|
self.run.update(active=None, not_before=time.time() + self.run['config']['interval'])
|
||||||
|
self._save(db, job)
|
||||||
|
self._persist(db)
|
||||||
|
return {'ok': True}
|
||||||
|
|
||||||
|
def resolve(self, data):
|
||||||
|
"""Explicit local UI resolution, never an automatic retry of an unknown submission."""
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
job = self._job(db, data['id'])
|
||||||
|
if job['state'] in ACCEPTED:
|
||||||
|
raise ValueError('已受理文章不能重置')
|
||||||
|
if self.run['running']:
|
||||||
|
raise ValueError('请先暂停流程')
|
||||||
|
if data.get('confirmed_not_accepted') is not True:
|
||||||
|
raise ValueError('必须先人工核对网站未受理,并清理编辑器/草稿')
|
||||||
|
job.update(state='queued' if data.get('retry') else 'cancelled', error='', snapshot={})
|
||||||
|
if self.run['active'] == job['id']:
|
||||||
|
self.run.update(active=None, owner=None)
|
||||||
|
self._save(db, job)
|
||||||
|
self._persist(db)
|
||||||
|
return {'ok': True}
|
||||||
|
|
||||||
|
def takeover(self, data):
|
||||||
|
with self.lock, self.db() as db:
|
||||||
|
if self.run['running']:
|
||||||
|
raise ValueError('请先在控制台暂停流程,再接管')
|
||||||
|
self.run['owner'] = data['owner']
|
||||||
|
self._persist(db)
|
||||||
|
return {'ok': True}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
# 测试与归档
|
||||||
|
|
||||||
|
- `history/`:从 auto_news_upload 根目录移入的历史截图和演示任务 JSON,保留原内容。不是自动测试,不能用作本次测试结论。
|
||||||
|
- `fixtures/article/`:自造 Markdown 和三张 1×1 PNG,其中两张内容相同,用于验证上传去重及插图顺序。
|
||||||
|
- `test_queue.py`:持久队列、去重、重启、终止、未知提交和按篇数换号。
|
||||||
|
- `test_controller.py`:本地接口权限、模拟下载到队列、静态页面路由。
|
||||||
|
- `test_download_core.py`:复制的下载器原有离线测试,针对合并项目中的下载器副本运行。
|
||||||
|
- `importer.test.cjs`:导入器回归与自动模式的停止/冲突保护。
|
||||||
|
- `browser.test.cjs`:本地页面、移动布局与模拟编辑器;从不请求真实论坛。
|
||||||
|
- `artifacts/`:本轮页面截图,属于生成产物。
|
||||||
|
|
||||||
|
原始受理去重记录继续保存在 `../发布去重记录.json`,服务启动时自动导入;没有作为测试残留移走或删除。
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
'use strict';
|
||||||
|
const {test}=require('node:test');const assert=require('node:assert/strict');const fs=require('node:fs');const path=require('node:path');const vm=require('node:vm');
|
||||||
|
test('终止后较早的 heartbeat 响应不能恢复流程',async()=>{
|
||||||
|
const handlers=new Map(),requests=[],responses=[],values={pairToken:'test-token',pairPort:8770};
|
||||||
|
class Event{constructor(type,options={}){this.type=type;this.detail=options.detail;}}
|
||||||
|
const doc={readyState:'loading',addEventListener(type,fn){handlers.set(type,fn);},dispatchEvent(event){if(event.type==='dww11:local-response')responses.push(JSON.parse(event.detail));else handlers.get(event.type)?.(event);}};
|
||||||
|
vm.runInNewContext(fs.readFileSync(path.join(__dirname,'../browser/local-bridge.user.js'),'utf8'),{document:doc,CustomEvent:Event,GM_getValue:(k,d)=>values[k]??d,GM_setValue:(k,v)=>values[k]=v,GM_addValueChangeListener:()=>{},GM_xmlhttpRequest:r=>requests.push(r)});
|
||||||
|
const call=(id,route)=>handlers.get('dww11:local-request')(new Event('',{detail:JSON.stringify({id,route})}));
|
||||||
|
const heartbeat=call('old','heartbeat');const stop=call('stop','stop');
|
||||||
|
requests[0].onload({status:200,responseText:JSON.stringify({running:true,epoch:1})});await heartbeat;
|
||||||
|
assert.equal(responses.find(r=>r.id==='old').value.running,false);assert.equal(values.halted,true);
|
||||||
|
requests[1].onload({status:200,responseText:JSON.stringify({running:false,epoch:2})});await stop;
|
||||||
|
});
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
'use strict';
|
||||||
|
const {chromium}=require('playwright');
|
||||||
|
const fs=require('node:fs');
|
||||||
|
const path=require('node:path');
|
||||||
|
const assert=require('node:assert/strict');
|
||||||
|
const ROOT=path.resolve(__dirname,'..');
|
||||||
|
const output=path.join(__dirname,'artifacts');fs.mkdirSync(output,{recursive:true});
|
||||||
|
(async()=>{
|
||||||
|
const browser=await chromium.launch({headless:true,executablePath:process.env.CHROME_PATH||'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'});
|
||||||
|
try{
|
||||||
|
const page=await browser.newPage({viewport:{width:1440,height:1100}}),errors=[];
|
||||||
|
page.on('pageerror',error=>errors.push(error.message));
|
||||||
|
await page.route('**/*',route=>new URL(route.request().url()).hostname==='127.0.0.1'?route.continue():route.abort());
|
||||||
|
await page.goto('http://127.0.0.1:8771/');
|
||||||
|
await page.getByText('原创导入测试稿',{exact:true}).waitFor();
|
||||||
|
await page.screenshot({path:path.join(output,'desktop.png'),fullPage:true});
|
||||||
|
await page.locator('#fid').fill('125');await page.locator('#minimum').fill('1');await page.locator('#maximum').fill('2');await page.locator('#interval').fill('10');await page.locator('#history-checked').check();
|
||||||
|
await page.getByRole('button',{name:'保存处理规则'}).click();await page.getByText('处理规则已保存',{exact:true}).waitFor();
|
||||||
|
await page.locator('#start').click();await page.getByText('流程运行中',{exact:true}).waitFor();
|
||||||
|
await page.locator('#stop').click();await page.getByText('流程已暂停',{exact:true}).waitFor();
|
||||||
|
await page.setViewportSize({width:390,height:844});await page.screenshot({path:path.join(output,'mobile.png'),fullPage:true});
|
||||||
|
assert.equal(await page.evaluate(()=>document.documentElement.scrollWidth>innerWidth),false,'mobile overflow');
|
||||||
|
assert.deepEqual(errors,[]);console.log('PASS local UI: configuration, start/stop, desktop/mobile layout');
|
||||||
|
await page.close();
|
||||||
|
const mock=await browser.newPage();
|
||||||
|
await mock.route('**/*',route=>route.fulfill({contentType:'text/html; charset=utf-8',body:'<!doctype html><a id="mumucms_username" href="/home.php?mod=space&uid=123">测试账号</a><input id="subject"><iframe id="e_iframe"></iframe><button id="postsubmit">发表帖子</button>'}));
|
||||||
|
await mock.goto('https://www.dww11.com/forum.php?mod=post&action=newthread&fid=125');
|
||||||
|
await mock.evaluate(()=>{
|
||||||
|
window.wysiwyg=1;window.events=[];window.clicks=0;window.formats=0;
|
||||||
|
window.job={id:'fixture',state:'preparing',title:'测试标题',source:'测试来源',fid:125,account:{uid:'123',name:'测试账号'}};
|
||||||
|
window.flow={running:true,epoch:1,active:'fixture',config:{fid:125}};
|
||||||
|
document.querySelector('#postsubmit').addEventListener('click',()=>window.clicks++);
|
||||||
|
window.DWW11Importer={stagePackage:async()=>({warnings:[]}),importPrepared:async()=>{
|
||||||
|
document.querySelector('#subject').value='测试标题';
|
||||||
|
document.querySelector('#e_iframe').contentDocument.body.innerHTML='<p>正文第一段</p><img id="attachimg_1" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"><p>正文第二段</p><p>来源:测试来源,如有侵权,请联系删除。</p>';
|
||||||
|
return {images:{'a.gif':'1'}};
|
||||||
|
}};
|
||||||
|
document.addEventListener('dww11:local-request',event=>{
|
||||||
|
const {id,route,body}=JSON.parse(event.detail);let value={};
|
||||||
|
window.events.push({route,body});
|
||||||
|
if(route==='heartbeat'){window.flow.owner=body.owner;value=window.flow;}
|
||||||
|
if(route==='next')value={action:window.job.state==='preparing'?'prepare':'resume',job:window.job,run:window.flow};
|
||||||
|
if(route==='current')value={job:window.job,run:window.flow};
|
||||||
|
if(route==='event'){window.job.state=body.state;value={ok:true};}
|
||||||
|
if(route==='stop'){window.flow.running=false;value=window.flow;}
|
||||||
|
document.dispatchEvent(new CustomEvent('dww11:local-response',{detail:JSON.stringify({id,value})}));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await mock.addScriptTag({path:path.join(ROOT,'browser/formatter.js')});
|
||||||
|
await mock.addScriptTag({path:path.join(ROOT,'browser/runner.js')});
|
||||||
|
await mock.waitForFunction(()=>window.job.state==='awaiting_user_review');
|
||||||
|
await mock.waitForTimeout(2400);
|
||||||
|
assert.equal(await mock.evaluate(()=>window.clicks),0,'runner must never publish');
|
||||||
|
assert.equal(await mock.evaluate(()=>document.querySelector('#e_iframe').contentDocument.body.querySelectorAll('[data-dww11-wx-source]').length),1);
|
||||||
|
const before=await mock.evaluate(()=>document.querySelector('#e_iframe').contentDocument.body.innerHTML);
|
||||||
|
await mock.evaluate(()=>window.DWW11Formatter.format('测试来源'));
|
||||||
|
const after=await mock.evaluate(()=>document.querySelector('#e_iframe').contentDocument.body.innerHTML);
|
||||||
|
assert.equal(after,before,'repeated formatting should be idempotent');
|
||||||
|
await mock.locator('#postsubmit').click();
|
||||||
|
await mock.waitForFunction(()=>window.job.state==='user_submitting');
|
||||||
|
await mock.evaluate(()=>history.pushState({},'', '/thread/999.html'));
|
||||||
|
await mock.waitForFunction(()=>window.job.state==='submit_unknown');
|
||||||
|
assert.equal(await mock.evaluate(()=>window.flow.running),false);
|
||||||
|
assert.equal(await mock.evaluate(()=>window.events.some(e=>e.route==='accept')),false,'unknown page must not count accepted');
|
||||||
|
assert.equal(await mock.evaluate(()=>window.clicks),1);
|
||||||
|
console.log('PASS browser fixture: import/format, source uniqueness, repeated formatting, manual-only publish, unknown result stops');
|
||||||
|
await mock.close();
|
||||||
|
const ext=await browser.newPage();
|
||||||
|
await ext.route('**/*',route=>route.fulfill({contentType:'text/html; charset=utf-8',body:'<!doctype html><div class="topwp"><a href="#">立即登录</a></div>'}));
|
||||||
|
await ext.goto('https://www.dww11.com/');
|
||||||
|
await ext.evaluate(()=>{
|
||||||
|
window.routes=[];window.chrome.runtime={sendMessage:async message=>{
|
||||||
|
window.routes.push(message.route);
|
||||||
|
if(message.route==='/hello')return {ok:true,data:{total:1,active:1,round:1,remaining:1,flow:null}};
|
||||||
|
if(message.route==='/start')return new Promise(resolve=>window.finishStart=()=>resolve({ok:true,data:{flow:{id:'test-run',phase:'logout',paused:false}}}));
|
||||||
|
if(message.route==='/suspend')return {ok:true,data:{}};
|
||||||
|
throw new Error('Unexpected extension operation: '+message.route);
|
||||||
|
}};
|
||||||
|
});
|
||||||
|
await ext.addScriptTag({path:path.join(ROOT,'../dww11-account-switch-browser-extension/content.js')});
|
||||||
|
await ext.waitForFunction(()=>JSON.parse(document.querySelector('#dww11-extension-panel').dataset.autoNewsState).idle);
|
||||||
|
await ext.locator('#dww11-extension-next').click();
|
||||||
|
await ext.waitForFunction(()=>!!window.finishStart);
|
||||||
|
await ext.evaluate(()=>{document.dispatchEvent(new CustomEvent('dww11:auto-stop'));window.finishStart();});
|
||||||
|
await ext.waitForFunction(()=>window.routes.includes('/suspend'));
|
||||||
|
assert.equal(await ext.evaluate(()=>JSON.parse(document.querySelector('#dww11-extension-panel').dataset.autoNewsState).paused),true);
|
||||||
|
assert.deepEqual(await ext.evaluate(()=>window.routes),['/hello','/start','/suspend']);
|
||||||
|
console.log('PASS extension bridge: stop during pending start prevents login and persists pause');
|
||||||
|
await ext.close();
|
||||||
|
}finally{await browser.close();}
|
||||||
|
})().catch(error=>{console.error(error);process.exitCode=1;});
|
||||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 B |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 B |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 B |
Vendored
+15
@@ -0,0 +1,15 @@
|
|||||||
|
# 原创导入测试稿
|
||||||
|
|
||||||
|
这是项目自有的离线测试稿,用于验证正文、段落和图片顺序。
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
第二段保留原文标点,以及普通文字。
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
最后一段说明图片上传不等于发表文章。
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
来源:项目测试素材,如有侵权,请联系删除。
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
// Offline verification only. No browser account, uploads or publication requests.
|
||||||
|
'use strict';
|
||||||
|
const { test } = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const path = require('node:path');
|
||||||
|
const vm = require('node:vm');
|
||||||
|
const { webcrypto, createHash } = require('node:crypto');
|
||||||
|
const { execFileSync } = require('node:child_process');
|
||||||
|
const core = require('../browser/importer.js');
|
||||||
|
const directory = path.join(__dirname, 'fixtures/article');
|
||||||
|
const mdName = '原创导入测试稿.md';
|
||||||
|
function articleFiles() {
|
||||||
|
return fs.readdirSync(directory).map(name => {
|
||||||
|
const blob = new Blob([fs.readFileSync(path.join(directory, name))], { type: name.endsWith('.png') ? 'image/png' : 'text/markdown' });
|
||||||
|
blob.name = name; blob.webkitRelativePath = '文章/' + name; return blob;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
test('测试文章:标题、3 张配图、插入顺序、来源与段落完整', async () => {
|
||||||
|
const article = await core.prepareFiles(articleFiles());
|
||||||
|
assert.equal(article.title, mdName.slice(0, -3));
|
||||||
|
assert.deepEqual([...article.images.keys()], ['文章/image001.png', '文章/image002.png', '文章/image003.png']);
|
||||||
|
const output = article.render(new Map([...article.images.keys()].map((p, i) => [p, String(101 + i)])));
|
||||||
|
assert.deepEqual([...output.matchAll(/\[attachimg\](\d+)\[\/attachimg\]/g)].map(m => m[1]), ['101', '102', '103']);
|
||||||
|
assert.equal(output.includes('# 原创导入测试稿'), false);
|
||||||
|
const md = fs.readFileSync(path.join(directory, mdName), 'utf8');
|
||||||
|
const originalBody = md.split('\n').slice(1).join('\n').trim().replace(/!\[\]\(image00(\d)\.png\)/g, (_, n) => `[attachimg]${100 + Number(n)}[/attachimg]`);
|
||||||
|
assert.equal(output, originalBody);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('缺图、远程图片、超出目录、超大图片与超长标题在上传前被拒绝', () => {
|
||||||
|
assert.throws(() => core.compile('# 标题\n\n', 'a.md', new Map()), /找不到配图/);
|
||||||
|
assert.throws(() => core.compile('# 标题\n\n', 'a.md', new Map()), /目录内/);
|
||||||
|
assert.throws(() => core.imagePath('a.md', '../secret.jpg'), /超出/);
|
||||||
|
assert.throws(() => core.imagePath('a.md', '%2fetc/passwd'), /相对路径/);
|
||||||
|
assert.throws(() => core.compile('# ' + '长'.repeat(81) + '\n正文', 'a.md', new Map()), /80/);
|
||||||
|
assert.throws(() => core.compile('# 标题\n\n', 'a.md', new Map([['big.png', {size:20*1024*1024}]])), /20 MB/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('子目录、Unicode 规范化、空格编码、括号、重复图片和格式转换', () => {
|
||||||
|
const fileMap = new Map([['目录/图 (1).jpg', new Blob(['image'])]]);
|
||||||
|
const article = core.compile('# 标题\n\n**粗体**和*斜体*\n\n.jpg)\n.jpg> "说明")\n\n[链接](https://example.com/a)\n\n- 一\n- 二\n\n> 引用', '目录/a.md', fileMap);
|
||||||
|
assert.equal(article.images.size, 1);
|
||||||
|
assert.match(article.render(new Map([['目录/图 (1).jpg', '100']])), /\[b\]粗体\[\/b\]/);
|
||||||
|
assert.equal(core.normalizePath('cafe\u0301/a.jpg'), 'café/a.jpg');
|
||||||
|
assert.equal(core.readLink("", 0).target, "child's.jpg");
|
||||||
|
assert.throws(() => core.compile('# 标题\n![][a]\n[a]: x.jpg', 'a.md', fileMap), /引用式/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('多个 Markdown 必须选择;无 Markdown 不接受', async () => {
|
||||||
|
const files = articleFiles();
|
||||||
|
const extra = new Blob(['# 另一篇\n正文']); extra.name = 'b.md'; extra.webkitRelativePath = '文章/b.md';
|
||||||
|
const result = await core.prepareFiles([...files, extra]);
|
||||||
|
assert.equal(result.candidates.length, 2);
|
||||||
|
assert.equal((await core.prepareFiles([...files, extra], '文章/b.md')).title, '另一篇');
|
||||||
|
await assert.rejects(core.prepareFiles(files.filter(f => !f.name.endsWith('.md'))), /没有/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('下载程序对接:Python 文章包与目录直接导入完全一致', async () => {
|
||||||
|
const data = JSON.parse(execFileSync('python3', [path.join(__dirname, '../modules/package.py'), directory], {encoding:'utf8'}));
|
||||||
|
const packaged = await core.preparePackage(data);
|
||||||
|
const direct = await core.prepareFiles(articleFiles());
|
||||||
|
assert.equal(packaged.title, direct.title);
|
||||||
|
assert.equal(packaged.body, direct.body);
|
||||||
|
assert.equal(packaged.images.size, 3);
|
||||||
|
for (const [p, blob] of packaged.images) assert.equal(blob.size, fs.statSync(path.join(directory, p)).size);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('服务器返回错误、登录 HTML 或混入脚本时不猜测附件编号', () => {
|
||||||
|
assert.equal(core.parseAid(' 123456\n'), '123456');
|
||||||
|
for (const value of ['-1', '0', '<html>123</html>', '<script>123</script>', '123error', '{"aid":123}']) assert.throws(() => core.parseAid(value));
|
||||||
|
});
|
||||||
|
|
||||||
|
// Small DOM harness exercises the public API and site adapter without installing browser software.
|
||||||
|
function harness({ failUpload = false, loseImages = false, existing = false } = {}) {
|
||||||
|
const ids = new Map(), callbacks = new Map(), requests = [];
|
||||||
|
class Element {
|
||||||
|
constructor(id='') { this.id=id; this.style={}; this.value=''; this.disabled=false; this.checked=false; this.hidden=false; this.listeners=new Map(); this.children=[]; }
|
||||||
|
attachShadow() { return this; }
|
||||||
|
set innerHTML(value) { this._html=value; for (const m of value.matchAll(/\bid="([^"]+)"/g)) ids.set(m[1], new Element(m[1])); }
|
||||||
|
get innerHTML() { return this._html || ''; }
|
||||||
|
getElementById(id) { return ids.get(id); }
|
||||||
|
appendChild(el) { this.children.push(el); return el; }
|
||||||
|
replaceChildren(...els) { this.children=els; }
|
||||||
|
addEventListener(name, fn) { this.listeners.set(name, fn); }
|
||||||
|
removeEventListener(name) { this.listeners.delete(name); }
|
||||||
|
dispatchEvent() {}
|
||||||
|
click() { this.onclick?.(); }
|
||||||
|
querySelectorAll() { return []; }
|
||||||
|
}
|
||||||
|
for (const id of ['subject', 'e_textarea', 'e_switchercheck', 'postform']) ids.set(id, new Element(id));
|
||||||
|
ids.get('subject').value=existing ? '原标题' : '';
|
||||||
|
ids.get('e_textarea').value=existing ? '原正文' : '';
|
||||||
|
const doc = { body:new Element(), createElement: () => new Element(), getElementById:id=>ids.get(id),
|
||||||
|
querySelector: s=>ids.get(s.replace(/^#/,'')), querySelectorAll:()=>[] };
|
||||||
|
const win = { document:doc, location:{search:'?mod=post&action=newthread&fid=95',href:'https://www.dww11.com/forum.php?mod=post&action=newthread&fid=95',origin:'https://www.dww11.com'},
|
||||||
|
wysiwyg:0, allowbbcode:1, allowimgcode:1, crypto:webcrypto,
|
||||||
|
imgUpload:{settings:{upload_url:'https://www.dww11.com/misc.php?mod=swfupload&operation=upload&fid=95',post_params:{uid:'fake-user',hash:'fake-hash',type:'image'},file_size_limit:20480}},
|
||||||
|
confirm:()=>true, dispatchEvent:()=>{}, addEventListener:()=>{}, editdoc:{body:new Element()} };
|
||||||
|
win.getEditorContents=()=>win.wysiwyg ? win.editdoc.body.innerHTML : ids.get('e_textarea').value;
|
||||||
|
win.switchEditor=mode=>{
|
||||||
|
if (mode===win.wysiwyg) return;
|
||||||
|
if (mode===1) win.editdoc.body.innerHTML=ids.get('e_textarea').value.replace(/\[attachimg\](\d+)\[\/attachimg\]/g, (_,aid)=>loseImages ? '' : `<img aid="attachimg_${aid}">`);
|
||||||
|
else ids.get('e_textarea').value=win.editdoc.body.innerHTML;
|
||||||
|
win.wysiwyg=mode;
|
||||||
|
};
|
||||||
|
win.updateImageList=(_action,aids)=>{ for (const id of aids.split('|')) ids.set('image_'+id,new Element()); };
|
||||||
|
win.fetch=async (url,options)=>{
|
||||||
|
requests.push({url,options});
|
||||||
|
return {ok:true,text:async()=>failUpload ? '-1' : String(1000+requests.length)};
|
||||||
|
};
|
||||||
|
const context = {window:win,URL,URLSearchParams,Blob,FormData,AbortController,Uint8Array,Option:Element,Event:class {},CustomEvent:class {},
|
||||||
|
MutationObserver:class {observe(){} disconnect(){}},atob,setTimeout,clearTimeout,console};
|
||||||
|
vm.runInNewContext(fs.readFileSync(path.join(__dirname, '../browser/importer.js'),'utf8'), context);
|
||||||
|
return {api:win.DWW11Importer, win, ids, requests};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('完整导入:暂存零请求、相同图片去重、3处附件映射、富文本结果、重试复用、没有发布请求', async () => {
|
||||||
|
const h=harness();
|
||||||
|
await h.api.stageFiles(articleFiles());
|
||||||
|
assert.equal(h.requests.length,0);
|
||||||
|
const result=await h.api.importPrepared();
|
||||||
|
assert.equal(Object.keys(result.images).length,3);
|
||||||
|
const uniqueImages=new Set(fs.readdirSync(directory).filter(p=>p.endsWith('.png')).map(p=>createHash('sha256').update(fs.readFileSync(path.join(directory,p))).digest('hex'))).size;
|
||||||
|
assert.equal(h.requests.length,uniqueImages);
|
||||||
|
assert.equal(h.ids.get('subject').value,mdName.slice(0,-3));
|
||||||
|
assert.match(h.win.getEditorContents(),/attachimg_1001/);
|
||||||
|
assert.equal(h.api.getState().busy,false);
|
||||||
|
assert.equal(h.api.getBackup().content,'');
|
||||||
|
await h.api.importPrepared();
|
||||||
|
assert.equal(h.requests.length,uniqueImages);
|
||||||
|
for (const req of h.requests) {
|
||||||
|
assert.equal(new URL(req.url).pathname,'/misc.php');
|
||||||
|
assert.equal(req.options.body.get('type'),'image');
|
||||||
|
assert.ok(req.options.body.get('Filedata') instanceof Blob);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('上传失败保留原文;富文本丢图片时回滚;取消覆盖不上传', async () => {
|
||||||
|
for (const options of [{failUpload:true,existing:true},{loseImages:true,existing:true}]) {
|
||||||
|
const h=harness(options);
|
||||||
|
await h.api.stageFiles(articleFiles());
|
||||||
|
await assert.rejects(h.api.importPrepared());
|
||||||
|
assert.equal(h.ids.get('subject').value,'原标题');
|
||||||
|
assert.equal(h.win.getEditorContents(),'原正文');
|
||||||
|
assert.equal(h.api.getState().busy,false);
|
||||||
|
}
|
||||||
|
const h=harness({existing:true}); h.win.confirm=()=>false;
|
||||||
|
await h.api.stageFiles(articleFiles());
|
||||||
|
assert.equal((await h.api.importPrepared()).cancelled,true);
|
||||||
|
assert.equal(h.requests.length,0);
|
||||||
|
});
|
||||||
|
|
||||||
|
// New integration guards must not prompt or overwrite a user's draft.
|
||||||
|
test('自动导入:非空编辑器停止,不弹窗、不上传', async () => {
|
||||||
|
const h=harness({existing:true});h.win.confirm=()=>{throw new Error('unexpected prompt');};
|
||||||
|
await h.api.stageFiles(articleFiles());
|
||||||
|
await assert.rejects(h.api.importPrepared({automatic:true}),/编辑器已有内容/);
|
||||||
|
assert.equal(h.requests.length,0);assert.equal(h.ids.get('subject').value,'原标题');
|
||||||
|
});
|
||||||
|
test('自动导入:预先终止不上传,不填写', async () => {
|
||||||
|
const h=harness();await h.api.stageFiles(articleFiles());const c=new AbortController();c.abort(new Error('stopped'));
|
||||||
|
await assert.rejects(h.api.importPrepared({automatic:true,signal:c.signal}),/stopped/);
|
||||||
|
assert.equal(h.requests.length,0);assert.equal(h.ids.get('subject').value,'');
|
||||||
|
});
|
||||||
|
test('自动导入:上传中终止不会填写正文或继续下一张', async () => {
|
||||||
|
const h=harness();await h.api.stageFiles(articleFiles());const c=new AbortController();
|
||||||
|
const original=h.win.fetch;h.win.fetch=async (...args)=>{const result=await original(...args);c.abort(new Error('stopped'));return result;};
|
||||||
|
await assert.rejects(h.api.importPrepared({automatic:true,signal:c.signal}),/stopped/);
|
||||||
|
assert.equal(h.requests.length,1);assert.equal(h.ids.get('subject').value,'');
|
||||||
|
});
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
"""Temporary local UI fixture; never contacts the news API or forum."""
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from http.server import ThreadingHTTPServer
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
from app import Controller, make_handler
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
controller = Controller(Path(temp) / 'data', Path(temp) / 'downloads', ledger=None)
|
||||||
|
controller.store.enqueue(ROOT / 'test/fixtures/article')
|
||||||
|
server = ThreadingHTTPServer(('127.0.0.1', 8771), make_handler(controller))
|
||||||
|
print('UI fixture listening on 8771', flush=True)
|
||||||
|
try:
|
||||||
|
server.serve_forever()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
server.server_close()
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import unittest
|
||||||
|
from http.server import ThreadingHTTPServer
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
from urllib.error import HTTPError
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
from app import Controller, make_handler
|
||||||
|
|
||||||
|
|
||||||
|
class ControllerTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.tmp.name)
|
||||||
|
self.controller = Controller(self.root / 'data', self.root / 'downloads', ledger=None)
|
||||||
|
self.server = ThreadingHTTPServer(('127.0.0.1', 0), make_handler(self.controller))
|
||||||
|
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
|
||||||
|
self.thread.start()
|
||||||
|
self.url = 'http://127.0.0.1:' + str(self.server.server_port)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.server.shutdown();self.server.server_close();self.thread.join();self.tmp.cleanup()
|
||||||
|
|
||||||
|
def request(self, path, body=None, token=None, origin=None, host=None):
|
||||||
|
headers = {}
|
||||||
|
if token: headers['X-Session-Token'] = token
|
||||||
|
if origin: headers['Origin'] = origin
|
||||||
|
if host: headers['Host'] = host
|
||||||
|
if body is not None: headers['Content-Type'] = 'application/json'
|
||||||
|
request = Request(self.url + path, data=None if body is None else json.dumps(body).encode(), headers=headers)
|
||||||
|
with urlopen(request, timeout=5) as response:
|
||||||
|
return response.status, response.read()
|
||||||
|
|
||||||
|
def test_auth_and_origin_boundaries(self):
|
||||||
|
for path, kwargs in [('/api/state', {}), ('/api/state', {'token': self.controller.bridge_token}),
|
||||||
|
('/api/state', {'token': self.controller.ui_token, 'origin': 'https://example.com'}),
|
||||||
|
('/bridge/heartbeat', {'token': self.controller.ui_token})]:
|
||||||
|
with self.assertRaises(HTTPError) as error:
|
||||||
|
self.request(path, {}, **kwargs)
|
||||||
|
self.assertEqual(error.exception.code, 403)
|
||||||
|
with self.assertRaises(HTTPError):
|
||||||
|
self.request('/api/session', host='attacker.example')
|
||||||
|
code, body = self.request('/api/state', {}, token=self.controller.ui_token)
|
||||||
|
self.assertEqual(code, 200)
|
||||||
|
self.assertNotIn(self.controller.bridge_token, body.decode())
|
||||||
|
|
||||||
|
def test_static_and_unknown_routes(self):
|
||||||
|
self.assertIn('新闻工作台', self.request('/')[1].decode())
|
||||||
|
self.assertEqual(self.request('/browser/auto-news.user.js')[0], 200)
|
||||||
|
with self.assertRaises(HTTPError):
|
||||||
|
self.request('/../store.py')
|
||||||
|
with self.assertRaises(HTTPError):
|
||||||
|
self.request('/bridge/scan', {}, token=self.controller.bridge_token)
|
||||||
|
|
||||||
|
def test_download_to_queue_without_network(self):
|
||||||
|
self.controller.manager.requester = lambda *_: {'code': 1, 'data': {'list': [dict(title='模拟新闻', src='模拟来源', content='<p>模拟正文</p>', url='https://example.com/news')]}}
|
||||||
|
self.controller.ui('download', dict(key='fake-key', channel='模拟频道', target=1))
|
||||||
|
self.controller.manager.thread.join(5)
|
||||||
|
state = self.controller.ui('state', {})
|
||||||
|
self.assertEqual(state['download']['saved'], 1)
|
||||||
|
self.assertEqual(len(state['jobs']), 1)
|
||||||
|
self.assertEqual(state['jobs'][0]['state'], 'queued')
|
||||||
|
self.assertTrue(state['jobs'][0]['source_keys'])
|
||||||
|
self.assertNotIn('fake-key', json.dumps(state))
|
||||||
|
self.controller.scan(self.controller.manager.root)
|
||||||
|
self.assertEqual(len(self.controller.store.state()['jobs']), 1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,165 @@
|
|||||||
|
import base64
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import unittest
|
||||||
|
from http.server import ThreadingHTTPServer
|
||||||
|
|
||||||
|
import requests
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||||
|
from app import downloader
|
||||||
|
Downloader, make_handler, fingerprints = downloader.Downloader, downloader.make_handler, downloader.fingerprints
|
||||||
|
from article_writer import save_article
|
||||||
|
|
||||||
|
|
||||||
|
def article(i):
|
||||||
|
return dict(title=f'新闻{i}', content=f'<p>第{i}篇新闻内容</p>', url=f'https://example.com/{i}', src='测试来源', time='2026-10-31')
|
||||||
|
|
||||||
|
|
||||||
|
class DownloadTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
def run_job(self, manager, target):
|
||||||
|
manager.start('test-key', '头条', target)
|
||||||
|
manager.thread.join(10)
|
||||||
|
self.assertFalse(manager.thread.is_alive())
|
||||||
|
return manager.snapshot()
|
||||||
|
|
||||||
|
def test_format_and_local_image(self):
|
||||||
|
data = article(1)
|
||||||
|
image = 'data:image/png;base64,' + base64.b64encode(b'\x89PNG\r\n\x1a\nexample').decode()
|
||||||
|
data['content'] += f'<img src="{image}"><img src="{image}">'
|
||||||
|
result = save_article(data, self.root)
|
||||||
|
body = Path(result.markdown).read_text()
|
||||||
|
self.assertTrue(body.startswith('# 新闻1\n\n第1篇新闻内容'))
|
||||||
|
self.assertNotIn('发布时间:2026-10-31', body)
|
||||||
|
self.assertNotIn('\n---\n', body)
|
||||||
|
self.assertTrue(body.endswith('来源:测试来源,如有侵权,请联系删除。\n'))
|
||||||
|
self.assertEqual(len(result.images), 1)
|
||||||
|
self.assertEqual(body.count('image001.png'), 2)
|
||||||
|
self.assertNotIn('data:image', body)
|
||||||
|
second = save_article(data, self.root)
|
||||||
|
self.assertEqual(Path(second.markdown).name, '新闻1 (2).md')
|
||||||
|
|
||||||
|
def test_bad_image_cleans_partial(self):
|
||||||
|
data = article(1)
|
||||||
|
data['content'] += '<img src="data:image/png;base64,aW52YWxpZA==">'
|
||||||
|
with self.assertRaises(Exception):
|
||||||
|
save_article(data, self.root)
|
||||||
|
self.assertEqual(list(self.root.iterdir()), [])
|
||||||
|
|
||||||
|
def test_duplicate_then_fill_and_restart(self):
|
||||||
|
items = [article(1), article(1), article(2), article(3)]
|
||||||
|
calls = []
|
||||||
|
def request(key, endpoint, params):
|
||||||
|
calls.append(params.copy())
|
||||||
|
return {'code': 1, 'data': {'list': items[params['start']:params['start']+params['num']]}}
|
||||||
|
manager = Downloader(self.root, request)
|
||||||
|
state = self.run_job(manager, 3)
|
||||||
|
self.assertEqual((state['status'], state['saved'], state['skipped']), ('completed', 3, 1))
|
||||||
|
self.assertEqual([c['start'] for c in calls], [0, 3])
|
||||||
|
restarted = Downloader(self.root, request)
|
||||||
|
state = self.run_job(restarted, 3)
|
||||||
|
self.assertEqual((state['status'], state['saved'], state['skipped']), ('partial', 0, 4))
|
||||||
|
self.assertEqual(len(list(self.root.glob('*/*.md'))), 3)
|
||||||
|
|
||||||
|
def test_offset_limit_and_partial_progress(self):
|
||||||
|
offsets = []
|
||||||
|
def request(key, endpoint, params):
|
||||||
|
offsets.append(params['start'])
|
||||||
|
return {'code': 1, 'data': {'list': [article(i) for i in range(params['start'], params['start']+params['num'])]}}
|
||||||
|
state = self.run_job(Downloader(self.root, request), 500)
|
||||||
|
self.assertEqual(state['status'], 'partial')
|
||||||
|
self.assertEqual(state['saved'], 440)
|
||||||
|
self.assertEqual(max(offsets), 400)
|
||||||
|
|
||||||
|
def test_error_and_key_redaction(self):
|
||||||
|
def request(*args):
|
||||||
|
return {'code': 10001, 'msg': 'invalid test-key'}
|
||||||
|
manager = Downloader(self.root, request)
|
||||||
|
state = self.run_job(manager, 10)
|
||||||
|
self.assertEqual(state['status'], 'error')
|
||||||
|
self.assertNotIn('test-key', json.dumps(state))
|
||||||
|
|
||||||
|
def test_stop_and_concurrent_start(self):
|
||||||
|
entered, release = threading.Event(), threading.Event()
|
||||||
|
def request(*args):
|
||||||
|
entered.set()
|
||||||
|
release.wait(2)
|
||||||
|
return {'code': 1, 'data': {'list': [article(1)]}}
|
||||||
|
manager = Downloader(self.root, request)
|
||||||
|
manager.start('key', '头条', 10)
|
||||||
|
self.assertTrue(entered.wait(2))
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
manager.start('key', '头条', 10)
|
||||||
|
manager.stop.set()
|
||||||
|
release.set()
|
||||||
|
manager.thread.join(3)
|
||||||
|
self.assertEqual(manager.snapshot()['status'], 'cancelled')
|
||||||
|
self.assertEqual(manager.snapshot()['saved'], 0)
|
||||||
|
|
||||||
|
def test_failed_save_can_retry_next_job(self):
|
||||||
|
attempts = []
|
||||||
|
def writer(data, root):
|
||||||
|
attempts.append(1)
|
||||||
|
if len(attempts) == 1:
|
||||||
|
raise ValueError('模拟图片失败')
|
||||||
|
return save_article(data, root)
|
||||||
|
def request(key, endpoint, params):
|
||||||
|
return {'code': 1, 'data': {'list': [article(1)] if params['start'] == 0 else []}}
|
||||||
|
manager = Downloader(self.root, request, writer)
|
||||||
|
self.assertEqual(self.run_job(manager, 1)['failed'], 1)
|
||||||
|
self.assertEqual(self.run_job(manager, 1)['saved'], 1)
|
||||||
|
|
||||||
|
def test_fingerprint_content_and_tracking(self):
|
||||||
|
a, b = article(1), article(2)
|
||||||
|
b['content'] = a['content'].replace('<p>', '<div>').replace('</p>', '</div>')
|
||||||
|
self.assertTrue(set(fingerprints(a)) & set(fingerprints(b)))
|
||||||
|
b['content'] = '其他内容'
|
||||||
|
b['url'] = a['url'] + '?utm_source=abc#fragment'
|
||||||
|
self.assertTrue(set(fingerprints(a)) & set(fingerprints(b)))
|
||||||
|
|
||||||
|
def test_log_cap(self):
|
||||||
|
manager = Downloader(self.root)
|
||||||
|
for i in range(400):
|
||||||
|
manager.log(str(i))
|
||||||
|
self.assertEqual(len(manager.snapshot()['logs']), 300)
|
||||||
|
self.assertEqual(manager.snapshot()['logs'][0]['message'], '100')
|
||||||
|
|
||||||
|
def test_http_routes_and_access(self):
|
||||||
|
def request(key, endpoint, params=None):
|
||||||
|
return {'code': 1, 'data': {'list': ['头条', '科技']}}
|
||||||
|
manager = Downloader(self.root, request)
|
||||||
|
server = ThreadingHTTPServer(('127.0.0.1', 0), make_handler(manager, 'token'))
|
||||||
|
worker = threading.Thread(target=server.serve_forever, daemon=True)
|
||||||
|
worker.start()
|
||||||
|
url = f'http://127.0.0.1:{server.server_port}'
|
||||||
|
try:
|
||||||
|
self.assertIn('新闻下载器', requests.get(url).text)
|
||||||
|
self.assertEqual(requests.get(url+'/api/state').json()['status'], 'idle')
|
||||||
|
self.assertEqual(requests.post(url+'/api/channels', json={'key': 'test'}).status_code, 403)
|
||||||
|
self.assertEqual(requests.get(url, headers={'Host':'evil.example'}).status_code, 403)
|
||||||
|
headers = {'X-Session-Token': 'token'}
|
||||||
|
self.assertEqual(requests.post(url+'/api/channels', headers=headers, json={'key':'test'}).json()['channels'], ['头条','科技'])
|
||||||
|
invalid = requests.post(url+'/api/start', headers=headers, json={'key':'test', 'target':0,'channel':'头条'})
|
||||||
|
self.assertEqual(invalid.status_code, 400)
|
||||||
|
invalid = requests.post(url+'/api/debug', headers=headers, json={'key':'test','endpoint':'index','num':41,'start':0})
|
||||||
|
self.assertEqual(invalid.status_code, 400)
|
||||||
|
debug = requests.post(url+'/api/debug', headers=headers, json={'key':'test','endpoint':'channel'})
|
||||||
|
self.assertEqual(debug.status_code, 200)
|
||||||
|
self.assertNotIn('key=test', debug.text)
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
server.server_close()
|
||||||
|
worker.join()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
from articles import inspect_article, short_title
|
||||||
|
from store import Store
|
||||||
|
|
||||||
|
|
||||||
|
class QueueTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.tmp.name)
|
||||||
|
self.article = self.root / 'article'
|
||||||
|
self.article.mkdir()
|
||||||
|
self.md = self.article / 'a.md'
|
||||||
|
self.md.write_text('# 测试新闻\n\n正文内容。\n\n来源:测试来源,如有侵权,请联系删除。\n')
|
||||||
|
self.store = Store(self.root / 'queue.sqlite3')
|
||||||
|
self.store.configure(dict(fid=125, minimum=1, maximum=1, interval=0, history_checked=True))
|
||||||
|
self.owner = dict(owner='tab-a', account=dict(uid='123', name='测试账号'), extension={'idle': True})
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def begin(self):
|
||||||
|
self.store.enqueue(self.article)
|
||||||
|
self.store.control('start')
|
||||||
|
return self.store.next(self.owner)['job']
|
||||||
|
|
||||||
|
def transition(self, job, state, **extra):
|
||||||
|
return self.store.event(dict(owner='tab-a', id=job['id'], state=state, epoch=self.store.run['epoch'], **extra))
|
||||||
|
|
||||||
|
def submit(self, job):
|
||||||
|
for state in ('importing', 'formatting', 'awaiting_user_review'):
|
||||||
|
self.transition(job, state)
|
||||||
|
self.transition(job, 'user_submitting', snapshot={'title': job['title']})
|
||||||
|
|
||||||
|
def accept(self, job):
|
||||||
|
return self.store.accept(dict(owner='tab-a', id=job['id'], post_url='https://www.dww11.com/thread/12345.html',
|
||||||
|
author_uid='123', fid=125, title=job['title'], state='accepted_pending_moderation', evidence={'test': 'fixture only'}))
|
||||||
|
|
||||||
|
def test_stable_identity_and_original_preserved(self):
|
||||||
|
before = self.md.read_bytes()
|
||||||
|
item = inspect_article(self.article)
|
||||||
|
copy = self.root / 'renamed'
|
||||||
|
shutil.copytree(self.article, copy)
|
||||||
|
(copy / 'a.md').write_text(self.md.read_text().replace('测试新闻', '改标题'))
|
||||||
|
self.assertEqual(inspect_article(copy)['article_key'], item['article_key'])
|
||||||
|
self.assertEqual(self.md.read_bytes(), before)
|
||||||
|
self.assertEqual(item['source'], '测试来源')
|
||||||
|
|
||||||
|
def test_title_does_not_break_emoji(self):
|
||||||
|
short = short_title('😀' * 50)
|
||||||
|
self.assertTrue(short.endswith('...'))
|
||||||
|
self.assertLessEqual(len(short.encode('utf-16-le')) // 2, 80)
|
||||||
|
self.assertEqual(short_title('短标题'), '短标题')
|
||||||
|
|
||||||
|
def test_unsupported_images_stop_before_queue(self):
|
||||||
|
(self.article / 'a.webp').write_bytes(b'webp')
|
||||||
|
with self.assertRaisesRegex(ValueError, '暂不支持'):
|
||||||
|
self.store.enqueue(self.article)
|
||||||
|
self.assertEqual(self.store.state()['jobs'], [])
|
||||||
|
|
||||||
|
def test_missing_and_outside_image_rejected(self):
|
||||||
|
for image in ('missing.png', '../outside.png', 'https://example.com/a.png'):
|
||||||
|
self.md.write_text('# 标题\n\n正文 \n\n来源:测试,如有侵权,请联系删除。')
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
self.store.enqueue(self.article)
|
||||||
|
|
||||||
|
def test_no_duplicate_queue_or_cross_tab_claim(self):
|
||||||
|
job = self.begin()
|
||||||
|
self.assertIn('已在队列', self.store.enqueue(self.article))
|
||||||
|
self.assertEqual(self.store.next({**self.owner, 'owner': 'tab-b'})['action'], 'other_tab')
|
||||||
|
self.assertEqual(self.store.next(self.owner)['job']['id'], job['id'])
|
||||||
|
|
||||||
|
def test_stop_invalidates_old_continuation(self):
|
||||||
|
job = self.begin()
|
||||||
|
epoch = self.store.run['epoch']
|
||||||
|
self.store.control('stop')
|
||||||
|
self.store.control('start')
|
||||||
|
with self.assertRaisesRegex(ValueError, '过期'):
|
||||||
|
self.store.event(dict(owner='tab-a', id=job['id'], state='importing', epoch=epoch))
|
||||||
|
|
||||||
|
def test_restart_pauses_and_preserves_claim(self):
|
||||||
|
job = self.begin()
|
||||||
|
restarted = Store(self.root / 'queue.sqlite3')
|
||||||
|
self.assertFalse(restarted.run['running'])
|
||||||
|
self.assertEqual(restarted.run['active'], job['id'])
|
||||||
|
self.assertEqual(restarted.run['batch']['target'], 1)
|
||||||
|
|
||||||
|
def test_accept_requires_manual_submission(self):
|
||||||
|
job = self.begin()
|
||||||
|
with self.assertRaisesRegex(ValueError, '人工提交'):
|
||||||
|
self.accept(job)
|
||||||
|
|
||||||
|
def test_accept_is_atomic_and_idempotent(self):
|
||||||
|
job = self.begin()
|
||||||
|
self.submit(job)
|
||||||
|
self.accept(job)
|
||||||
|
self.accept(job)
|
||||||
|
self.assertEqual(self.store.run['batch']['count'], 1)
|
||||||
|
self.assertIn('已受理', self.store.enqueue(self.article))
|
||||||
|
with self.assertRaisesRegex(ValueError, '已受理'):
|
||||||
|
self.store.resolve(dict(id=job['id'], confirmed_not_accepted=True, retry=True))
|
||||||
|
|
||||||
|
def test_unknown_never_requeues_automatically(self):
|
||||||
|
job = self.begin()
|
||||||
|
self.submit(job)
|
||||||
|
self.transition(job, 'submit_unknown')
|
||||||
|
self.assertEqual(self.store.next(self.owner)['job']['state'], 'submit_unknown')
|
||||||
|
self.assertEqual(self.store.run['batch']['count'], 0)
|
||||||
|
self.assertIsNone(self.store.run['switch'])
|
||||||
|
|
||||||
|
def test_migrate_real_ledger_shape(self):
|
||||||
|
record = dict(markdown_sha256=hashlib.sha256(self.md.read_bytes()).hexdigest(), block_repost=True,
|
||||||
|
markdown_path=str(self.md), status='accepted_pending_moderation', post_id='12345')
|
||||||
|
self.store.import_history([record])
|
||||||
|
self.assertIn('已受理', self.store.enqueue(self.article))
|
||||||
|
|
||||||
|
def test_switch_only_after_target_and_with_more_work(self):
|
||||||
|
job = self.begin()
|
||||||
|
self.submit(job)
|
||||||
|
self.accept(job)
|
||||||
|
other = self.root / 'other';shutil.copytree(self.article, other)
|
||||||
|
(other / 'a.md').write_text(self.md.read_text().replace('正文内容', '另一篇内容'))
|
||||||
|
self.store.enqueue(other)
|
||||||
|
self.assertEqual(self.store.next(self.owner)['action'], 'switch')
|
||||||
|
self.assertEqual(self.store.next(self.owner)['action'], 'switch')
|
||||||
|
self.store.switch_begin({'owner': 'tab-a'})
|
||||||
|
with self.assertRaisesRegex(ValueError, '禁止重复'):
|
||||||
|
self.store.switch_begin({'owner': 'tab-a'})
|
||||||
|
self.assertEqual(self.store.next(self.owner)['action'], 'switch_wait')
|
||||||
|
next_job = self.store.next({**self.owner, 'account': {'uid': '456', 'name': '新测试账号'}})
|
||||||
|
self.assertEqual(next_job['action'], 'prepare')
|
||||||
|
self.assertEqual(self.store.run['batch']['count'], 0)
|
||||||
|
|
||||||
|
def test_no_switch_when_queue_done(self):
|
||||||
|
job = self.begin();self.submit(job);self.accept(job)
|
||||||
|
self.assertEqual(self.store.next(self.owner)['action'], 'idle')
|
||||||
|
self.assertIsNone(self.store.run['switch'])
|
||||||
|
|
||||||
|
def test_bad_result_does_not_increment(self):
|
||||||
|
job = self.begin();self.submit(job)
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
self.store.accept(dict(owner='tab-a', id=job['id'], post_url='https://example.com/thread/12345.html'))
|
||||||
|
self.assertEqual(self.store.run['batch']['count'], 0)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# 本轮验证记录
|
||||||
|
|
||||||
|
日期:2026-10-31。测试均未使用真实账号,未向新闻 API 请求内容,未向论坛上传或发布。
|
||||||
|
|
||||||
|
| 范围 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| Python 下载器、队列与 HTTP 接口 | 27/27 通过 |
|
||||||
|
| JavaScript 导入器与停止/覆盖保护 | 11/11 通过 |
|
||||||
|
| 连接桥旧响应与终止竞态 | 1/1 通过 |
|
||||||
|
| 本地前端 | 配置保存、开始/停止、桌面与手机布局通过,无页面脚本错误、无手机横向溢出 |
|
||||||
|
| 模拟编辑器 | 导入→排版→待人工审核;来源唯一、重复排版保持一致;程序无自动发帖点击 |
|
||||||
|
| 模拟人工发表与未知结果 | 真实测试点击被记录;候选结果页未适配时暂停,不调用受理接口 |
|
||||||
|
| 模拟换号插件 | 启动请求尚未返回时终止,返回后调用暂停,不执行登录;状态明确 paused |
|
||||||
|
| JavaScript 语法 | 合并脚本、连接脚本、更新后的插件 content.js 通过 |
|
||||||
|
|
||||||
|
浏览器使用本机 Chrome 的独立临时配置。对论坛域名的所有请求均被测试 HTML 拦截。后端使用临时目录和 SQLite,不写入正式任务库。
|
||||||
|
|
||||||
|
页面截图:`artifacts/desktop.png`、`artifacts/mobile.png`。
|
||||||
|
|
||||||
|
测试中曾发现模拟网页未声明 UTF-8,导致中文登录入口无法匹配;已修正测试响应编码并重新通过。真实网页编码未由此假定。
|
||||||
|
|
||||||
|
尚未验证:用户浏览器中油猴本地通信权限、真实网站当前的导入与附件转换、真实换号账号链路、结果页受理/审核/公开状态。按用户决定,结果页 HTML 待提供,自动核对暂不实现;因此当前不宣称全流程批量处理已完成在线验收。
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
# 广谈大湾网:下载、导入排版、人工发布与换号
|
||||||
|
|
||||||
|
本文中的帐号名称和用户编号已匿名化;真实记录保存在本地排除目录中。
|
||||||
|
|
||||||
|
记录日期:2026-10-31(北京时间)。本文依据用户指导、Chrome 实际操作和现有源码检查编写。它是后续实现的流程依据;需求不等于功能已经实现。
|
||||||
|
|
||||||
|
## 1. 不可改变的边界
|
||||||
|
|
||||||
|
- **“发表帖子”只能由用户亲手点击。** 自动化只下载、选择、上传、填写、排版、校验、提示审核、观察提交结果和记录计数。禁止模拟点击发布、调用提交函数、直接发送发帖请求或通过快捷键提交。
|
||||||
|
- 标题被截断、正文或来源被修改后,都必须重新进入人工审核。发生标题过长错误后,“重新尝试发表”也由用户再次点击。
|
||||||
|
- 用户决定分区和文章根目录;“理财财经”只是本次例子,不能成为所有文章的固定分区。
|
||||||
|
- 全流程页面常驻“终止流程”按钮。点击后立即取消后续动作、重试、队列领取和自动换号,不自动恢复。已经到达网站的请求不能撤回,需保留记录并核对。
|
||||||
|
- 已发表或已被网站接收待审核的文章不能再次排入发布队列,换账号也不能绕过去重。
|
||||||
|
|
||||||
|
## 2. 本次演示记录
|
||||||
|
|
||||||
|
| 项目 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 站点 | https://www.dww11.com/ |
|
||||||
|
| 当前账号 | 示例用户,页面用户编号 <已匿名> |
|
||||||
|
| 分区 | 理财财经,fid=125 |
|
||||||
|
| 入口 | 首页右上角“免费发布信息”→ 分区页“理财财经” |
|
||||||
|
| 分区页 | https://www.dww11.com/special/post/ |
|
||||||
|
| 新帖页 | https://www.dww11.com/forum.php?mod=post&action=newthread&fid=125 |
|
||||||
|
| 标签行为 | 点击分区后新开标签,原分区页保留 |
|
||||||
|
| 测试标题 | 刚靠新电影赚23亿票房,周星驰就卖掉内地影院 |
|
||||||
|
| 原标题长度 | 22 个 Unicode 码点;本篇无需截断。网站计数方式尚未实测 |
|
||||||
|
| 目录 | `<项目根目录>/tanshuapi_news_downloader/downloads/刚靠新电影赚23亿票房,周星驰就卖掉内地影院` |
|
||||||
|
| 目录内容 | 1 个 Markdown + 26 张 JPG,共 27 个文件 |
|
||||||
|
| 来源 | 洲洲影视娱评(取文章末尾来源,不取正文“信息来源”截图) |
|
||||||
|
| 导入 | 已完成;面板提示“已填写标题、正文和 26 张配图” |
|
||||||
|
| 排版 | 已完成,正文 4 号黑色,尾注 1 号银色 |
|
||||||
|
| 核对 | 编辑器 DOM 中图片 26 张,最终来源尾注仅 1 条 |
|
||||||
|
| 发布 | 用户已亲手点击;网站已受理,详情页显示“审核中” |
|
||||||
|
| 帖子链接 | https://www.dww11.com/thread/23697411.html |
|
||||||
|
| 帖子标识 | 主题 ID 23697411;首帖 PID 238766 |
|
||||||
|
| 数量 | 已明确受理 1 篇;已确认审核通过并公开 0 篇 |
|
||||||
|
| 发布后图片 | 正文 21 张 + 附件区 5 张,共 26 张 |
|
||||||
|
| 换号 | 未执行;本次未设置最小值和最大值 |
|
||||||
|
|
||||||
|
本次页面没有“终止流程”按钮。用户明确同意本次演示暂用浏览器控制条停止,常驻页面按钮仍是后续自动化的必备条件。该例外只适用于本次演示。
|
||||||
|
|
||||||
|
本次已验证导入、排版、用户亲手发表以及网站受理后的帖子详情页;页面明确显示“审核中”。尚未验证审核通过后的访客可见性、标题超限错误或换号链路。用户称发表后自动回首页,但本次实际读取的原编辑器标签已导航至帖子详情页,不能将“提交后必然自动回首页”写成网站行为。应在结果登记后主动回首页并核对。
|
||||||
|
|
||||||
|
## 3. 用户操作步骤与自动化对应
|
||||||
|
|
||||||
|
1. 启动前让用户选择分区、下载目录、待处理文章范围,并填写每个账号发布数量的最小值和最大值。展示当前账号。先初始化常驻终止按钮与持久任务记录。
|
||||||
|
2. 从下载器接收完整落盘的文章目录。检查正文、图片引用、来源和去重记录;保存中或缺文件的目录不进入导入队列。
|
||||||
|
3. 点击“免费发布信息”,展示实际可用分区,由用户选择。等待新标签打开并核对分区名称及 fid;不能继续操作旧标签或仅凭坐标判断。
|
||||||
|
4. 在右下角“本地文章导入 · 1.0”选择“选择文章目录”。日常演示选择单篇目录,目录应包含 Markdown 和全部配图;不要只选 `.md`。如果选择根目录导致出现多个 Markdown,必须明确选中哪一篇。
|
||||||
|
5. 在系统文件选择器进入用户指定根目录,再选择对应文章文件夹。系统“Upload”是把所选文件提供给页面读取,不是发表帖子;浏览器另有目录文件数量确认。
|
||||||
|
6. 阅读导入面板标题、正文预览、图片数量和告警。点击“上传配图并填写”后才开始向网站上传正文引用的图片。等待明确完成提示,不能用固定等待时间代替完成判断。
|
||||||
|
7. 读取文章最后的标准尾注 `来源:名称,如有侵权,请联系删除。`,只取名称。本篇是“洲洲影视娱评”;用户先前茅台文章例子是“酒业内参”,不能硬编码。
|
||||||
|
8. 点击编辑工具栏绿色“一键排版”(不是网站自带“自动排版”,也不是“一键导入微信公众号并排版”)。实际弹窗要求输入**来源公众号名称**,并非标题。本篇填“洲洲影视娱评”,确认后会提示排版完成。
|
||||||
|
9. 标题由本地导入器填入页面标题框;仍需核对完整标题。当前“一键排版”不弹标题输入框。如果用户说“输入标题”,应在标题框填写和确认,而不是填到来源弹窗。
|
||||||
|
10. 校验标题、正文、图片数量和顺序、尾注唯一性、账号和分区。保留用户原文;本篇的发布时间、“打开网易新闻 查看精彩图片”等原始文字未擅自清理。需要清理时应成为用户可见的规则。
|
||||||
|
11. 页面提示“已完成,等待人工审核,请自行点击发表帖子”。自动化停止修改编辑器,保留终止按钮,等待用户操作;不自动保存草稿或发表。网站原有自动保存仍可能运行。
|
||||||
|
12. 用户自行点击“发表帖子”。自动化只观察结果。若标题超长,按第 5 节修正,重新提示审核,等待用户再次点击。
|
||||||
|
13. 收集帖子链接/编号、作者、分区和结果证据后记账。审核中单独记录并阻止重发;结果不明时先核对,不能自动重试发布。
|
||||||
|
14. 已明确成功后先持久保存发布记录与计数,再回到首页。达到当前账号目标篇数且没有未决提交时,点击页面顶部“登陆下一帐号”,核对登录成功后再处理下一篇。
|
||||||
|
|
||||||
|
## 4. 页面适配依据与当前缺口
|
||||||
|
|
||||||
|
以下选择器来自本次可见页面或本地源码,未来运行时须重新检查,不能保证网站改版后仍成立。
|
||||||
|
|
||||||
|
| 控件 | 当前依据 | 注意事项 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 当前用户 | `#mumucms_username` | 同时读取页面用户名与用户链接编号 |
|
||||||
|
| 下一账号 | `#dww11-extension-next` | 完成记录后才允许换号;核对成功,不以按钮点击作为成功 |
|
||||||
|
| 标题 | `#subject` | 本次读取到 maxLength=-1,不能只依赖 HTML maxlength |
|
||||||
|
| 可视化正文 | `iframe#e_iframe` 内的 body | 图文与尾注检查应在 iframe 内进行 |
|
||||||
|
| 纯文本开关 | `#e_switchercheck` | 不将 Markdown 直接当网站 BBCode |
|
||||||
|
| 一键排版 | `#dww11-format-oneclick` | 原生 prompt 输入来源,随后 alert 提示完成 |
|
||||||
|
| 发表 | `#postsubmit` | 仅用于识别状态和观察人工提交,禁止自动触发 |
|
||||||
|
| 导入面板 | 油猴脚本面板,包含 choose/import 等控件 | 面板可能位于 Shadow DOM;普通页面选择器未必能定位,优先按可见控件操作 |
|
||||||
|
|
||||||
|
本次实际遇到的兼容问题:
|
||||||
|
|
||||||
|
- 浏览器连接器直接设置目录文件被“允许访问文件网址”权限阻止。本次没有更改扩展权限,改用 Chrome 原生系统选择器完成。后续实现应优先复用本地文章包接口,减少对文件对话框的依赖。
|
||||||
|
- macOS 路径含中文,逐键输入曾丢失中文;通过文本框完整赋值成功定位。输入后必须核对完整路径,不能只看末级数字或英文。
|
||||||
|
- 导入器能完成 26 张图片上传,并临时禁用发表按钮;上传完成后恢复。上传过程中不允许换号。
|
||||||
|
- **来源重复:** 一键排版保留了导入正文中的原尾注,又追加银色尾注。本次在编辑器中删除原来的那一条,保留银色尾注,复查最终只出现一次。后续应在排版适配器中精确处理末尾标准尾注,不能全局删除正文中的“来源”。
|
||||||
|
- 页面“还可输入 80 个字符”在导入后仍显示 80,不能将这个显示值当作可靠的标题长度校验结果。
|
||||||
|
- 页面有“上次未提交成功的数据”恢复提示;本次没有点击恢复,避免把旧稿混入新稿。后续遇到已有正文或恢复冲突必须暂停处理。
|
||||||
|
|
||||||
|
## 5. 标题超过 80 字符
|
||||||
|
|
||||||
|
保留 `original_title`,派生 `submitted_title`。超过站点限制时截取前缀并追加字面量 `...`,总长不超过 80;即长度允许时保留前 77 个字符再加 3 个点。示例中的“标题文本...”表示实际标题前缀和省略号,不额外加入中文括号。
|
||||||
|
|
||||||
|
当前站点的“字符”计数方式尚未通过超长标题实测。实现时兼容中文和 emoji,不能切断代理对;先按保守规则校验,再根据站点明确错误调整。若站点按字节计数,则按其实际字节预算截取,不能反复盲目减少。
|
||||||
|
|
||||||
|
现有导入器在读取阶段对超过 80 个 Unicode 码点的标题直接报错。后续需把截断放在导入前,否则到不了人工审核阶段。截断不覆盖原 Markdown 标题,也不改变文章去重身份。
|
||||||
|
|
||||||
|
如果用户点击发表后才收到标题超限提示:确认网站明确未接受 → 修正标题 → 显示新旧标题 → 重新人工审核 → 用户再次点击。任何自动修正都不等于用户已批准发表。
|
||||||
|
|
||||||
|
## 6. 发布记录、去重与恢复
|
||||||
|
|
||||||
|
建议独立 SQLite 发布任务库,与下载去重库分开;下载成功不能视为发布成功。本次先保存演示任务 JSON,尚未实现自动数据库或网站历史帖子同步。
|
||||||
|
|
||||||
|
稳定文章身份优先采用下载器原始文章 ID 或规范化源 URL,同时保存正文与图片内容指纹用于发现重新下载/改名副本。禁止只按本地目录名、截断标题或当前账号去重。默认整个站点跨账号、跨版块不重复发表同一篇;如以后需要重发,必须由用户显式决定。
|
||||||
|
|
||||||
|
建议字段:article_key、source_url/source_id、content_hash、原始/最终标题、目录、来源、状态、账号 UID、fid、附件映射、排版版本、人工提交时间、提交尝试编号、帖子 ID/URL、网站受理/审核/公开状态、错误、创建和更新时间。日志不保存密码、Cookie 或上传凭证。
|
||||||
|
|
||||||
|
主状态:
|
||||||
|
|
||||||
|
```text
|
||||||
|
downloaded → queued → importing → formatting → awaiting_user_review
|
||||||
|
→ user_submitting → published / accepted_pending_moderation
|
||||||
|
```
|
||||||
|
|
||||||
|
分支:`stopped`、`needs_attention`、`submit_unknown`、`rejected`。未决提交不得重新上传或领取同一文章。单批次串行,跨标签领取加持久锁;刷新后先恢复状态,不自动启动。
|
||||||
|
|
||||||
|
去重记录需要覆盖用户已发表帖子:优先导入原有记录,或从“修改/删除信息”进入本人帖子列表进行核对。仅标题相同只能标记疑似重复,不能据此自动认定是同一正文。本次未扫描历史帖子,后续批量运行前必须完成历史对账;本次文章已记为“网站已受理、审核中”,已有帖子 ID 与 URL,必须阻止重发;不能误写为等待用户点击,也不能写为审核通过。
|
||||||
|
|
||||||
|
成功记录与计数使用同一个事务:帖子唯一键(站点+帖子 ID)防止页面刷新重复加一。先记录再返回首页。遇到网络超时、页面关闭或不明确提示,记录 `submit_unknown`,人工/页面核对后再决定,禁止自动重复发表。
|
||||||
|
|
||||||
|
网站明确受理待审核时,记 `accepted_pending_moderation` 并锁住文章。本规范建议把明确受理计入账号本轮提交数量一次,另存公开发表数量;后续审核通过只改变公开状态,不再加本轮数量。本次已观察到网站审核中状态,因此本地记录明确受理数为 1、已确认公开数为 0;批量自动记账尚未实现。驳回不自动补发或重投。
|
||||||
|
|
||||||
|
## 7. 每个账号随机篇数与换号
|
||||||
|
|
||||||
|
用户填写整数 `min`、`max`,要求 `1 ≤ min ≤ max`。首次为当前账号创建发布批次及每次确认换号成功后,从闭区间 `[min,max]` 均匀抽取整数 n,保存目标和当前计数。min=max 时固定该数量。不要按每篇重新抽 n,刷新、暂停或重启也不重抽。
|
||||||
|
|
||||||
|
- 本轮计数只含当前发布批次明确成功/明确受理的唯一文章,不含上传、排版、等待人工审核、失败和结果不明。
|
||||||
|
- 达到 n 后先保存记录与 `switch_pending`,回首页,核对没有待提交草稿,再点击“登陆下一帐号”。
|
||||||
|
- 观察并确认实际 UID/用户名变化且登录成功,再建立下一账号批次、重置批次计数并抽新 n。不可在不确定结果时重复点击换号。
|
||||||
|
- 验证码、账号异常、频率限制、登录失败或账号池无可用账号时暂停,保留当前文章与计数;不把受限文章换账号重投。
|
||||||
|
- 用户手动改变账号时立即暂停,重新关联批次与上传附件;不能沿用旧账号附件上下文。
|
||||||
|
- 队列已经处理完时直接结束,不为“凑满 n”重复文章,也不无意义换号。
|
||||||
|
|
||||||
|
本次没有给定 min/max,所以不随机设置目标,也不执行换号。账号扩展现有自己的抽号/登录持久记录,应复用其登录流程,不另存账号密码。
|
||||||
|
|
||||||
|
## 8. 常驻终止按钮的实现要求
|
||||||
|
|
||||||
|
按钮在首页、分区页、新帖页、结果页、换号页面保持可见,不依赖折叠的导入面板。跨标签共享同一批次停止状态,导航后先恢复按钮和停止状态再执行任务。
|
||||||
|
|
||||||
|
点击立即写入持久 `stopped` 并提升运行版本号,取消定时器、队列和可取消的上传/下载请求;每个异步步骤返回时检查停止标志和版本,过期回调不得再填写、跳转或换号。浏览器自带模态弹窗会阻挡网页按钮,因此自动流程应把来源/审核提示改成页面内面板,保留停止入口。
|
||||||
|
|
||||||
|
已被服务器接收的图片/帖子不能被停止按钮撤回。停止后允许被动记录已在途请求的最终结果,但不得继续业务动作;需要清楚显示“已停止,已发出的请求可能仍已完成”。人工终止后必须由用户明确重新启动,不能因刷新或服务重启自动恢复。
|
||||||
|
|
||||||
|
## 9. 后续实现与验收清单
|
||||||
|
|
||||||
|
1. 优先实现常驻终止按钮、持久队列、站点级去重和历史发布对账,再接批量导入。
|
||||||
|
2. 改造导入前标题截断、唯一来源尾注和可重复执行排版;保留原文与本地文件。
|
||||||
|
3. 建立本地任务服务与浏览器执行器之间的配对通信,只传当前文章所需数据,不传下载 API Key 或账号库。
|
||||||
|
4. 人工发布边界验收:没有用户点击时,等待任意时间都不会发表;标题修正后仍等待新的用户点击。
|
||||||
|
5. 验证缺图、上传部分失败、重复来源、长标题、emoji、多 Markdown、非空编辑器、人工修改、刷新/断网、多标签竞争。
|
||||||
|
6. 用户实际发布后,补录成功页或审核中页面证据、帖子 URL/ID,并验证重复打开不会增加计数。
|
||||||
|
7. 给定 min/max 后验证 n 范围、每账号仅抽一次、达到目标换号、核对新 UID、恢复不丢计数。
|
||||||
|
8. 在下载、上传、排版、等待审核、观察结果和换号各阶段点击终止,确认不再触发后续动作。
|
||||||
|
|
||||||
|
## 10. 当前交接
|
||||||
|
|
||||||
|
用户已亲手点击发表。原编辑器标签现为帖子详情页,标题与作者“示例用户”一致,面包屑为理财财经,显示“审核中”。已保存主题 ID、首帖 PID、URL、受理数量与去重记录,禁止再次发同一文章。登记后已点击网站首页入口,并核对 URL 为 `https://www.dww11.com/`,当前账号仍为示例用户(UID <已匿名>);没有设置 min/max,因此未换号。
|
||||||
|
|
||||||
|
发布结果检查发现正文内有 21 张图片,另外 5 张在附件区:image009.jpg、image014.jpg、image016.jpg、image025.jpg、image026.jpg,总数 26 张;来源尾注仅一条。发布后的正文与初次待审稿存在差异(例如原发布时间及“打开网易新闻 查看精彩图片”文字已不在结果正文中)。本次未记录用户审核期间的具体编辑动作,无法将差异归因于用户修改、编辑器同步或网站转换;后续需在用户点击的时刻保存最终待提交快照,再与结果对照。不得自动编辑或重发这篇已受理的帖子。
|
||||||
|
|
||||||
|
截图:`test/history/周星驰文章_待人工审核.jpg`(历史待审稿)、`test/history/周星驰文章_网站审核中.jpg`(网站受理结果)。演示任务:`test/history/演示任务记录.json`;站点级去重登记:`发布去重记录.json`。记录保存本次结果,不代表完整历史同步或批量程序已经实现。
|
||||||
|
|
||||||
|
|
||||||
|
## 11. 发布结果判断与本次补录
|
||||||
|
|
||||||
|
- 用户点击后,不按“回到首页”判断成功;先记录实际跳转 URL 和页面结果。
|
||||||
|
- 本次 URL 为 `/thread/23697411.html`,页面标题、理财财经面包屑、楼主示例用户及“审核中”共同确认网站已受理。DOM 首帖容器为 `#postmessage_238766`。
|
||||||
|
- 网站审核中并非失败,不能为追求公开状态而再次提交;也不能仅凭当前登录用户能看到详情,宣称所有访客可见。
|
||||||
|
- 刷新、重复读取或后来审核通过时,都用站点+主题 ID 保证受理数量只计一次。
|
||||||
|
- 本次没有出现标题超限,相关处理仍是待验证需求。
|
||||||
|
- 完整历史对账、常驻终止按钮、随机数量和自动换号仍属后续实现及验收范围。
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
# 新闻下载、导入排版、人工发表与换号实施计划
|
||||||
|
|
||||||
|
本文中的帐号名称和用户编号已匿名化;真实记录保存在本地排除目录中。帖子编号仍用于说明既有受理证据,不代表当前公开状态。
|
||||||
|
|
||||||
|
更新日期:2026-10-31(北京时间)。本版整体替换早期自动发帖设想,依据当前源码、[发布流程操作记录与自动化规范](发布流程操作记录与自动化规范.md)、[演示任务记录](演示任务记录.json)和[发布去重记录](发布去重记录.json)重新梳理。
|
||||||
|
|
||||||
|
本次进行了本地源码与记录核对,未重新操作线上页面、下载新闻、上传图片或发表帖子。下文“演示已验证”引用既有操作记录;“源码已有”不等于最新版本已在线上重新验收。
|
||||||
|
|
||||||
|
## 1. 当前目标与确定规则
|
||||||
|
|
||||||
|
目标是:自动下载、选文、上传、填写、排版和校验,等待用户亲手发表,然后观察结果、记录数量,达到账号目标后换号,继续下一篇。
|
||||||
|
|
||||||
|
- **发表及重试发表必须由用户亲手点击。** 不模拟点击、不调用提交函数、不直接发送发帖请求、不通过快捷键提交。自动化没有发布接口。
|
||||||
|
- 用户选择分区、文章根目录、处理范围及每账号篇数 min/max;理财财经 fid=125 是演示样例,不硬编码为全部任务的目标。
|
||||||
|
- 标题、正文或来源修改后重新进入人工审核;等待审核期间自动化不继续修改编辑器。
|
||||||
|
- 全流程常驻“终止流程”按钮,跨标签、导航和换号保持生效;停止后不因刷新或服务重启自动恢复。
|
||||||
|
- 同一文章在整个站点跨账号、跨版块去重。已公开发表或已受理待审核均阻止再次入队;结果不明时先核对。
|
||||||
|
- 新下载文章不额外插入接口发布时间和分隔线。历史文件及源正文中原有的时间、文字不因此自动删除。
|
||||||
|
|
||||||
|
结论:工作流可行,单篇人工链路已有实机记录,但持久队列、本地桥接、统一终止、自动计数和与换号扩展联动仍需实现。实施重点从“寻找全部页面元素”转为“连接已有组件并保证停止、审核、去重和结果核对”。
|
||||||
|
|
||||||
|
## 2. 本轮变化与现状
|
||||||
|
|
||||||
|
| 模块 | 当前确认 | 后续工作 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 排版脚本 `dww11-wechat-oneclick.user.js` v1.0.9 | 新增独立“一键排版”;`existingSource()` 识别文末来源/作者及版权行;`runFormat()` 取出原尾注后交给 `formatBody()` 重新排版;无来源才弹窗补充 | 提供可等待、可取消、无模态弹窗的程序接口;验证复杂尾注与编辑器转换后重复执行 |
|
||||||
|
| 下载保存 `article_writer.py` | 标题后直接接正文;不再插入接口 time;移除无文字且无图片的空标题节点;末尾仍生成标准来源尾注 | 输出持久文章清单与结构化来源/源网址/指纹,建立完整保存后的入队衔接 |
|
||||||
|
| 下载调度 `app.py` | 频道、分页、下载去重、进度、停止;历史库仍只有指纹到目录映射;状态接口含汇总、日志、输出目录 | 独立发布队列、每篇完成登记、重启对账;不能拿日志当可靠文章队列 |
|
||||||
|
| 导入器 v1.0.0 | `stagePackage()`、`importPrepared()`、附件上传及登记、编辑器填写校验 | 接入取消信号、无弹窗冲突状态、持久恢复;导入前处理长标题 |
|
||||||
|
| `dww11_make_package.py` | 已能生成 v1 Markdown/base64 图片包 | 包大小限制、当前任务按需传输、引用完整性校验 |
|
||||||
|
| 换号扩展 v3.0.4 | 自有账号库、抽号、登录、暂停与持久流程;下一账号按钮已有 ID | 接收统一终止、输出可核对的换号结果,与发文批次计数协调;不能假定网站脚本能直接调用扩展内部路由 |
|
||||||
|
| `auto_news_upload/` | 操作规范和两份 JSON 记录已存在 | 尚无完整自动执行程序或 SQLite 发布任务库 |
|
||||||
|
|
||||||
|
新版排版的“保留来源”指保留识别到的尾注文字;内部会规范化空白并重建为纯文本格式节点,不保证原 HTML、链接或空格逐字不变。自动化应展示提取结果,疑似误识别时暂停。
|
||||||
|
|
||||||
|
旧操作记录中“每次输入来源”和“排版后手工删除重复来源”属于当时演示行为。当前 v1.0.9 的独立排版入口已针对已有来源提供处理,不能再把这一功能整体列为未实现;仍需回归验证。微信 URL 采集入口仍走另一条流程,本项目只接独立排版逻辑。
|
||||||
|
|
||||||
|
下载器测试文件已含不插入发布时间的断言,本次未重跑测试。新下载文件的时间格式已明确,不再列为待确认项;这不授权清理历史稿件。
|
||||||
|
|
||||||
|
## 3. 已有实机证据与尚未解决的问题
|
||||||
|
|
||||||
|
既有演示记录:账号示例用户(UID <已匿名>),理财财经 fid=125,文章“刚靠新电影赚23亿票房,周星驰就卖掉内地影院”,来源“洲洲影视娱评”。用户亲手发表后,网站返回主题 23697411、首帖 PID 238766,记录时显示审核中。
|
||||||
|
|
||||||
|
- 已明确受理 1 篇;已确认公开 0 篇。不得根据旧记录推断现在已审核通过,也不得重发这篇文章。
|
||||||
|
- 导入阶段编辑器有 26 张图片;结果页正文 21 张、附件区 5 张。总数齐全不能证明正文位置和顺序正确。
|
||||||
|
- 附件区图片为 image009.jpg、image014.jpg、image016.jpg、image025.jpg、image026.jpg。需要以附件 ID 与正文引用逐一对照,不只比较文件名或总数量。
|
||||||
|
- 结果正文与待审稿有文字差异;记录未覆盖用户审核期间编辑动作,不能归因于网站转换或用户操作。
|
||||||
|
- 本次未重新读取线上结果页;原演示未验证审核后的访客可见性、超长标题报错、随机数量与换号链路。
|
||||||
|
|
||||||
|
后续准备完成时保存基线快照,用户亲手点击时再捕获最终标题、编辑器正文、图片引用和附件映射,记录内容修订号与提交尝试。持久化可能来不及完成时要能识别缺失证据;不能为补记状态而代替用户重新提交。发布后分别检查受理状态与内容完整性,内容异常只标记待处理,不自动编辑或重发。
|
||||||
|
|
||||||
|
## 4. 目标工作流与架构
|
||||||
|
|
||||||
|
```text
|
||||||
|
用户设置分区、根目录、处理范围、min/max
|
||||||
|
→ 初始化终止按钮与持久批次
|
||||||
|
→ 历史去重对账
|
||||||
|
→ 已完整下载文章校验 / 持续接收新文章
|
||||||
|
→ 单篇领取与文章包准备
|
||||||
|
→ 确认新帖标签、账号、分区
|
||||||
|
→ stagePackage → importPrepared
|
||||||
|
→ 独立一键排版 → 标题/正文/图片/尾注检查
|
||||||
|
→ 等待人工审核与用户亲手点击发表
|
||||||
|
→ 只观察提交结果
|
||||||
|
→ 原子登记受理记录、去重与本轮计数
|
||||||
|
→ 回首页并核对
|
||||||
|
→ 未达目标:下一篇;达到目标且仍有任务:换号并核对
|
||||||
|
→ 队列结束:结束批次
|
||||||
|
```
|
||||||
|
|
||||||
|
建议继续采用“本地 Python 任务服务 + 浏览器执行器 + 网站适配层”,复用导入器与换号扩展。第一步验证油猴特权请求与回环服务通信;若目标浏览器不兼容,再改为扩展桥接,不同时维护多套主链路。
|
||||||
|
|
||||||
|
本地服务仅监听回环地址,使用配对凭据和限定接口;只向网站传当前文章内容,不传下载 API Key 或账号库。不能把现有下载器 `/api/session` 直接当作对任意网页开放的任务授权入口。导入器是 `@grant none`,排版器有特权权限,必须实测页面与脚本隔离上下文间的调用,不能简单假定合并后仍可访问原来的 `window` 对象。
|
||||||
|
|
||||||
|
文件传递优先复用文章包,不依赖自动操作系统目录选择器。原生选择器可作为人工导入备选。同一版块的点击可能打开新标签,任务必须绑定实际标签及导航版本,不能在旧分区页继续执行。
|
||||||
|
|
||||||
|
电脑、本地服务及浏览器需保持运行;没有新增定时无人值守发表需求。发布始终需要用户点击。新任务的下载 Key 仍按现有方式在内存中使用,不默认落盘。
|
||||||
|
|
||||||
|
## 5. 数据接入、格式与来源
|
||||||
|
|
||||||
|
### 完整落盘与元数据
|
||||||
|
|
||||||
|
复用下载器现有保存及去重逻辑,新增每篇成功结果清单,至少保存稳定身份候选、原始标题、来源、源 URL、目录、Markdown 路径、图片清单和校验值。新发布记录写入失败时可通过清单对账补录,不应为回滚发布登记而删除完整文章。
|
||||||
|
|
||||||
|
当前保存器先创建最终目录,再逐个移动文件,扫描到目录并不代表下载结束。新下载采用成功回调/完成标记与清单协调入队;旧目录在确认非保存中且引用完整后显式导入。下载历史库失败时原下载器自有清理行为,与未来发布登记失败处理分开。
|
||||||
|
|
||||||
|
### 标题
|
||||||
|
|
||||||
|
保留 `original_title`,派生 `submitted_title`。超过 80 字符时按允许的计数预算截取前缀并追加字面量 `...`,不能截断 emoji 代理对;站点实际按码点、UTF-16 或字节计数仍需实测。当前页面曾读到 maxLength=-1,“还可输入80个字符”也未随填入更新,不能作为可靠校验。
|
||||||
|
|
||||||
|
截断必须发生在导入器 `compile()` 之前。现有 `preparePackage()` 已接受 `data.title`,可以由适配层提供派生标题,不改原 Markdown;现有打包器本身尚未生成这个覆盖值。显示新旧标题并进入人工审核。若人工提交后收到明确超限拒绝,修正后仍由用户再次点击;结果不明时不改稿重投。
|
||||||
|
|
||||||
|
### 来源与正文
|
||||||
|
|
||||||
|
复用 v1.0.9 的 `existingSource()` 与 `formatBody()`,保留 4 号黑色正文、1 号银色尾注、图文间隔及图片适配。识别末尾来源时保留整段来源/版权文字,不只抽出名称再拼回默认句子。没有来源则在页面内要求补充,不能让原生 prompt 阻挡终止按钮。
|
||||||
|
|
||||||
|
需要验证:标准下载尾注、多行版权说明、重复尾注、正文中出现“来源”、来源后有图片、嵌套 FONT、纯文本换行、编辑器来回切换、重复排版。不得全局删除正文中的来源文字;原稿本身已有多条来源时应提示核对,而非宣称新函数能语义去重。
|
||||||
|
|
||||||
|
正文清理只沿用已确认规则;不自动删除历史发布时间、“打开网易新闻 查看精彩图片”等文字。下载器新生成格式与旧文件格式并存,文章身份不能仅靠整个 Markdown 的精确哈希判断。
|
||||||
|
|
||||||
|
### 图片与编辑器
|
||||||
|
|
||||||
|
下载器支持 WebP/AVIF/SVG,打包器和导入器仍只支持 JPG/JPEG/PNG/GIF。先在入队前拦截不兼容格式,不能等打包时静默漏掉;如后续增加转码,生成派生副本并重写引用,保留原文件。上传大小同时遵守导入器 20 MB 和网站当前限制。
|
||||||
|
|
||||||
|
校验顺序为:本地引用完整 → 上传附件返回编号 → 网站附件登记 → BBCode/可视化图片关联 → 排版前后附件引用保持 → 人工提交时最终快照 → 结果页正文与附件区对照。现有缓存仅在页面内有效,持久复用必须绑定账号与上传上下文并重新验证。
|
||||||
|
|
||||||
|
遇到旧稿恢复提示、非空编辑器或上传期间人工修改,暂停并保留现场。导入器现有 confirm 弹窗需要转换为页面内状态;不能自动确认覆盖。
|
||||||
|
|
||||||
|
## 6. 持久状态、发布去重与观察
|
||||||
|
|
||||||
|
独立 SQLite 任务库与下载历史库分开。启动前先迁入两份现有 JSON 的已受理记录,保证主题 23697411 不再入队;保留原 JSON 作为证据。再核对原有发布记录或本人历史帖子列表。标题相同只标记疑似重复,不能据此确认同一正文。完整历史对账尚未完成,不能把这一个样本当全量历史。
|
||||||
|
|
||||||
|
稳定身份优先关联原始文章 ID/规范化源 URL,同时保存正文与图片指纹、文件版本哈希和身份别名。当前下载历史只有指纹与目录,原始 URL/来源并未形成完整持久清单;历史稿无法恢复的信息要标为未知,不能猜造。跨账号、跨版块去重,改名、标题截断、删除后重新下载均不改变已受理身份。
|
||||||
|
|
||||||
|
建议记录:article_key、身份别名、source_url/source_id、内容及图片指纹、原始/提交标题、来源尾注、目录、账号 UID、fid、附件映射、排版版本、内容修订号、任务状态、批次 ID、领取租约/执行版本、尝试 ID、用户点击时间、最终快照、帖子 ID/URL、受理/审核/公开状态和错误。
|
||||||
|
|
||||||
|
```text
|
||||||
|
downloaded → queued → importing → formatting → awaiting_user_review
|
||||||
|
→ user_submitting → published / accepted_pending_moderation
|
||||||
|
```
|
||||||
|
|
||||||
|
异常分支为 `needs_attention`、`submit_unknown`、`rejected`、`stopped`。其中 `published` 仅在有公开发表证据时使用;提交意图或点击事件不是网站受理证据。人工修改后记录新修订,继续等待人工审核。
|
||||||
|
|
||||||
|
- 自动化只监听用户操作及网站结果,不主动触发提交。点击前预登记候选尝试,点击时关联最终快照;即使快照写入失败,也不能自动补交。
|
||||||
|
- 结果需综合帖子 ID/URL、作者、分区、标题和受理/审核提示;跳转到首页、HTTP 200 或按钮变灰均不能单独证明成功。
|
||||||
|
- 以“站点+帖子 ID”唯一键登记受理记录,并在同一事务更新去重与批次计数。重复打开、刷新及后续审核通过不重复加一。
|
||||||
|
- 审核中计入明确受理数一次,公开数量另记;驳回不自动补发。结果页内容异常单独记录,不撤销受理去重。
|
||||||
|
- 网络中断、关闭标签或证据不足进入 `submit_unknown`,先对账;租约过期也不能重投未决文章。
|
||||||
|
- 多标签单篇互斥;刷新或服务重启先显示恢复状态,等待用户明确继续。已停止批次不得自动恢复。
|
||||||
|
|
||||||
|
## 7. 账号目标与统一终止
|
||||||
|
|
||||||
|
用户填写整数 `1 ≤ min ≤ max`。首次建立当前账号批次、以及确认换号成功后,从闭区间均匀抽取目标 n 并持久化。每账号批次只抽一次;暂停、刷新和重启不重抽。没有 min/max 时不能擅自设值或启动自动换号。
|
||||||
|
|
||||||
|
本轮只计明确受理的唯一文章,导入、排版、等待审核、失败和结果不明不计。达到 n 后,先完成记账,确认没有未决提交,回首页并核对,再触发现有“登陆下一帐号”。核对扩展流程结束和实际账号 UID/用户名;无法确认时暂停,不连续点换号。账号池只有一个可用账号等导致 UID 未变化时不能假装换号成功,应显示原因等待处理。
|
||||||
|
|
||||||
|
队列空时结束,不为凑数重发,也不无意义换号。账号手动变化、验证码、限制或登录异常时暂停,不带着旧附件上下文继续。账号密码继续由现有扩展管理。
|
||||||
|
|
||||||
|
统一终止必须覆盖首页、分区页、新帖页、结果页和换号页面。点击先持久写入 stopped 并更新执行版本;立即禁止新领取、下一张上传、排版写入、跳转和换号,取消可取消请求。所有异步返回重新检查停止状态,过期回调不得继续。
|
||||||
|
|
||||||
|
**现有组件的停止并不贯通:** 下载器停止是等待当前请求/保存结束;导入器没有对外取消接口;换号扩展有独立流程;排版仍使用模态弹窗。仅在外层增加按钮不能满足全流程终止。需要给这些模块增加取消/暂停衔接,尤其要在扩展开始退出/登录前以及后续各步骤检查共享停止状态。不能把扩展内部消息接口当作已公开的油猴 API。
|
||||||
|
|
||||||
|
停止不撤回已经发出的请求,允许被动核对其最终结果,但不继续业务动作。原演示使用浏览器控制条停止的例外不延续至正式实现。
|
||||||
|
|
||||||
|
## 8. 页面适配依据与剩余信息
|
||||||
|
|
||||||
|
| 环节 | 已有依据 | 仍需核对 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 分区入口 | `/special/post/`;演示选择后新开标签 | 用户实际目标分区及各区必填项,不固定 fid=125 |
|
||||||
|
| 当前账号 | `#mumucms_username`,用户名/用户链接编号 | 登录失效与换号成功的稳定判定 |
|
||||||
|
| 标题/正文 | `#subject`、`#e_textarea`、`#e_iframe`、`#e_switchercheck` | 当前网站编辑器同步与标题计数 |
|
||||||
|
| 导入 | `window.DWW11Importer`,面板在 Shadow DOM | 实际浏览器中程序接口与停止信号 |
|
||||||
|
| 排版 | `#dww11-format-oneclick` | v1.0.9 来源识别与发布后往返结果;目前没有公开排版 API |
|
||||||
|
| 发表 | `#postsubmit`、`#postform` | 只观察人工操作;失败、超限、审核及成功提示适配 |
|
||||||
|
| 受理结果 | `/thread/23697411.html`;样本首帖 `#postmessage_238766` | 从当前结果提取动态帖子 ID/PID,不硬编码样本 |
|
||||||
|
| 换号 | `#dww11-extension-next` | 扩展完成/暂停状态及全局终止联动 |
|
||||||
|
| 历史对账 | “修改/删除信息”进入本人帖子列表 | 列表 URL、分页、审核项与重复证据 |
|
||||||
|
|
||||||
|
无需重新提供已记录的完整单篇演示。下一批所需输入主要是:实际分区与文章范围、min/max、其他版块必填项、历史帖子入口及异常提示样本。标题计数、5 张图片落到附件区的原因、审核后的公开状态需要针对性验证。不需要密码、Cookie 或 API Key 值进入文档。
|
||||||
|
|
||||||
|
## 9. 实施顺序与验收
|
||||||
|
|
||||||
|
| 阶段 | 工作 | 完成条件 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1:基础状态与停止 | 独立任务库、现有 JSON 迁入、历史对账、统一终止状态和常驻按钮;确定各组件取消协议 | 已受理样本不可重入队;跨标签互斥;刷新不自动运行;停止后无后续动作 |
|
||||||
|
| 2:文章衔接 | 完整落盘清单、文章身份、包传输、图片检查、导入前标题派生、本地配对通信 | 新旧格式均可处理;半成品不入队;长标题不改原稿;缺图和不兼容格式明确报错 |
|
||||||
|
| 3:排版与人工审核 | 封装 v1.0.9 独立排版接口,替换模态提示,接入导入取消和编辑冲突检查 | 尾注文字保留、标准来源只出现一次;反复排版及编辑器转换无累积问题;附件引用与顺序不变 |
|
||||||
|
| 4:人工提交观察 | 提交快照、动态结果解析、审核/公开区分、唯一受理计数、正文与附件核对 | 不经用户点击永不发表;用户修改后重新审核;刷新不重复计数;结果不明不重投;定位图片落附件区问题 |
|
||||||
|
| 5:批次与换号 | min/max 持久目标、记账后回首页、复用登录流程、账号核对、终止联动 | 每批仅抽一次目标;达到目标且队列非空才换号;异常不连续换号;无旧账号附件串用 |
|
||||||
|
| 6:小批实机验收 | 覆盖故障恢复与多个页面场景 | 下载/上传/排版/待审/结果/换号各阶段终止均有效;断网、重启、多标签和人工改稿状态正确 |
|
||||||
|
|
||||||
|
重点回归场景:普通/多图/重复图片、空标题节点、多行尾注、来源后图片、重复排版、长标题与 emoji、非空编辑器、旧稿恢复、部分上传失败、上传超时、用户提交后响应丢失、审核中、驳回和换号中终止。先模拟验证,再做用户亲手提交的少量实机验收;不以已受理样本重发来测试。
|
||||||
|
|
||||||
|
## 10. 目录与本次交付范围
|
||||||
|
|
||||||
|
新实现仍放在 `auto_news_upload/`,后续规划为 `service/`、`browser/`、`tests/`、`data/` 和启动说明;当前这些实现尚未建立。保留原下载目录、原历史库及现有演示记录。复用源码时记录版本,明确安装时启用的脚本组合,避免重复工具争用编辑器。
|
||||||
|
|
||||||
|
本次仅重写本实施计划,未修改业务脚本和历史记录。历史操作规范用于追溯当时实机行为;当前组件能力以本轮核对的源码为准,人工发表、去重、终止和换号规则继续遵循操作规范。
|
||||||
+150
@@ -0,0 +1,150 @@
|
|||||||
|
# dww11 自动化流程检查与 Git 文件清单
|
||||||
|
|
||||||
|
检查日期:2026-10-31(北京时间)。本轮仅检查本地说明、源码和离线测试;没有访问网站、消耗下载 API 额度、登录、上传或发表,最初检查时未执行 Git 操作。用户随后已确认仓库范围并授权本地初始化和首次提交;下面保留检查发现,并补充本次整理结果。
|
||||||
|
|
||||||
|
## 一、检查结论
|
||||||
|
|
||||||
|
当前是可以人工衔接的组件集合,尚不是已实现的端到端自动流程。
|
||||||
|
|
||||||
|
```text
|
||||||
|
探数 Key → 下载 Markdown 和图片
|
||||||
|
→ 手工选择文章目录,或运行打包器生成文章包
|
||||||
|
→ 已登录的论坛新帖页面导入并上传图片
|
||||||
|
→ 点击独立“一键排版”并核对标题、正文、配图和来源
|
||||||
|
→ 用户审核并亲手点击发表
|
||||||
|
→ 核对受理/审核结果,登记去重,再决定下一篇或换号
|
||||||
|
```
|
||||||
|
|
||||||
|
发表与再次尝试发表均由用户亲手点击,这是现有流程规范要求。下载器、打包器、导入器、排版器和换号扩展之间没有完整的任务调度连接。
|
||||||
|
|
||||||
|
## 二、逐环节检查
|
||||||
|
|
||||||
|
| 环节 | 主要文件 | 当前能力 | 缺口或注意事项 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 下载 | `tanshuapi_news_downloader/app.py`、`article_writer.py`、`static/` | 频道、分页、正文图片落盘、下载历史去重、进度和停止 | 下载历史仅保存指纹与目录,没有供发布队列可靠消费的完整文章清单和完成标记;停止需要等待当前请求/保存结束 |
|
||||||
|
| 打包 | `dww11_make_package.py` | 将目录转换为 v1 Markdown/base64 图片包;可直接作为 Python 函数调用 | 不会自动把文章送进浏览器;仅收集 JPG/JPEG/PNG/GIF,没有完整的 Markdown 引用预检或整包大小上限 |
|
||||||
|
| 导入 | `dww11-article-importer.user.js` | 选目录或文章包、预览、按正文引用上传、附件登记、填写校验与失败回滚 | 标题超过 80 个 Unicode 码点直接拒绝;没有对外取消接口;非空编辑器确认仍使用模态弹窗 |
|
||||||
|
| 排版 | `dww11-wechat-oneclick.user.js` v1.0.9 | 独立“一键排版”,识别文末来源并重建尾注;无来源时要求补充 | 没有公开可等待/取消的排版接口,仍有 prompt/alert;复杂尾注、重复执行和发布后的图片位置仍需浏览器回归 |
|
||||||
|
| 发布 | `auto_news_upload/` 两份流程文档和两份记录 JSON | 已有单篇人工操作的历史记录 | 本目录没有完整发布执行程序、持久任务库、自动受理计数、全量历史对账或统一终止功能 |
|
||||||
|
| 换号 | `dww11-extension/` v3.0.4 | CSV 导入、扩展 IndexedDB 帐号库、抽号与登录、暂停恢复 | 未与文章任务的 min/max、受理计数和统一终止联动;扩展内部路由不能直接视作油猴接口 |
|
||||||
|
|
||||||
|
### 优先处理的问题
|
||||||
|
|
||||||
|
1. **图片格式衔接不一致。** 下载器可保存 WebP/AVIF/SVG,打包器和导入器只支持 JPG/JPEG/PNG/GIF。打包器会跳过不支持格式,导入时可能报缺图;应在打包/入队前给出明确错误,或另外实现保留原文件的转码。
|
||||||
|
2. **长标题衔接未实现。** 规范要求保留原题并派生截断标题,现导入器却在编译时直接拒绝超过 80 字的标题。需在导入前生成并展示提交标题,站点真实计数方式仍需实测。
|
||||||
|
3. **缺少可靠调度和统一终止。** 下载完成不等于可以发布;目录存在也不等于完整保存。现有模块停止行为不贯通,不能直接启动无人看管的批量链路。
|
||||||
|
4. **发布去重仍是手工记录。** 下载去重不能替代跨帐号、跨版块的发布去重。需要持久队列、未决结果核对、唯一受理登记及恢复逻辑。
|
||||||
|
5. **历史样本的图片位置有异常。** 原记录显示 26 张图片中正文 21 张、附件区 5 张,总数相同不能证明位置正确。应按附件 ID 对照最终待提交快照与结果页。
|
||||||
|
6. **旧路径已整理。** 原导入测试与打包说明引用了搬迁前目录;首次提交准备时改用 `tests/fixtures/article/` 的自造 Markdown 和 PNG,消除 legacy 依赖。
|
||||||
|
7. **扩展测试资料缺失。** 扩展 README 提到 `tests/core.test.mjs`、`tests/browser-integration.mjs`、`tests/browser-restart.mjs` 和 `测试说明.md`,当前项目中未找到,不能把文档中的历史测试声明当成本轮复验结果。
|
||||||
|
|
||||||
|
历史记录中的主题 23697411 已被网站受理并显示审核中,必须保留其本地去重证据。本轮没有核实现在是否已经公开。
|
||||||
|
|
||||||
|
## 三、运行时需要帐号或凭据的文件
|
||||||
|
|
||||||
|
下表区分“程序需要凭据才能运行”和“源码里保存真实凭据”。程序需要登录不代表源码不能进入 Git。
|
||||||
|
|
||||||
|
| 文件 | 所需帐号/凭据 | 使用和保存方式 | Git 建议 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `tanshuapi_news_downloader/app.py` | 探数 API Key | 调用新闻 API;Key 从页面传入,在运行内存使用 | 源码可纳入;不写入真实 Key |
|
||||||
|
| `tanshuapi_news_downloader/static/app.js`、`static/index.html` | 用户填写探数 API Key | 页面输入和内存变量,经本机会话接口传给服务;源码未配置固定 Key | 可纳入 |
|
||||||
|
| `dww11-article-importer.user.js` | 已登录的大湾网帐号及页面上传凭据 | 从当前页面读取上传参数,使用当前登录会话上传图片;无须写入密码 | 可纳入源码 |
|
||||||
|
| `dww11-wechat-oneclick.user.js` | 发布页面的登录会话 | 独立排版入口在当前编辑器操作;微信 URL 入口属于另一条采集路径 | 可纳入源码 |
|
||||||
|
| `dww11-extension/manager.js`、`manager.html`、`core.js` | 帐号 CSV | 选择、解析、确认导入帐号和密码 | 可纳入程序及虚拟模板,排除真实 CSV |
|
||||||
|
| `dww11-extension/db.js`、`service.js` | 扩展帐号库 | 明文密码保存在浏览器扩展来源的 IndexedDB;列表返回时去除密码 | 可纳入程序;排除浏览器资料及数据库备份 |
|
||||||
|
| `dww11-extension/background.js`、`content.js` | 当前抽取的帐号和密码 | 扩展消息传递,并填入目标网站登录表单 | 可纳入程序 |
|
||||||
|
| `today-jobs.user.js` | 可访问招聘管理列表的登录帐号/权限 | 使用当前页面和同源会话读取职位 | 辅助功能,按需纳入 |
|
||||||
|
| `dww-today-threads.user.js` | 论坛后台管理权限与页面 formhash | 使用当前管理会话执行统计查询 | 辅助功能,按需纳入 |
|
||||||
|
| `legacy/acc_switch_lite_python_legacy/server.py`、两个 lite 目录的 `import_excel.py` 与换号油猴脚本 | 旧版帐号文件、Excel 和网站登录 | 旧版实现;与当前扩展分开 | 不属于当前主链路;归档时排除真实数据 |
|
||||||
|
| `legacy/acc_switch_heavy_sqlite_legacy/init_db.py`、`import_excel.py` | 旧版帐号 Excel/数据库 | 可生成并导入保存密码的 `accounts.db` | 不属于当前主链路;数据库不纳入 |
|
||||||
|
|
||||||
|
`dww11_make_package.py` 仅处理本地文件,不需要帐号、API Key 或网站 Cookie。
|
||||||
|
|
||||||
|
## 四、已经存在的帐号数据文件:排除真实版本
|
||||||
|
|
||||||
|
| 文件 | 本轮确认 | 建议 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `账号密码-去重可导入.csv` | 表头外 1152 条数据行、3 列;为真实帐号导入文件 | 本地保留,不纳入 Git |
|
||||||
|
| `legacy/acc_switch_lite/accounts.json` | 169 条帐号记录,169 个非空 password 字段 | 本地保留,不纳入 Git |
|
||||||
|
| `legacy/acc_switch_lite_python_legacy/accounts.json` | 169 条帐号记录,169 个非空 password 字段 | 本地保留,不纳入 Git |
|
||||||
|
|
||||||
|
以上仅统计结构和数量,不展示用户名、密码或密钥。没有执行帐号有效性验证,也没有核对 CSV 行与 JSON 记录之间的对应关系。
|
||||||
|
|
||||||
|
当前扩展运行帐号库在浏览器内,不在 `dww11-extension/` 源码目录。不要把浏览器配置目录、Cookie 导出文件或 IndexedDB 备份作为源码提交。`example.csv` 是示例模板,需保持为虚拟凭据。
|
||||||
|
|
||||||
|
## 五、拟纳入 Git 的文件清单
|
||||||
|
|
||||||
|
### A. 当前主链路源码与说明
|
||||||
|
|
||||||
|
```text
|
||||||
|
dww11_make_package.py
|
||||||
|
dww11-article-importer.user.js
|
||||||
|
dww11-wechat-oneclick.user.js
|
||||||
|
本地文章导入脚本使用说明.md
|
||||||
|
tanshuapi_news_downloader/app.py
|
||||||
|
tanshuapi_news_downloader/article_writer.py
|
||||||
|
tanshuapi_news_downloader/requirements.txt
|
||||||
|
tanshuapi_news_downloader/start.command
|
||||||
|
tanshuapi_news_downloader/README.md
|
||||||
|
tanshuapi_news_downloader/.gitignore
|
||||||
|
tanshuapi_news_downloader/static/index.html
|
||||||
|
tanshuapi_news_downloader/static/app.js
|
||||||
|
tanshuapi_news_downloader/static/style.css
|
||||||
|
tanshuapi_news_downloader/tests/test_downloader.py
|
||||||
|
dww11-extension/manifest.json
|
||||||
|
dww11-extension/background.js
|
||||||
|
dww11-extension/content.js
|
||||||
|
dww11-extension/core.js
|
||||||
|
dww11-extension/db.js
|
||||||
|
dww11-extension/service.js
|
||||||
|
dww11-extension/manager.html
|
||||||
|
dww11-extension/manager.js
|
||||||
|
dww11-extension/manager.css
|
||||||
|
dww11-extension/example.csv
|
||||||
|
dww11-extension/README.md
|
||||||
|
auto_news_upload/实施计划.md
|
||||||
|
auto_news_upload/发布流程操作记录与自动化规范.md
|
||||||
|
auto_news_upload/流程检查与Git文件清单.md
|
||||||
|
```
|
||||||
|
|
||||||
|
两份流程文档的帐号名称、UID 和电脑绝对路径已匿名化,原版保存在被排除的 `auto_news_upload/local-records/`。历史帖子编号与链接保留为受理证据说明。本文不含密码,但也引用了本地敏感文件名和数量,可按仓库公开范围进一步精简。
|
||||||
|
|
||||||
|
### B. 修复或检查后再决定
|
||||||
|
|
||||||
|
- `test-dww11-importer.cjs`:已改用仓库内自造测试稿与三张 PNG,一并纳入首次提交。
|
||||||
|
- `auto_news_upload/演示任务记录.json`、`发布去重记录.json`:属于真实运行状态。建议保留本地,另提供匿名结构模板;不因 Git 排除而删除原去重记录。
|
||||||
|
- `dww11-extension-3.0.4.zip`:本轮逐文件比对,与当前扩展目录的 11 个文件一致。可作为发行附件;日常优先跟踪源码,避免双份维护。
|
||||||
|
- `today-jobs.user.js`、`dww-today-threads.user.js`:辅助工具,不是四步主流程的必需文件,按你的范围选择。
|
||||||
|
- `.wechat-script-test/`:本地排版验证页、源码副本、截图和结果,需整理为可复现测试后再选择纳入。
|
||||||
|
- `legacy/`:默认不进入主链路首批提交;若保留历史实现,必须逐项排除帐号数据、虚拟环境、缓存及下载文章。
|
||||||
|
- `credits-video/`:滚动字幕视频工具,与本次流程无直接关系,默认不纳入本批。
|
||||||
|
|
||||||
|
### C. 建议排除的本地产物
|
||||||
|
|
||||||
|
真实帐号 CSV/JSON、下载文章及图片、下载历史数据库、真实发布记录与截图、文章包 `*.dww11.json`、浏览器配置和 Cookie、`.venv/`、`__pycache__/`、`*.pyc`、`.DS_Store`、视频输出和临时诊断文件。
|
||||||
|
|
||||||
|
首次提交准备时已增加根目录 `.gitignore`,覆盖真实帐号、运行记录、下载目录及本批范围之外的组件;按白名单添加源码和说明。
|
||||||
|
|
||||||
|
## 六、本轮验证结果
|
||||||
|
|
||||||
|
| 检查 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| Git 状态 | 当前目录不是 Git 仓库;本轮没有新建仓库 |
|
||||||
|
| 原始导入测试 `node --test test-dww11-importer.cjs` | 8 项中 3 项通过、5 项因旧素材目录不存在失败 |
|
||||||
|
| 只在内存中修正素材路径后执行同一测试 | 8/8 通过,覆盖打包与直接导入一致、缺图/长标题拦截、上传失败保留原文、富文本丢图回滚、无发表请求;磁盘测试文件未修改 |
|
||||||
|
| 下载器单元测试 | 默认 Python 3.15 与 Python 3.12 均因缺少 `requests` 无法加载测试;未安装依赖,不能认定通过或认定下载逻辑失败 |
|
||||||
|
| JavaScript 语法 | 6 个扩展 JS 按 ES 模块检查通过;4 个根目录油猴脚本检查通过 |
|
||||||
|
| Python 语法 | 打包器、下载器 app 与保存器均通过语法解析 |
|
||||||
|
| 扩展 ZIP | 11 个文件与源码目录逐一一致 |
|
||||||
|
| 在线验收 | 本轮未执行;历史文档中的单篇演示不等于当前版本全链路验收 |
|
||||||
|
|
||||||
|
## 七、首次提交整理范围
|
||||||
|
|
||||||
|
用户已授权新建本地仓库并完成 initial commit,范围为总流程文档、当前换号扩展、新闻下载器、文章打包/导入/排版及相关说明与测试。
|
||||||
|
|
||||||
|
- 增加根目录 README 与 `.gitignore`。
|
||||||
|
- 导入测试改用项目自有测试稿及三张程序生成的 PNG,修正打包说明路径。
|
||||||
|
- 原始流程说明备份到本地排除目录,再匿名化仓库文档中的帐号名称、UID 与电脑路径。
|
||||||
|
- 真实帐号、下载历史、演示 JSON、截图及原始说明均保留本地,不纳入提交。
|
||||||
|
- 下载器测试仍需先安装依赖,扩展缺失的测试文件尚未补齐;不以首次提交表示全流程已经完成。
|
||||||
|
- 本次只建立本地版本历史,未配置远程仓库或推送。
|
||||||
Reference in new issue
Block a user