diff --git a/CLAUDE.md b/CLAUDE.md index f75bbae..8e2ad32 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -222,7 +222,51 @@ servers: - 控速依据是 **GPU 温度**(DCGM),目标机的机箱风扇,写 `ipmitool raw 0x3a 0x01` - 运行时状态(模式 / 管控 GPU / 分配关系 / 审计)**全部落 SQLite**,配置文件只是首次运行的种子 -**实时数据源:三个外部 exporter(自备组件,不随仓库提供,装法不限)** +### 控制台模块地图(`app/`) + +| 模块 | 职责 | +|---|---| +| `main.py` | 入口:单进程 = 控制回路 + API + 静态托管;`STATIC_DIR` 来自 `runtime.py` | +| `api.py` | REST + WebSocket(`/api/status`、`/api/curve/preview`、`/ws` 等) | +| `controller.py` | 控制回路(默认 15s 一轮,可配) | +| `curve.py` | 分段曲线 + 滞回(升温立即生效,降温须跌出滞回带) | +| `ipmi.py` | `raw 0x3a 0x01` 命令族与占空比编码 | +| `sensors.py` | DCGM / ipmi_exporter / node_exporter / nvidia-smi 兜底 / Prometheus 历史 | +| `safety.py` | 安全护栏(退出回退 BMC 自动档 + atexit + 心跳) | +| `store.py` | SQLite(`data/fan-console.db`,唯一权威数据源) | +| `config.py` | pydantic 配置模型;`load_config` 缺文件时优雅降级为默认值 | +| `runtime.py` | **资源定位中枢**:源码运行根 = `app/` 包目录;PyInstaller 冻结运行根 = exe 所在目录。config / DB / static / heartbeat 四处路径全部从这里取——**改路径相关代码时只动这里,别在各自模块里再写 `__file__` 推导** | + +### 打包与 CI/CD + +**CI**(`.github/workflows/ci.yml`,推 main / PR / 打 `v*` tag 触发): + +1. `test-and-build`:后端单测(Python 3.13)→ 前端 `npm ci && npm run build`(含 vue-tsc 全量类型检查)→ 打部署包 `gpu-fan-console-app.tgz`(成员路径 `app/...`,排除 `app/data`)挂 Artifacts +2. `build-console` / `build-legacy`(非 PR 才跑):PyInstaller 双平台可执行;分发压缩包由 `packaging/build_console.py` 内置生成(zipfile/tarfile,**不要**在 workflow 里用 tar/PowerShell 打——Windows runner 的 cp1252 控制台编不了中文输出,打包脚本必须 `sys.stdout.reconfigure(encoding="utf-8")`) +3. `release`(打 tag 才跑):`gh release create` 自动发版附全部产物;**必须显式 `--repo "$GITHUB_REPOSITORY"`**(该 job 无 checkout,gh 推断不出仓库) + +**冻结(PyInstaller)相关**: + +- 打包入口 `entry_console.py`(控制台 onedir)、`packaging/build_console.py` / `packaging/build_legacy.py`(旧脚本 onefile) +- 冻结后**可变资源必须落 exe 旁**(`_MEIPASS` 每次启动清空):config.yaml / `data/` / heartbeat 由 `runtime.py` 定位到 exe 旁;旧脚本 `fancontroller(.once).py` 的 `_app_directory()` 同理 +- 本机已实测:控制台 exe(API/静态页/SQLite 落位)、旧脚本 exe(缺配置报错、空配置正常退出) + +### 部署 + +- 部署方式、踩坑(tar 路径对齐 / upload 分片 base64 / 远端 rm 黑名单)、回滚、验证清单 → **[`DEPLOY.md`](./DEPLOY.md)** +- 一键脚本 `./deploy.sh`:**`CONN=` 必填**,可选 `REMOTE_DIR` / `SERVICE` / `URL`; + 脚本打包时排除 `app/data` 与 `app/config.yaml` +- 服务器直连 GitHub 可用时,更快的更新方式:直接 `curl -L` Release 页的 + `gpu-fan-console-app.tgz`,免去本地构建与分片上传(v2.0 实测可行) +- ⚠️ 升级已配置的机器时,解压必须 `--exclude='app/config.yaml'`——仓库里的 + config.yaml 是**模板**(示例地址),覆盖会毁掉现场配置 + +### 仓库约定 + +- **公开仓库,严禁出现真实内网 IP / 主机名 / 凭据**。示例地址一律用 RFC 5737 + 文档段(192.0.2.0/24 等);写文档、注释、测试样例时同样遵守 +- `config.yaml` 按模板维护;`README_EN.md` 与中文版结构对齐 +- 实时数据源:三个外部 exporter(自备组件,装法不限): | 数据 | exporter | 项目地址 | 指标 | 默认端口 | |---|---|---|---|---| @@ -234,5 +278,3 @@ servers: (仅 `/api/history` 用)可配;DCGM 不可用时 `sensors.py` 降级 `nvidia-smi` 兜底。 ipmi_exporter in-band 读 `/dev/ipmi0` 需要 root。安装/自检详见 [`DEPLOY.md`](./DEPLOY.md) 2.1 节。 -- 部署方式、踩坑、回滚、验证清单 → **见 [`DEPLOY.md`](./DEPLOY.md)**,一键脚本 `./deploy.sh` - diff --git a/DEPLOY.md b/DEPLOY.md index 3d25161..2feeaac 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -11,8 +11,8 @@ ```bash # 在项目根目录(Windows Git Bash) -./deploy.sh # 全量:后端 + 前端,自动重启服务并自检 -./deploy.sh --static-only # 只更新前端(不改后端、不重启服务) +CONN=<连接名> ./deploy.sh # 全量:后端 + 前端,自动重启服务并自检 +CONN=<连接名> ./deploy.sh --static-only # 只更新前端(不改后端、不重启服务) ``` 下面是人话版说明,出问题时看这里。 @@ -139,7 +139,7 @@ curl -s http://192.0.2.10:8765/api/status | head -c 300 后端没动时不用重启进程 —— `StaticFiles` 每次请求读盘,`index.html` 也不会被缓存: ```bash -./deploy.sh --static-only +CONN=<连接名> ./deploy.sh --static-only ``` > ⚠️ **别手动只拷 `index-xxxx.js`**。文件名带**内容 hash**,源码改一行文件名就变。 @@ -226,7 +226,7 @@ find -type f ! -name '新文件1' ! -name '新文件2' -delete # 代码回滚 git log --oneline -5 git checkout <上一个好提交> -./deploy.sh +CONN=<连接名> ./deploy.sh # 数据回滚(SQLite 有备份时) agentsshcli exec <连接名> "systemctl stop gpu-fan-console" diff --git a/README.md b/README.md index 1135a10..29c76ac 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -[English Readme](./README_EN.md)(英文版目前只覆盖旧版脚本,尚未同步新控制台) +[English Readme](./README_EN.md) # python-ipmitool @@ -119,8 +119,8 @@ python -m unittest discover -s app/tests -t . -v **日常更新一条命令**: ```bash -./deploy.sh # 全量:后端 + 前端,自动重启服务并自检 -./deploy.sh --static-only # 只更新前端(不改后端、不重启服务) +CONN=<连接名> ./deploy.sh # 全量:后端 + 前端,自动重启服务并自检 +CONN=<连接名> ./deploy.sh --static-only # 只更新前端(不改后端、不重启服务) ``` 首次部署、踩坑记录、回滚与验证清单见 **[DEPLOY.md](./DEPLOY.md)**。 @@ -246,7 +246,7 @@ Linux 长期运行建议配置为 systemd 服务(`/etc/systemd/system/fancontr - [DEPLOY.md](./DEPLOY.md) —— GPU 风扇控制台部署手册(一键脚本、踩坑、回滚、验证清单) - [CLAUDE.md](./CLAUDE.md) —— 仓库架构说明(新旧两套的实现细节、加新机型的方法) -- [README_EN.md](./README_EN.md) —— 英文说明(仅旧版脚本) +- [README_EN.md](./README_EN.md) —— 英文说明 ## 贡献与反馈 diff --git a/README_EN.md b/README_EN.md index ce884eb..cd21433 100644 --- a/README_EN.md +++ b/README_EN.md @@ -1,213 +1,232 @@ [中文文档](./README.md) -# Python IPMI Fan Controller - -A Python script to monitor server CPU temperature via IPMI and adjust fan speeds according to predefined temperature ranges. - -## Cross-Platform Support (Windows & Linux)! - -## Compatible Servers - -The following models have been tested and are confirmed to work. More models are pending testing; contributions are welcome! +# python-ipmitool + +Server fan control via IPMI. The repository contains two parts: + +| Component | Location | Status | Description | +|---|---|---|---| +| **GPU Fan Console** | `app/` + `frontend/` | Current main project | Web console that regulates chassis fans in a closed loop by GPU temperature (FastAPI + Vue3) | +| **Fan Control Scripts** | `fancontroller.py` etc. | Legacy, still usable | Command-line scripts, CPU-temperature-based fan control (Dell 730 etc.) | + +## GPU Fan Console + +### What it does + +Target platform: **ASRock Rack EPYCD8** (BMC firmware 2.20). Passive cards like the +Tesla T10 rely entirely on chassis fans for cooling, so the console reads GPU +temperature, computes a duty cycle from an editable step curve with hysteresis, +and writes it to the BMC via `ipmitool raw 0x3a 0x01`. + +- **Data sources**: GPU temperature from a local DCGM exporter (`:9400`), fan speed + from ipmi_exporter (`:9290`), CPU temperature from node_exporter (`:9100`, requires + `--collector.hwmon`) — all read directly from the exporters; historical trends go + through Prometheus. Falls back to `nvidia-smi` when DCGM is unavailable +- **Control curve**: piecewise curve + hysteresis band (rising temperature applies + immediately; falling temperature must leave the hysteresis band before downshift, + avoiding the "helicopter effect"). Editable in the UI with a **live preview** + (`/api/curve/preview` runs the real algorithm) +- **Three modes**: auto (curve) / manual (fixed duty from the UI) / BMC auto +- **GPU ↔ fan-slot assignments**: each fan slot is bound to a GPU; the highest + temperature of the bound GPU drives the speed. Unassigned slots go back to the BMC +- **State lives in SQLite**: mode, curve, assignments and audit log all persist in + `app/data/fan-console.db`, the single source of truth; `app/config.yaml` is only + a first-run seed +- **Safety net (three layers)**: + 1. On shutdown / crash / SIGTERM, `SafetyGuard` hands all managed fan slots back to BMC auto + 2. In-process `atexit` second layer + 3. **Independent heartbeat watchdog** (systemd timer, every 2 min): if the heartbeat + expires it force-writes `8×0x00` — covering even SIGKILL / power loss +- **Dashboard**: GPU cards (temperature / SM clock), fan speeds, separate temperature + and fan-speed trend charts, audit log + +### Repository layout + +``` +app/ + main.py # Entry: one process = control loop + API + static hosting + api.py # REST + WebSocket routes + controller.py # Control loop (15 s per tick) + curve.py # Piecewise curve + hysteresis + ipmi.py # raw 0x3a 0x01 command family + sensors.py # DCGM / ipmi_exporter / nvidia-smi / Prometheus readers + safety.py # Safety guard + store.py # SQLite persistence + runtime.py # Resource paths (source run = app/ dir; frozen = exe dir) + config.yaml # Configuration (template with example addresses; seed for first run) + deploy/ # systemd units (main service + watchdog) and watchdog script + data/ # fan-console.db (generated at runtime, never overwrite) + static/ # Frontend build output (npm run build lands here) + tests/ # Unit tests +frontend/ # Vue3 + TS + Vite + Tailwind (Dashboard / Fans / Settings) +deploy.sh # One-click deploy script +DEPLOY.md # Deployment manual (pitfalls, rollback, verification checklist) +``` + +### Run locally + +```bash +# 1) Build the frontend (output goes straight into app/static/) +cd frontend && npm install && npm run build && cd .. + +# 2) Install backend dependencies +pip install -r app/requirements.txt + +# 3) Start (one process serves pages + API + control loop) +python -m uvicorn app.main:app --host 0.0.0.0 --port 8765 +# Open http://127.0.0.1:8765 +``` + +Frontend development with hot reload: + +```bash +cd frontend && npm run dev # vite on :5173, /api proxied to 127.0.0.1:8765 +``` + +Run tests: + +```bash +python -m unittest discover -s app/tests -t . -v +``` + +### API overview + +| Method | Path | Description | +|---|---|---| +| GET | `/api/status` | Current state (mode, managed slots, temperatures, duty) | +| GET | `/api/gpus` / `/api/fans` | GPU / fan lists | +| GET/PUT | `/api/curve` | Read/write the control curve | +| POST | `/api/curve/preview` | Curve preview (real algorithm) | +| GET | `/api/history` | Historical trends (from Prometheus) | +| GET/PUT | `/api/assignments` | GPU ↔ fan-slot assignments | +| POST | `/api/mode` | Switch auto / manual / bmc-auto | +| POST | `/api/manual` | Manually set duty cycle | +| POST | `/api/restore-auto` | Hand everything back to BMC auto | +| GET | `/api/audit` | Audit log | +| GET | `/api/health` | Health check | +| WS | `/ws` | Live push | + +### Deployment + +Production target is Linux + systemd (in-band `/dev/ipmi0` access requires root). +**Daily updates are one command**: + +```bash +CONN= ./deploy.sh # full: backend + frontend, restart + self-check +CONN= ./deploy.sh --static-only # frontend only (no backend change, no restart) +``` + +First-time deployment, pitfalls, rollback and the verification checklist live in +**[DEPLOY.md](./DEPLOY.md)**. Last-resort fallback: send +`ipmitool raw 0x3a 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00` to the BMC to hand +all fan slots back to BMC auto (can be done from the BMC's own management address, +no host OS required). + +### CI & releases (GitHub Actions) + +Every push / PR runs backend tests, frontend type-checked build and produces a +deployable bundle artifact. Pushes to main additionally build **standalone +executables for Windows and Linux** (PyInstaller); pushing a `v*` tag automatically +publishes a GitHub Release with all artifacts. See +[.github/workflows/ci.yml](./.github/workflows/ci.yml). + +--- + +## Fan Control Scripts (Legacy CLI) + +Interface-free command-line version: monitors CPU temperature and adjusts fan speeds +per predefined ranges. Works on Windows and Linux. Fan speed and temperature readings +are unified through Prometheus while write commands remain per-model, with +multi-server, multi-threading, daily log rotation and optional e-mail alerting +(off by default). + +### Compatible servers | Brand | Model | Compatible | Type Name | -|:-----:|:-----:|:----------:|:---------:| -| Dell | 730XD | Y | `dell730` | -| Dell | 730 | Y | `dell730` | - -## How to Use - -> On Linux, `ipmitool` must be installed. -> -> For Debian-based systems, install it with `apt install -y ipmitool`. -> For Red Hat-based systems, use `yum install -y ipmitool`. - -1. **Clone the project** - - ``` - git clone https://github.com/dongyu6/python-ipmitool.git - ``` - -2. **Navigate to the project directory** - - ``` - cd python-ipmitool - ``` - -3. **Install dependencies** - - ``` - pip install -r requirements.txt - ``` - -4. **Copy the template file** `fan_settings.yaml.template` to `fan_settings.yaml`. - - ```bash - # Linux/Mac - cp fan_settings.yaml.template fan_settings.yaml - - # Windows - copy fan_settings.yaml.template fan_settings.yaml - ``` - -5. **Edit the newly created `fan_settings.yaml`** file. The meaning of each field is as follows. You need to configure the IP addresses and fan speeds yourself. - - > Note: Only IP addresses are supported, not domain names. - - ```yaml - # IPMI Fan Controller Configuration - - # true for automatic fan control, false for manual - auto: true - - # The interval in seconds for checking temperature and adjusting fan speed - interval: 60 - - # Number of days to retain log files - log_backup_count: 30 - - # Path to the ipmitool executable on Windows - windows_ipmi_tool_path: ".\\ipmitool\\ipmitool.exe" - - # List of servers to manage - servers: - - type: dell730 # Server type - ip: "192.168.71.90" # Server IP address. Set to "local" if running on the target machine - user: root # IPMI username - password: "123123" # IPMI password - temperature_ranges: # List of temperature ranges and corresponding fan speeds - - min_temp: 0 # Minimum temperature of the range (inclusive) - max_temp: 60 # Maximum temperature of the range (inclusive) - fan_speeds: [20, 20, 20, 20, 20, 20] # List of fan speeds in percent - - min_temp: 61 - max_temp: 80 - fan_speeds: [25, 25, 25, 25, 25, 25] - ``` - - -6. **Run the Project** - - The project offers two execution modes: - - ## Mode 1: Loop Control Mode (Recommended for Long-term Operation) - - The program continuously monitors temperature and adjusts fan speeds. Suitable for running as a background service. - - **Foreground Execution (for debugging)** - ```bash - # Windows - python fancontroller.py - - # Linux - python3 fancontroller.py - ``` - - **Background Execution** - ```bash - # Windows - start /b python fancontroller.py - - # Linux - nohup python3 fancontroller.py & - ``` - - ## Mode 2: One-Shot Execution Mode (Recommended for External Scheduling) - - Executes once and exits after temperature detection and fan adjustment. Suitable for being called by external scheduling tools like cron, systemd timer, etc. - - **Direct Execution** - ```bash - # Windows - python fancontroller_once.py - - # Linux - python3 fancontroller_once.py - ``` - - **Using cron for Scheduled Execution (Linux)** - ```bash - # Edit crontab - crontab -e - - # Execute every 10 minutes - */10 * * * * /usr/bin/python3 /path/to/python-ipmitool/fancontroller_once.py - ``` - - **Using Windows Task Scheduler** - ```powershell - # Create a task that runs every 10 minutes - schtasks /create /tn "IPMI Fan Controller" /tr "python C:\path\to\python-ipmitool\fancontroller_once.py" /sc minute /mo 10 - ``` - -### Setup as a systemd Service (Linux Recommended) - -For reliable operation on a Linux server, setting up a systemd service is highly recommended. This provides features like auto-start on boot and process supervision. - -1. **Create the Service File** - - Use a text editor (like `nano` or `vim`) to create a new service file: - ``` - sudo nano /etc/systemd/system/fancontroller.service - ``` - -2. **Paste the Service Configuration** - - Paste the following content into the file. **Note:** You must replace the paths for `User`, `WorkingDirectory`, and `ExecStart` with the actual paths on your server. - - ```ini - [Unit] - Description=Python IPMI Fan Controller - After=network.target - - [Service] - Type=simple - # If you use a non-root user, ensure they have permissions for ipmitool. - User=root - # Absolute path to the project directory. - WorkingDirectory=/path/to/python-ipmitool - # Absolute path to the Python interpreter and the script. - ExecStart=/usr/bin/python3 /path/to/python-ipmitool/fancontroller.py - Restart=always - RestartSec=3 - - [Install] - WantedBy=multi-user.target - ``` - -3. **Reload and Enable the Service** - - Run the following commands to reload the systemd configuration, start the service, and enable it to start on boot. - - ```bash - # Reload the systemd configuration - sudo systemctl daemon-reload - - # Start the service - sudo systemctl start fancontroller.service - - # Check the service status to ensure there are no errors - sudo systemctl status fancontroller.service - - # Enable the service to start automatically on boot - sudo systemctl enable fancontroller.service - ``` - -4. **View Logs** - - Once configured as a service, all output (including errors) can be viewed with `journalctl`: - ```bash - journalctl -u fancontroller.service -f - ``` - -## Contributing and Feedback - -Contributions via Issues and Pull Requests are welcome to help improve the project. If you have any questions or suggestions, please provide feedback through GitHub Issues. +|:---:|:---:|:---:|:---:| +| Dell | 730XD | Y | `dell730` | +| Dell | 730 | Y | `dell730` | +| ASRock Rack | EPYCD8 | Y | `epycd8` | + +### How to use + +> On Linux install `ipmitool` first: Debian-based `apt install -y ipmitool`, +> Red Hat-based `yum install -y ipmitool`. On Windows use the bundled +> `ipmitool/ipmitool.exe`. + +```bash +git clone https://github.com/chennest/python-ipmitool.git +cd python-ipmitool +pip install -r requirements.txt + +# Copy and edit the config (IP addresses only, no domains; use "local" for in-band) +cp fan_settings.yaml.template fan_settings.yaml +``` + +Key fields of `fan_settings.yaml` (full example in the template file): + +```yaml +auto: true # true = auto, false = manual +interval: 60 # control interval (seconds) +log_backup_count: 30 # log retention (days) +windows_ipmi_tool_path: ".\\ipmitool\\ipmitool.exe" +alert: # e-mail alerts (optional, off by default) + enabled: false + fan_speed_threshold: 10000 + max_failed_attempts: 3 + email: { ... } # SMTP config, multiple recipients, 1 h anti-spam +prometheus: + base_url: "http://:9090" +servers: + - type: dell730 + ip: "192.0.2.10" # example address, replace with yours + user: root + password: "your-password" + temperature_ranges: # temperature range → per-fan percentages + - { min_temp: 0, max_temp: 60, fan_speeds: [20, 20, 20, 20, 20, 20] } + - { min_temp: 61, max_temp: 80, fan_speeds: [25, 25, 25, 25, 25, 25] } +``` + +Two run modes: + +```bash +# Mode 1: loop control (recommended for long-running background service) +python fancontroller.py + +# Mode 2: run once (recommended for cron / systemd timer / Task Scheduler) +python fancontroller_once.py +# crontab example: every 10 minutes +# */10 * * * * /usr/bin/python3 /path/to/python-ipmitool/fancontroller_once.py +``` + +For long-running Linux setups configure a systemd service +(`/etc/systemd/system/fancontroller.service`, `After=network.target` + +`Restart=always`; logs via `journalctl -u fancontroller -f`). + +> Alert triggers: fan speed above the threshold (default 10000 RPM) or consecutive +> failures reaching the limit (default 3); per-server alert interval is at least +> 1 hour. Gmail requires an app password. Controller implementation details are in +> [CLAUDE.md](./CLAUDE.md). + +--- + +## Documentation index + +- [DEPLOY.md](./DEPLOY.md) — GPU Fan Console deployment manual (one-click script, pitfalls, rollback, verification checklist) +- [CLAUDE.md](./CLAUDE.md) — repository architecture (both generations, how to add a new model) +- [README.md](./README.md) — Chinese documentation + +## Contributing + +Issues and pull requests are welcome. For questions or suggestions, please open a +GitHub issue. ## License -This project is licensed under the MIT License. See the LICENSE file for details. +This project is licensed under **GPL-3.0** — see the [LICENSE](./LICENSE) file. + +## Credits -## Acknowledgements +[perryclements/r410-fancontroller: Python fan controller for Dell R410 server (GitHub.com)](https://github.com/perryclements/r410-fancontroller) -- [perryclements/r410-fancontroller: Python fan controller for Dell R410 server (GitHub.com)](https://github.com/perryclements/r410-fancontroller) -- [ipmitool/ipmitool: An open-source tool for controlling IPMI-enabled systems (GitHub.com)](https://github.com/ipmitool/ipmitool) +[ipmitool/ipmitool: An open-source tool for controlling IPMI-enabled systems (GitHub.com)](https://github.com/ipmitool/ipmitool) diff --git a/app/__init__.py b/app/__init__.py index 4467afe..a86e271 100644 --- a/app/__init__.py +++ b/app/__init__.py @@ -8,4 +8,4 @@ 没有跨机通信、没有独立 agent —— 它只管部署它的这台机器的风扇。 """ -__version__ = "0.1.0" +__version__ = "2.0.0"