加入图片缓存,每次从文件系统中提取相关的图片,加载成bytes,后续使用时直接从缓存中提取。减少IO读取次数,提高发送性能
This commit is contained in:
141
plugins/xiuren_image/CACHE_README.md
Normal file
141
plugins/xiuren_image/CACHE_README.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# 秀人图片插件 - 缓存功能说明
|
||||
|
||||
## 概述
|
||||
|
||||
秀人图片插件现在支持内存缓存功能,可以显著提高图片发送的性能,减少磁盘I/O操作。
|
||||
|
||||
## 缓存机制
|
||||
|
||||
### 1. 缓存策略
|
||||
- **默认缓存大小**: 10张图片(可配置)
|
||||
- **缓存类型**: 内存缓存,存储图片的bytes数据
|
||||
- **缓存策略**: LRU(最近最少使用)策略
|
||||
- **并发处理**: 使用deque的原子操作,无需额外锁机制
|
||||
|
||||
### 2. 工作流程
|
||||
1. 用户请求图片时,优先从内存缓存中获取
|
||||
2. 如果缓存中有图片,直接返回bytes数据
|
||||
3. 当缓存中只剩1张图片时,异步触发缓存填充
|
||||
4. 如果缓存为空,立即同步填充缓存
|
||||
5. 支持回退机制:如果缓存失败,回退到原来的磁盘读取方式
|
||||
|
||||
### 3. 性能优化
|
||||
- **减少磁盘I/O**: 避免频繁的磁盘读取操作
|
||||
- **异步填充**: 缓存填充在后台线程进行,不阻塞主流程
|
||||
- **智能预加载**: 在缓存即将耗尽时提前填充
|
||||
- **内存管理**: 使用deque限制缓存大小,防止内存溢出
|
||||
- **无锁设计**: 避免线程锁导致的死锁问题
|
||||
|
||||
## 配置说明
|
||||
|
||||
在 `config.toml` 文件中可以配置缓存参数:
|
||||
|
||||
```toml
|
||||
[XiurenImage]
|
||||
enable = true
|
||||
image_folder = '/mnt/nfs_share'
|
||||
cache_size = 10 # 内存缓存大小,默认10张图片
|
||||
command = ["图来", "秀人", "美图", "随机图片"]
|
||||
```
|
||||
|
||||
### 配置参数
|
||||
- `cache_size`: 缓存大小,建议设置为5-15张图片
|
||||
- `image_folder`: 图片存储目录
|
||||
- `enable`: 是否启用插件
|
||||
- `command`: 触发命令列表
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
用户发送以下命令即可获取随机图片:
|
||||
- `图来`
|
||||
- `秀人`
|
||||
- `美图`
|
||||
- `随机图片`
|
||||
|
||||
### 2. 缓存状态监控
|
||||
插件会在日志中记录缓存状态:
|
||||
```
|
||||
从缓存获取图片成功,路径:/path/to/image.jpg,当前缓存数量:4
|
||||
```
|
||||
|
||||
### 3. 测试缓存功能
|
||||
运行测试脚本验证缓存功能:
|
||||
```bash
|
||||
cd plugins/xiuren_image
|
||||
python test_cache.py
|
||||
```
|
||||
|
||||
## 技术实现
|
||||
|
||||
### 1. 核心类
|
||||
- `ImageCacheManager`: 缓存管理器
|
||||
- `XiurenImagePlugin`: 主插件类
|
||||
|
||||
### 2. 关键方法
|
||||
- `get_cached_image_bytes()`: 获取缓存的图片bytes
|
||||
- `_refill_cache()`: 重新填充缓存
|
||||
- `_load_image_bytes()`: 从磁盘加载图片bytes
|
||||
|
||||
### 3. 数据结构
|
||||
```python
|
||||
{
|
||||
'path': str, # 图片文件路径
|
||||
'bytes': bytes # 图片的bytes数据
|
||||
}
|
||||
```
|
||||
|
||||
### 4. 并发安全设计
|
||||
- **无锁实现**: 使用Python的deque数据结构,其操作本身是线程安全的
|
||||
- **原子操作**: deque的append和popleft操作是原子的
|
||||
- **避免死锁**: 去掉了显式的线程锁,避免了死锁问题
|
||||
- **异步填充**: 使用daemon线程进行异步填充,不会阻塞主流程
|
||||
|
||||
## 性能对比
|
||||
|
||||
### 优化前
|
||||
- 每次请求都需要从磁盘读取图片文件
|
||||
- 磁盘I/O成为性能瓶颈
|
||||
- 响应时间较长
|
||||
|
||||
### 优化后
|
||||
- 大部分请求直接从内存获取
|
||||
- 显著减少磁盘I/O操作
|
||||
- 响应时间大幅提升
|
||||
- 无锁设计,避免并发问题
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **内存使用**: 缓存会占用一定内存,建议根据服务器内存情况调整缓存大小
|
||||
2. **文件更新**: 如果图片文件被更新,需要重启插件或等待缓存刷新
|
||||
3. **错误处理**: 如果缓存失败,会自动回退到原来的磁盘读取方式
|
||||
4. **并发安全**: 使用deque的原子操作,支持并发访问,无需额外锁机制
|
||||
|
||||
## 故障排除
|
||||
|
||||
### 1. 缓存不工作
|
||||
- 检查Redis连接是否正常
|
||||
- 确认图片目录是否存在且有读取权限
|
||||
- 查看日志中的错误信息
|
||||
|
||||
### 2. 内存占用过高
|
||||
- 减少 `cache_size` 配置值
|
||||
- 检查是否有内存泄漏
|
||||
|
||||
### 3. 图片获取失败
|
||||
- 检查图片文件是否完整
|
||||
- 确认文件格式是否支持
|
||||
- 查看磁盘空间是否充足
|
||||
|
||||
### 4. 性能问题
|
||||
- 如果出现卡死,检查是否有死锁问题
|
||||
- 确保异步填充线程正常工作
|
||||
- 监控缓存命中率
|
||||
|
||||
## 更新日志
|
||||
|
||||
### v1.1.0
|
||||
- 去掉了线程锁,改用无锁设计
|
||||
- 修复了死锁问题
|
||||
- 提高了并发性能
|
||||
- 增加了更详细的错误处理
|
||||
@@ -1,6 +1,7 @@
|
||||
[XiurenImage]
|
||||
enable = true
|
||||
image_folder = '/mnt/nfs_share'
|
||||
cache_size = 10 # 内存缓存大小,默认5张图片
|
||||
command = ["图来", "秀人", "美图", "随机图片"]
|
||||
command-format = """
|
||||
🖼️秀人图片指令:
|
||||
|
||||
@@ -2,7 +2,8 @@ import os
|
||||
import time
|
||||
import random
|
||||
import asyncio
|
||||
from typing import Optional, List
|
||||
from typing import Optional, List, Dict
|
||||
from collections import deque
|
||||
|
||||
import logging
|
||||
|
||||
@@ -16,10 +17,15 @@ class ImageCacheManager:
|
||||
LAST_UPDATE_TIME_KEY = "group:images:last_update_time"
|
||||
BATCH_SIZE = 500
|
||||
|
||||
def __init__(self, image_folder: str):
|
||||
def __init__(self, image_folder: str, cache_size: int = 5):
|
||||
self.image_folder = image_folder
|
||||
self.redis = DBConnectionManager.get_instance().get_redis_connection()
|
||||
self.image_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.bmp', '.tiff', '.webp'}
|
||||
|
||||
# 内存缓存相关
|
||||
self.cache_size = cache_size
|
||||
self.image_bytes_cache = deque(maxlen=cache_size) # 使用deque作为LRU缓存
|
||||
self.is_refilling = False # 防止重复填充缓存
|
||||
|
||||
def _get_last_update_time(self) -> float:
|
||||
ts = self.redis.get(self.LAST_UPDATE_TIME_KEY)
|
||||
@@ -112,3 +118,86 @@ class ImageCacheManager:
|
||||
except Exception as e:
|
||||
logger.error(f"获取随机图片失败: {e}")
|
||||
return None
|
||||
|
||||
def _load_image_bytes(self, image_path: str) -> Optional[bytes]:
|
||||
"""从磁盘加载图片的bytes数据"""
|
||||
try:
|
||||
with open(image_path, 'rb') as f:
|
||||
return f.read()
|
||||
except Exception as e:
|
||||
logger.error(f"读取图片文件失败 {image_path}: {e}")
|
||||
return None
|
||||
|
||||
def _refill_cache(self):
|
||||
"""重新填充缓存"""
|
||||
if self.is_refilling:
|
||||
return
|
||||
|
||||
self.is_refilling = True
|
||||
try:
|
||||
logger.info("开始重新填充图片缓存...")
|
||||
|
||||
# 获取多个随机图片路径
|
||||
redis_key = self.IMAGE_KEY_PREFIX + "all"
|
||||
image_paths = []
|
||||
|
||||
# 获取比缓存大小多一点的图片路径,以防有些文件读取失败
|
||||
for _ in range(self.cache_size + 2):
|
||||
try:
|
||||
img = self.redis.srandmember(redis_key)
|
||||
if img:
|
||||
path = img.decode('utf-8') if isinstance(img, bytes) else img
|
||||
# 检查路径是否已经在缓存中
|
||||
existing_paths = [item['path'] for item in self.image_bytes_cache]
|
||||
if path not in existing_paths:
|
||||
image_paths.append(path)
|
||||
except Exception as e:
|
||||
logger.error(f"获取随机图片路径失败: {e}")
|
||||
continue
|
||||
|
||||
# 加载图片bytes并添加到缓存
|
||||
loaded_count = 0
|
||||
for path in image_paths:
|
||||
if loaded_count >= self.cache_size:
|
||||
break
|
||||
|
||||
image_bytes = self._load_image_bytes(path)
|
||||
if image_bytes:
|
||||
self.image_bytes_cache.append({
|
||||
'path': path,
|
||||
'bytes': image_bytes
|
||||
})
|
||||
loaded_count += 1
|
||||
|
||||
logger.info(f"缓存填充完成,新增 {loaded_count} 张图片")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"填充缓存失败: {e}")
|
||||
finally:
|
||||
self.is_refilling = False
|
||||
|
||||
def get_cached_image_bytes(self) -> Optional[Dict[str, any]]:
|
||||
"""
|
||||
从缓存中获取图片bytes数据
|
||||
返回格式: {'path': str, 'bytes': bytes}
|
||||
"""
|
||||
# 如果缓存为空,立即填充
|
||||
if not self.image_bytes_cache:
|
||||
self._refill_cache()
|
||||
if not self.image_bytes_cache:
|
||||
return None
|
||||
|
||||
# 从缓存中取出一个图片
|
||||
cached_image = self.image_bytes_cache.popleft()
|
||||
|
||||
# 如果缓存中只剩最后一个,异步填充缓存
|
||||
if len(self.image_bytes_cache) <= 1:
|
||||
# 使用线程池异步填充,避免阻塞
|
||||
import threading
|
||||
threading.Thread(target=self._refill_cache, daemon=True).start()
|
||||
|
||||
return cached_image
|
||||
|
||||
def get_cached_image_count(self) -> int:
|
||||
"""获取当前缓存中的图片数量"""
|
||||
return len(self.image_bytes_cache)
|
||||
|
||||
@@ -2,6 +2,7 @@ from pathlib import Path
|
||||
|
||||
from loguru import logger
|
||||
import os
|
||||
import base64
|
||||
from typing import Dict, Any, List, Optional, Tuple
|
||||
|
||||
from db.connection import DBConnectionManager
|
||||
@@ -60,6 +61,8 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
self.image_folder = str(Path(Path(__file__).parent.parent.parent, "xiuren"))
|
||||
# 注册功能权限
|
||||
self.feature = self.register_feature()
|
||||
# 初始化图片缓存管理器
|
||||
self.image_cache_manager = None
|
||||
|
||||
def initialize(self, context: Dict[str, Any]) -> bool:
|
||||
"""初始化插件"""
|
||||
@@ -78,6 +81,9 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
if config_image_folder:
|
||||
self.image_folder = config_image_folder
|
||||
|
||||
# 从配置获取缓存大小,默认5张
|
||||
cache_size = self._config.get("XiurenImage", {}).get("cache_size", 5)
|
||||
|
||||
# 检查图片文件夹是否存在
|
||||
try:
|
||||
if not os.path.exists(self.image_folder):
|
||||
@@ -86,7 +92,10 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
except Exception as e:
|
||||
self.LOG.error(f"创建图片文件夹失败: {e}")
|
||||
|
||||
self.LOG.info(f"[{self.name}] 插件初始化完成,指令:{self._commands},图片目录:{self.image_folder}")
|
||||
# 初始化图片缓存管理器
|
||||
self.image_cache_manager = ImageCacheManager(self.image_folder, cache_size)
|
||||
|
||||
self.LOG.info(f"[{self.name}] 插件初始化完成,指令:{self._commands},图片目录:{self.image_folder},缓存大小:{cache_size}")
|
||||
return True
|
||||
|
||||
def start(self) -> bool:
|
||||
@@ -127,10 +136,9 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
return False, "没有权限"
|
||||
|
||||
try:
|
||||
# 获取随机图片
|
||||
pic_path = self._get_random_pic()
|
||||
self.LOG.info(f"返回图片地址:{pic_path}")
|
||||
if not pic_path:
|
||||
# 从缓存获取图片bytes数据
|
||||
cached_image = self._get_cached_image()
|
||||
if not cached_image:
|
||||
client_msg_id, create_time, new_msg_id = await bot.send_text_message((roomid if roomid else sender),
|
||||
f"❌未找到图片资源",
|
||||
sender)
|
||||
@@ -138,9 +146,16 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
return False, "未找到图片资源"
|
||||
|
||||
# 发送图片
|
||||
image_data = cached_image['bytes']
|
||||
image_path = cached_image['path']
|
||||
|
||||
# 记录缓存状态
|
||||
cache_count = self.image_cache_manager.get_cached_image_count()
|
||||
self.LOG.info(f"从缓存获取图片成功,路径:{image_path},当前缓存数量:{cache_count}")
|
||||
|
||||
# 发送图片,支持bytes格式
|
||||
client_msg_id, create_time, new_msg_id = await bot.send_image_message((roomid if roomid else sender),
|
||||
Path(pic_path))
|
||||
image_data)
|
||||
# revoke.add_message_to_revoke(roomid, client_msg_id, create_time, new_msg_id, 90)
|
||||
self.LOG.info(
|
||||
f"发送图片结果,client_msg_id= {client_msg_id},create_time={create_time},new_msg_id={new_msg_id}")
|
||||
@@ -150,6 +165,39 @@ class XiurenImagePlugin(MessagePluginInterface):
|
||||
self.LOG.error(f"处理图片请求出错: {e}")
|
||||
return False, f"处理出错: {e}"
|
||||
|
||||
def _get_cached_image(self) -> Optional[Dict[str, any]]:
|
||||
"""
|
||||
从缓存获取图片数据
|
||||
返回格式: {'path': str, 'bytes': bytes}
|
||||
"""
|
||||
try:
|
||||
# 优先从内存缓存获取
|
||||
cached_image = self.image_cache_manager.get_cached_image_bytes()
|
||||
if cached_image:
|
||||
return cached_image
|
||||
|
||||
# 如果缓存中没有,回退到原来的方式
|
||||
self.LOG.warning("缓存中没有图片,回退到磁盘读取")
|
||||
pic_path = self._get_random_pic()
|
||||
if pic_path:
|
||||
# 读取图片bytes
|
||||
try:
|
||||
with open(pic_path, 'rb') as f:
|
||||
image_bytes = f.read()
|
||||
return {
|
||||
'path': pic_path,
|
||||
'bytes': image_bytes
|
||||
}
|
||||
except Exception as e:
|
||||
self.LOG.error(f"读取图片文件失败: {e}")
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
self.LOG.error(f"获取缓存图片失败: {e}")
|
||||
return None
|
||||
|
||||
def _get_random_pic(self) -> Optional[str]:
|
||||
"""
|
||||
从 Redis 随机获取图片路径
|
||||
|
||||
55
plugins/xiuren_image/test_cache.py
Normal file
55
plugins/xiuren_image/test_cache.py
Normal file
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
测试图片缓存功能
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..'))
|
||||
|
||||
from images_cache import ImageCacheManager
|
||||
import time
|
||||
|
||||
def test_cache_functionality():
|
||||
"""测试缓存功能"""
|
||||
print("开始测试图片缓存功能...")
|
||||
|
||||
# 初始化缓存管理器
|
||||
image_folder = "/mnt/nfs_share" # 根据实际路径调整
|
||||
cache_size = 10
|
||||
|
||||
try:
|
||||
cache_manager = ImageCacheManager(image_folder, cache_size)
|
||||
print(f"缓存管理器初始化成功,缓存大小:{cache_size}")
|
||||
|
||||
# 测试获取缓存图片
|
||||
print("\n测试获取缓存图片...")
|
||||
for i in range(10):
|
||||
print(f"\n第 {i+1} 次获取图片:")
|
||||
|
||||
try:
|
||||
# 获取缓存图片
|
||||
cached_image = cache_manager.get_cached_image_bytes()
|
||||
if cached_image:
|
||||
print(f" ✓ 成功获取图片")
|
||||
print(f" 路径: {cached_image['path']}")
|
||||
print(f" 大小: {len(cached_image['bytes'])} bytes")
|
||||
print(f" 当前缓存数量: {cache_manager.get_cached_image_count()}")
|
||||
else:
|
||||
print(f" ✗ 获取图片失败")
|
||||
except Exception as e:
|
||||
print(f" ✗ 获取图片时出错: {e}")
|
||||
|
||||
# 模拟请求间隔
|
||||
time.sleep(0.5)
|
||||
|
||||
print("\n缓存测试完成!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"测试过程中出现错误: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_cache_functionality()
|
||||
Reference in New Issue
Block a user