] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading...
🚨 错误信息
2026/05/08 14:03:20 [error] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://127.0.0.1:4022/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:20 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn"
2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://rix_api_cluster/pricing", host: "www.dmxapi.cn"
2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://rix_api_cluster/pricing", host: "www.dmxapi.cn"
2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /favicon.ico HTTP/2.0", upstream: "http://rix_api_cluster/favicon.ico", host: "www.dmxapi.cn", referrer: "https://www.dmxapi.cn/pricing"
2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn"
2026/05/08 14:03:24 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn"
2026/05/08 14:03:24 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:25 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:30 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:03:30 [error] 1974100#0: *156397 no live upstreams while connecting to upstream, client: 183.207.4.159, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://rix_api_cluster/api/status", host: "www.dmxapi.cn"
2026/05/08 14:03:32 [error] 1974100#0: *156398 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://127.0.0.1:4022/api/status", host: "www.dmxapi.cn"
2026/05/08 14:03:32 [error] 1974100#0: *156398 no live upstreams while connecting to upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://rix_api_cluster/api/status", host: "www.dmxapi.cn"
2026/05/08 14:03:49 [error] 1974100#0: *156408 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://127.0.0.1:4022/pricing", host: "www.dmxapi.cn"
2026/05/08 14:15:08 [error] 1976640#0: *208142 writev() failed (32: Broken pipe) while sending request to upstream, client: 59.175.239.240, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://127.0.0.1:3020/v1/messages?beta=true", host: "www.dmxapi.cn"
2026/05/08 14:58:26 [error] 1985668#0: *446411 writev() failed (32: Broken pipe) while sending request to upstream, client: 223.75.118.120, server: www.dmxapi.cn, request: "POST / HTTP/2.0", upstream: "http://127.0.0.1:4022/", host: "www.dmxapi.cn"
2026/05/08 14:59:04 [error] 1985668#0: *463602 open() "/www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice" failed (2: No such file or directory), client: 64.233.173.98, server: www.dmxapi.cn, request: "GET /.well-known/traffic-advice HTTP/1.1", host: "www.dmxapi.cn"
🤖 AI解决方案
Nginx 错误日志分析报告
根据您提供的日志,这是一个典型的Nginx反向代理服务不可用问题。以下是详细分析:
---
1.错误翻译
no live upstreams while connecting to upstream译文:在连接上游服务器时,没有可用的(存活的)上游节点。
recv() failed (104: Connection reset by peer)译文:从上游服务器接收数据时失败(错误104:连接被对等端重置)。
writev() failed (32: Broken pipe)译文:向上游服务器发送请求时失败(错误32:管道破裂)。
open() ... failed (2: No such file or directory)译文:打开文件
.../traffic-advice失败(错误2:没有这个文件或目录)。2. 原因分析
根本原因:Nginx作为反向代理,无法连接到其配置的后端服务(Upstream),导致所有转发到后端服务的请求失败。
具体分析:
no live upstreams:这是日志中出现最频繁的错误。表明在rix_api_cluster这个上游集群中,所有后端服务器都不可用。Nginx的健康检查(如果配置了)或主动探测失败,导致它认为没有可用的服务器来处理请求。Connection reset by peer:当Nginx尝试连接到127.0.0.1:4022等具体后端时,连接刚建立或正在传输数据时就被后端服务(或中间网络设备)强制关闭了。这通常意味着后端服务进程崩溃、过载或配置错误。Broken pipe:Nginx向后端发送请求时,后端已经关闭了连接。常见于后端服务处理不过来,或者Nginx超时后断开,而后端才开始响应该请求。No such file or directory:这是独立的非关键错误,是Google爬虫尝试访问一个不存在的Web标准文件(用于特定服务间的通信),与核心API服务崩溃无关。触发场景:
日志显示短时间内(14:03:20 - 14:03:32)大量不同的客户端IP请求不同的API端点(/v1/messages, /v1/chat/completions, /pricing)均失败,进一步印证了是整个后端服务集群(rix_api_cluster)或其负载均衡层出现了全局性问题,而非单个服务或路径的问题。
3决方案
由于问题出在后端服务,需优先检查后端服务状态。
nginx
步骤1:立即检查后端服务状态
假设后端运行在4022和3020端口
$ systemctl status your-api-service # 检查服务进程
$ netstat -tlnp | grep 4022 # 检查端口是否在监听
$ curl -v http://127.0.0.1:4022/health # 直接测试后端健康检查接口
``nginxrix_api_cluster步骤2:检查Nginx上游集群配置(通常位于 /etc/nginx/conf.d/ 或 /etc/nginx/nginx.conf)
确保
的服务器列表正确,并启用了健康检查和故障转移。
upstream rix_api_cluster {
# 示例配置,请根据实际情况修改
server 127.0.0.1:4022 max_fails=3 fail_timeout=30s;
server 127.0.0.1:3020 max_fails=3 fail_timeout=30s; # 备用节点示例
keepalive 32; # 保持连接,减少开销
}
nginx
步骤3:检查并优化Nginx的超时和缓冲区配置,应对Broken pipe
server {
...
proxy_connect_timeout 60s; # 连接超时
proxy_send_timeout 60s; # 发送超时
proxy_read_timeout 60s; # 读取超时
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 4 32k;
...
}
nginx步骤4:针对 /.well-known/traffic-advice 文件缺失
方案A:创建该文件(如果业务需要)
$ mkdir -p /www/wwwroot/www.dmxapi.cn/.well-known
$ echo '{}' > /www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice
方案B:在Nginx中优雅地忽略它(推荐)
location = /.well-known/traffic-advice {
default_type "application/json";
return 200 '{}';
}
`
4. 预防措施
实施健康检查与自动熔断:
- 为上游集群配置Nginx的主动健康检查(health_check),或借助外部监控系统(如Prometheus + Blackbox Exporter)实时监控后端服务。
- 在应用层面实现熔断降级,防止上游服务故障蔓延。 优化负载均衡与连接管理:
- 为upstream配置合理的max_fails和fail_timeout参数,实现自动故障转移。
- 启用keepalive连接池,提升性能并减少Broken pipe的发生。 建立完善的监控告警体系:
- 监控Nginx日志中的no live upstreams、upstream timed out`等关键错误模式。- 监控后端服务的端口存活、进程状态、系统资源(CPU、内存、连接数)。
- 设置即时告警(如企业微信、钉钉、短信),在服务不可用时第一时间通知运维人员。