] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading...

2026年05月08日 15:36 状态: processing

🚨 错误信息

2026/05/08 14:03:20 [error] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://127.0.0.1:4022/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:20 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn" 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://rix_api_cluster/pricing", host: "www.dmxapi.cn" 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://rix_api_cluster/pricing", host: "www.dmxapi.cn" 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /favicon.ico HTTP/2.0", upstream: "http://rix_api_cluster/favicon.ico", host: "www.dmxapi.cn", referrer: "https://www.dmxapi.cn/pricing" 2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn" 2026/05/08 14:03:24 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: "POST /v1/chat/completions HTTP/1.1", upstream: "http://rix_api_cluster/v1/chat/completions", host: "www.dmxapi.cn" 2026/05/08 14:03:24 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:25 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:30 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://rix_api_cluster/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:03:30 [error] 1974100#0: *156397 no live upstreams while connecting to upstream, client: 183.207.4.159, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://rix_api_cluster/api/status", host: "www.dmxapi.cn" 2026/05/08 14:03:32 [error] 1974100#0: *156398 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://127.0.0.1:4022/api/status", host: "www.dmxapi.cn" 2026/05/08 14:03:32 [error] 1974100#0: *156398 no live upstreams while connecting to upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: "HEAD /api/status HTTP/2.0", upstream: "http://rix_api_cluster/api/status", host: "www.dmxapi.cn" 2026/05/08 14:03:49 [error] 1974100#0: *156408 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: "GET /pricing HTTP/2.0", upstream: "http://127.0.0.1:4022/pricing", host: "www.dmxapi.cn" 2026/05/08 14:15:08 [error] 1976640#0: *208142 writev() failed (32: Broken pipe) while sending request to upstream, client: 59.175.239.240, server: www.dmxapi.cn, request: "POST /v1/messages?beta=true HTTP/1.1", upstream: "http://127.0.0.1:3020/v1/messages?beta=true", host: "www.dmxapi.cn" 2026/05/08 14:58:26 [error] 1985668#0: *446411 writev() failed (32: Broken pipe) while sending request to upstream, client: 223.75.118.120, server: www.dmxapi.cn, request: "POST / HTTP/2.0", upstream: "http://127.0.0.1:4022/", host: "www.dmxapi.cn" 2026/05/08 14:59:04 [error] 1985668#0: *463602 open() "/www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice" failed (2: No such file or directory), client: 64.233.173.98, server: www.dmxapi.cn, request: "GET /.well-known/traffic-advice HTTP/1.1", host: "www.dmxapi.cn"

🤖 AI解决方案

Nginx 错误日志分析报告

根据您提供的日志,这是一个典型的Nginx反向代理服务不可用问题。以下是详细分析:

---

1.错误翻译

  • 原文no live upstreams while connecting to upstream

  • 译文:在连接上游服务器时,没有可用的(存活的)上游节点。

  • 原文recv() failed (104: Connection reset by peer)

  • 译文:从上游服务器接收数据时失败(错误104:连接被对等端重置)。

  • 原文writev() failed (32: Broken pipe)

  • 译文:向上游服务器发送请求时失败(错误32:管道破裂)。

  • 原文open() ... failed (2: No such file or directory)

  • 译文:打开文件.../traffic-advice失败(错误2:没有这个文件或目录)。

    2. 原因分析

    根本原因:Nginx作为反向代理,无法连接到其配置的后端服务(Upstream),导致所有转发到后端服务的请求失败。

    具体分析

  • no live upstreams:这是日志中出现最频繁的错误。表明在rix_api_cluster这个上游集群中,所有后端服务器都不可用。Nginx的健康检查(如果配置了)或主动探测失败,导致它认为没有可用的服务器来处理请求。

  • Connection reset by peer:当Nginx尝试连接到127.0.0.1:4022等具体后端时,连接刚建立或正在传输数据时就被后端服务(或中间网络设备)强制关闭了。这通常意味着后端服务进程崩溃、过载或配置错误。

  • Broken pipe:Nginx向后端发送请求时,后端已经关闭了连接。常见于后端服务处理不过来,或者Nginx超时后断开,而后端才开始响应该请求。

  • No such file or directory:这是独立的非关键错误,是Google爬虫尝试访问一个不存在的Web标准文件(用于特定服务间的通信),与核心API服务崩溃无关。
  • 触发场景
    日志显示短时间内(14:03:20 - 14:03:32)大量不同的客户端IP请求不同的API端点(/v1/messages, /v1/chat/completions, /pricing)均失败,进一步印证了是整个后端服务集群(rix_api_cluster)或其负载均衡层出现了全局性问题,而非单个服务或路径的问题。

    3决方案

    由于问题出在后端服务,需优先检查后端服务状态。

    nginx

    步骤1:立即检查后端服务状态


    假设后端运行在4022和3020端口


    $ systemctl status your-api-service # 检查服务进程
    $ netstat -tlnp | grep 4022 # 检查端口是否在监听
    $ curl -v http://127.0.0.1:4022/health # 直接测试后端健康检查接口

    ``nginx

    步骤2:检查Nginx上游集群配置(通常位于 /etc/nginx/conf.d/ 或 /etc/nginx/nginx.conf)


    确保 rix_api_cluster 的服务器列表正确,并启用了健康检查和故障转移。


    upstream rix_api_cluster {
    # 示例配置,请根据实际情况修改
    server 127.0.0.1:4022 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3020 max_fails=3 fail_timeout=30s; # 备用节点示例
    keepalive 32; # 保持连接,减少开销
    }

    nginx

    步骤3:检查并优化Nginx的超时和缓冲区配置,应对Broken pipe


    server {
    ...
    proxy_connect_timeout 60s; # 连接超时
    proxy_send_timeout 60s; # 发送超时
    proxy_read_timeout 60s; # 读取超时
    proxy_buffering on;
    proxy_buffer_size 16k;
    proxy_buffers 4 32k;
    ...
    }

    nginx

    步骤4:针对 /.well-known/traffic-advice 文件缺失


    方案A:创建该文件(如果业务需要)


    $ mkdir -p /www/wwwroot/www.dmxapi.cn/.well-known
    $ echo '{}' > /www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice

    方案B:在Nginx中优雅地忽略它(推荐)


    location = /.well-known/traffic-advice {
    default_type "application/json";
    return 200 '{}';
    }
    `

    4. 预防措施

  • 实施健康检查与自动熔断

  • - 为上游集群配置Nginx的主动健康检查(
    health_check),或借助外部监控系统(如Prometheus + Blackbox Exporter)实时监控后端服务。
    - 在应用层面实现熔断降级,防止上游服务故障蔓延。

  • 优化负载均衡与连接管理

  • - 为
    upstream配置合理的max_failsfail_timeout参数,实现自动故障转移。
    - 启用
    keepalive连接池,提升性能并减少Broken pipe的发生。

  • 建立完善的监控告警体系

  • - 监控Nginx日志中的
    no live upstreamsupstream timed out`等关键错误模式。
    - 监控后端服务的端口存活、进程状态、系统资源(CPU、内存、连接数)。
    - 设置即时告警(如企业微信、钉钉、短信),在服务不可用时第一时间通知运维人员。