] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading...

2026年05月08日 15:36 processing

错误信息

2026/05/08 14:03:20 [error] 1974100#0: *156389 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://127.0.0.1:4022/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:20 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:21 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:22 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: ＂POST /v1/chat/completions HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/chat/completions＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: ＂GET /pricing HTTP/2.0＂, upstream: ＂http://rix_api_cluster/pricing＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: ＂GET /pricing HTTP/2.0＂, upstream: ＂http://rix_api_cluster/pricing＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:23 [error] 1974100#0: *156388 no live upstreams while connecting to upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: ＂GET /favicon.ico HTTP/2.0＂, upstream: ＂http://rix_api_cluster/favicon.ico＂, host: ＂www.dmxapi.cn＂, referrer: ＂https://www.dmxapi.cn/pricing＂ 2026/05/08 14:03:23 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: ＂POST /v1/chat/completions HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/chat/completions＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:24 [error] 1974100#0: *156392 no live upstreams while connecting to upstream, client: 120.235.182.29, server: www.dmxapi.cn, request: ＂POST /v1/chat/completions HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/chat/completions＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:24 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:25 [error] 1974100#0: *156389 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:30 [error] 1974100#0: *156390 no live upstreams while connecting to upstream, client: 180.154.36.86, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://rix_api_cluster/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:30 [error] 1974100#0: *156397 no live upstreams while connecting to upstream, client: 183.207.4.159, server: www.dmxapi.cn, request: ＂HEAD /api/status HTTP/2.0＂, upstream: ＂http://rix_api_cluster/api/status＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:32 [error] 1974100#0: *156398 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: ＂HEAD /api/status HTTP/2.0＂, upstream: ＂http://127.0.0.1:4022/api/status＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:32 [error] 1974100#0: *156398 no live upstreams while connecting to upstream, client: 14.145.50.172, server: www.dmxapi.cn, request: ＂HEAD /api/status HTTP/2.0＂, upstream: ＂http://rix_api_cluster/api/status＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:03:49 [error] 1974100#0: *156408 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 183.202.141.164, server: www.dmxapi.cn, request: ＂GET /pricing HTTP/2.0＂, upstream: ＂http://127.0.0.1:4022/pricing＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:15:08 [error] 1976640#0: *208142 writev() failed (32: Broken pipe) while sending request to upstream, client: 59.175.239.240, server: www.dmxapi.cn, request: ＂POST /v1/messages?beta=true HTTP/1.1＂, upstream: ＂http://127.0.0.1:3020/v1/messages?beta=true＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:58:26 [error] 1985668#0: *446411 writev() failed (32: Broken pipe) while sending request to upstream, client: 223.75.118.120, server: www.dmxapi.cn, request: ＂POST / HTTP/2.0＂, upstream: ＂http://127.0.0.1:4022/＂, host: ＂www.dmxapi.cn＂ 2026/05/08 14:59:04 [error] 1985668#0: *463602 open() ＂/www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice＂ failed (2: No such file or directory), client: 64.233.173.98, server: www.dmxapi.cn, request: ＂GET /.well-known/traffic-advice HTTP/1.1＂, host: ＂www.dmxapi.cn＂

AI 解决方案

Nginx 错误日志分析报告

根据您提供的日志，这是一个典型的Nginx反向代理服务不可用问题。以下是详细分析：

1.错误翻译

  • 原文：no live upstreams while connecting to upstream

    • 译文：在连接上游服务器时，没有可用的（存活的）上游节点。

    • 原文：recv() failed (104: Connection reset by peer)

      • 译文：从上游服务器接收数据时失败（错误104：连接被对等端重置）。

      • 原文：writev() failed (32: Broken pipe)

        • 译文：向上游服务器发送请求时失败（错误32：管道破裂）。

        • 原文：open() ... failed (2: No such file or directory)

          • 译文：打开文件.../traffic-advice失败（错误2：没有这个文件或目录）。

          2. 原因分析

          根本原因：Nginx作为反向代理，无法连接到其配置的后端服务（Upstream），导致所有转发到后端服务的请求失败。

          具体分析：

        • no live upstreams：这是日志中出现最频繁的错误。表明在rix_api_cluster这个上游集群中，所有后端服务器都不可用。Nginx的健康检查（如果配置了）或主动探测失败，导致它认为没有可用的服务器来处理请求。

        • Connection reset by peer：当Nginx尝试连接到127.0.0.1:4022等具体后端时，连接刚建立或正在传输数据时就被后端服务（或中间网络设备）强制关闭了。这通常意味着后端服务进程崩溃、过载或配置错误。

        • Broken pipe：Nginx向后端发送请求时，后端已经关闭了连接。常见于后端服务处理不过来，或者Nginx超时后断开，而后端才开始响应该请求。

        • No such file or directory：这是独立的非关键错误，是Google爬虫尝试访问一个不存在的Web标准文件（用于特定服务间的通信），与核心API服务崩溃无关。

        触发场景：
        日志显示短时间内（14:03:20 - 14:03:32）大量不同的客户端IP请求不同的API端点（/v1/messages, /v1/chat/completions, /pricing）均失败，进一步印证了是整个后端服务集群（rix_api_cluster）或其负载均衡层出现了全局性问题，而非单个服务或路径的问题。

        3决方案

        由于问题出在后端服务，需优先检查后端服务状态。

        # 步骤1：立即检查后端服务状态
# 假设后端运行在4022和3020端口
$ systemctl status your-api-service  # 检查服务进程
$ netstat -tlnp | grep 4022         # 检查端口是否在监听
$ curl -v http://127.0.0.1:4022/health  # 直接测试后端健康检查接口


        # 步骤2：检查Nginx上游集群配置（通常位于 /etc/nginx/conf.d/ 或 /etc/nginx/nginx.conf）
# 确保 `rix_api_cluster` 的服务器列表正确，并启用了健康检查和故障转移。
upstream rix_api_cluster {
    # 示例配置，请根据实际情况修改
    server 127.0.0.1:4022 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3020 max_fails=3 fail_timeout=30s; # 备用节点示例
    keepalive 32; # 保持连接，减少开销
}


        # 步骤3：检查并优化Nginx的超时和缓冲区配置，应对`Broken pipe`
server {
    ...
    proxy_connect_timeout 60s;      # 连接超时
    proxy_send_timeout    60s;      # 发送超时
    proxy_read_timeout    60s;      # 读取超时
    proxy_buffering       on;
    proxy_buffer_size     16k;
    proxy_buffers         4 32k;
    ...
}


        # 步骤4：针对 `/.well-known/traffic-advice` 文件缺失
# 方案A：创建该文件（如果业务需要）
$ mkdir -p /www/wwwroot/www.dmxapi.cn/.well-known
$ echo '{}' > /www/wwwroot/www.dmxapi.cn/.well-known/traffic-advice

# 方案B：在Nginx中优雅地忽略它（推荐）
location = /.well-known/traffic-advice {
    default_type "application/json";
    return 200 '{}';
}

        4. 预防措施

        • 实施健康检查与自动熔断：

          • - 为上游集群配置Nginx的主动健康检查（health_check），或借助外部监控系统（如Prometheus + Blackbox Exporter）实时监控后端服务。
          - 在应用层面实现熔断降级，防止上游服务故障蔓延。

          • 优化负载均衡与连接管理：

            • - 为upstream配置合理的max_fails和fail_timeout参数，实现自动故障转移。
            - 启用keepalive连接池，提升性能并减少Broken pipe的发生。

            • 建立完善的监控告警体系：

              • - 监控Nginx日志中的no live upstreams、upstream timed out等关键错误模式。
              - 监控后端服务的端口存活、进程状态、系统资源（CPU、内存、连接数）。
              - 设置即时告警（如企业微信、钉钉、短信），在服务不可用时第一时间通知运维人员。

分析新问题 查看更多