诡异的TCP out-of-memory问题分析

通过go写了一个简单的HTTP Server供测试端下载文件所用（秉着越少的代码出错越少的想法代码写的很简单----实说吧，就是偷懒！）

package main
import (
      "flag"
      "fmt"
      "net/http"
)
var ip = flag.String("ip", "127.0.0.1", "Listen IP Address")
var port = flag.String("port", "8080", "Listen Port")
var dir = flag.String("dir", "d:/pip", "path for downloading files")
 
func main() {
      flag.Parse()
      fmt.Printf("Listen at port:  %v, server Dir: %v\n", *port, *dir)
      http.Handle("/", http.FileServer(http.Dir(*dir)))
      var addr = fmt.Sprintf("%s:%s", *ip, *port)
      err := http.ListenAndServe(addr, nil)
      if err != nil {
           fmt.Printf("ERROR: %v", err)
      }
}

客户端模拟大量的用户并发下载文件（每一个文件一个sesssion）, 持续跑了一晚后，客户端发现无响应，服务端出现了如下日志打印：

[root@centos8 ~]# dmesg -c
[24114.589659] TCP: out of memory -- consider tuning tcp_mem
[24114.589756] TCP: out of memory -- consider tuning tcp_mem
[24114.589791] TCP: out of memory -- consider tuning tcp_mem
[24114.589890] TCP: out of memory -- consider tuning tcp_mem
[24114.589919] TCP: out of memory -- consider tuning tcp_mem
[24114.589941] TCP: out of memory -- consider tuning tcp_mem
[24114.589966] TCP: out of memory -- consider tuning tcp_mem
[24114.589990] TCP: out of memory -- consider tuning tcp_mem
[24114.590054] TCP: out of memory -- consider tuning tcp_mem
[24114.590080] TCP: out of memory -- consider tuning tcp_mem

通过vmstat，df, top , iostat, netstat, cat /proc/meminfo等都无异常

搜索一些网上资料后，均指向如下几个设置

• core.netdev_max_backlog
• ipv4.tcp_max_syn_backlog
• core.rmem_max
• core.wmem_max
• ipv4.tcp_rmem
• ipv4.tcp_wmem

查看设备当前设置为：

[root@centos8 proc]# sysctl -a | grep rmem
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.udp_rmem_min = 4096

  [root@centos8 proc]# sysctl -a | grep wmem

net.core.wmem_default = 212992

net.core.wmem_max = 212992

net.ipv4.tcp_wmem = 4096        16384   4194304

net.ipv4.udp_wmem_min = 4096

 [root@centos8 proc]# sysctl -a | grep backlog
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 256

按照搜索资料的建议，修改为(通过vim /etc/sysctl.conf， 在文件中增加如下内容)

net.core.netdev_max_backlog=30000
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864

通过sysctl –p使其立刻生效， 也可重启使参数生效

再连接用户后，查看server的状态：

[root@centos8 ~]# cat /proc/net/sockstat
sockets: used 5975
TCP: inuse 5682 orphan 394 tw 0 alloc 5697 mem 74065
UDP: inuse 15 mem 6
UDPLITE: inuse 0
RAW: inuse 3
FRAG: inuse 0 memory 0

 [root@centos8 ~]# cat /proc/net/sockstat6
TCP6: inuse 11
UDP6: inuse 4
UDPLITE6: inuse 0
RAW6: inuse 4
FRAG6: inuse 0 memory 0

如果连接数目太大，有时候会报类似如下错误：

2020/08/27 06:48:52 http: Accept error: accept tcp 172.12.0.101:45000: accept4: too many open files; retrying in 5ms

查看程序的PID

[root@centos8 ~]# ps aux | grep httpserver
root      2611 50.1  7.7 1140108 312556 pts/2  Sl+  02:34  15:07 ./httpserver -ip 172.12.0.101 -port 45000 -dir ./pip

 [root@centos8 ~]# cat /proc/2611/limits

......

Max open files            20000                20000                files

.....

将对应的文件数目限制提高，即可解决

[root@centos8 2611]# cat /etc/security/limits.conf
#<domain>      <type>  <item>         <value>
#
.....
*                soft    nofile       60000
*                hard    nofile       60000
....

通过如下命令查看进程打开文件句柄的数目统计

lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr|more

第一列为进程打开文件句柄数，第二列为进程PID

通过ps aux | grep 1931进一步查看应用名称