应用部署在 kubernets 集群中,请求是通过阿里云负载均衡+ nginx ingress 转发的,客户端 IP 是通过 X-Forwarded-For
请求头转发的,ASP.NET Core 应用是这么获取客户端 IP 的。
在 Startup.ConfigureServices 中的代码:
services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
options.KnownNetworks.Clear();
options.KnownProxies.Clear();
});
获取客户端 IP 地址的代码:
var ip = HttpContext.Connection.RemoteIpAddress.ToString();
之前部署在 docker swarm 集群上能正常获取,现在部署到 kubernetes 集群上却获取不到,而部署在同一个 k8s 集群上的其他 .net core 能正常获取。
经过2个负载均衡转发,先是阿里云负载均衡,接着是 nginx ingress ,nginx ingress 中对应的配置:
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
不使用 Forwarded Headers Middleware
时,X-Forwarded-For
的值是 "116.62.124.68, 192.168.107.192"
nginx ingress 使用的是 nginxinc/kubernetes-ingress
排查过程:
X-Forwarded-For
的值是 "116.62.124.68, 192.168.107.192"
,ForwardedHeadersMiddleware 是如何解析得到 192.168.107.192
的?release/3.1
分支build.cmd
命令安装 build 环境(在 windows 系统上操作)./activate.sh
切换 .net core sdk 版本XForwardedForDefaultSettingsChangeRemoteIpAndPort
重现问题。[Fact]
public async Task XForwardedForDefaultSettingsChangeRemoteIpAndPort()
{
var builder = new WebHostBuilder()
.Configure(app =>
{
var options = new ForwardedHeadersOptions
{
ForwardedHeaders = ForwardedHeaders.XForwardedFor
};
app.UseForwardedHeaders(options);
});
var server = new TestServer(builder);
var context = await server.SendAsync(c =>
{
c.Request.Headers["X-Forwarded-For"] = "116.62.124.68, 192.168.107.192";
});
Assert.Equal("116.62.124.68", context.Connection.RemoteIpAddress.ToString());
}
ForwardedHeadersMiddleware
是按从右到左的倒序读取 X-Forwarded-For
中的多个 IP 地址的。if (checkFor && i < forwardedFor.Length)
{
set.IpAndPortText = forwardedFor[forwardedFor.Length - i - 1];
}
"116.62.124.68, 192.168.107.192"
得到 "192.168.107.192"
说明只读了1次。if (_options.ForwardLimit.HasValue && entryCount > _options.ForwardLimit)
{
entryCount = _options.ForwardLimit.Value;
}
services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardLimit = 2;
//...
});
上面是从 .NET Core 层面下手的解决方法。
从 nginx ingress 层面排查,似乎是 compute-full-forwarded-for
搞的鬼。
Append the remote address to the X-Forwarded-For header instead of replacing it. When this option is enabled, the upstream application is responsible for extracting the client IP based on its own list of trusted proxies.
这个设置默认是 false
,现在看来是以 true
的方式在工作。
github 上的相关 PR :Add use-forwarded-headers configmap option
在 ingress-nginx 的实现源代码中找到的线索。(更新:这里弄错了,我们用的是 nginxinc/kubernetes-ingress,这里的源代码是 kubernetes/ingress-nginx)
{{ if and $cfg.UseForwardedHeaders $cfg.ComputeFullForwardedFor }}
# We can't use $proxy_add_x_forwarded_for because the realip module
# replaces the remote_addr too soon
map $http_x_forwarded_for $full_x_forwarded_for {
{{ if $all.Cfg.UseProxyProtocol }}
default "$http_x_forwarded_for, $proxy_protocol_addr";
'' "$proxy_protocol_addr";
{{ else }}
default "$http_x_forwarded_for, $realip_remote_addr";
'' "$realip_remote_addr";
{{ end}}
}
{{ end }}
...
{{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
proxy_set_header X-Forwarded-For $full_x_forwarded_for;
{{ else }}
proxy_set_header X-Forwarded-For $remote_addr;
{{ end }}
从上面的代码可以看到要么是 "116.62.124.68, 192.168.107.192"
要么是 "192.168.107.192"
,在 ingress-nginx 层面无法通过直接的配置解决。
nginxinc/kubernetes-ingress 中对应的源码在 nginx.virtualserver.tmpl 中:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
终于解决了!原来是 nginxinc/kubernetes-ingress
的一个坑,换成 kubernetes/ingress-nginx
立马解决,详见博文 见异思迁:K8s 部署 Nginx Ingress Controller 之 kubernetes/ingress-nginx 。
@dudu: 👍👍👍
先用tcpdump port 80 -A -s 0 -c 100
(输出80端口接收的100个数据包,http数据包的ASCII部分会显示到stdout)确认一下进入到容器的HTTP请求是不是携带了X-Forwarded-For
。
猜测应该是转发到你这个服务的HTTP流量并没有携带这个头部,调用这个服务的是ingress还是其他服务?
携带了 X-Forwarded-For ,用下面的代码测试:
public IActionResult ClientIP()
{
return Ok(new Dictionary<string, string>
{
{ "X-Forwarded-For", Request.Headers["X-Forwarded-For"].ToString() },
{ "X-Real-IP", Request.Headers["X-Real-IP"].ToString() },
{ "IP", HttpContext.Connection.RemoteIpAddress.ToString() }
});
}
部署在 k8s 上的响应结果是:
{
"X-Forwarded-For": "客户端公网IP",
"X-Real-IP": "192.168.234.192",
"IP": "192.168.234.192"
}
部署在 docker swarm 上的响应结果是:
{
"X-Forwarded-For": "",
"X-Real-IP": "",
"IP": "客户端公网IP"
}
@dudu:
是不是这个原因
有人遇见了同样的问题,是上面的文档中提到的原因引起的。
https://stackoverflow.com/questions/52577279/x-forwareded-for-is-not-used-by-asp-net-core-behind-reverse-proxy