Nginx 日志 worker_connections are not enough while connecting to upstream
发布于2018-04-27 12:18:39
阅读
1.9K
0
记一次，排查错误所遇到的问题，和学习到的内容。
上周五，刚上线的项目出现了503 ，查看日志发现如下内容：
System.Exception: Request api/blogpost/zzkDocs
<html>^M
<head><title>500 Internal Server Error</title></head>^M
<body bgcolor="white">^M
<center><h1>500 Internal Server Error</h1></center>^M
<hr><center>nginx/1.10.3 (Ubuntu)</center>^M
</body>^M
</html>^M
---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at System.Net.Http.HttpClientExtensions.<ReadAsAsync>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CNBlogs.Zzk.ServiceAgent.ZzkDocumentManager`1.<GetZzkDocAsync>d__8.MoveNext() in /home/xiaokang/CNBlogsZzk/CNBlogsZzk/src/CNBlogs.Zzk.ServiceAgent/ZzkDocumentManager.cs:line 33
<html>^M
<head><title>500 Internal Server Error</title></head>^M
<body bgcolor="white">^M
<center><h1>500 Internal Server Error</h1></center>^M
<hr><center>nginx/1.10.3 (Ubuntu)</center>^M
</body>^M
</html>^M
原来是本来请求的文档却反馈了503的html，这样当然没有办法转成json。站点是从另外一台服务器上获取的文档。于是打开提供数据的服务器，查看nginx日志终于发现了日志中存在786 worker_connections are not enough while connecting to upstream：
最后绝望的我去查看了站点的日志，发现果然出现503的时候请求量高的惊人，再仔细查看发现，这些请求都是广告请求！！原来出现worker_connections are not enough while connecting to upstream 不是什么tcp连接的问题，也不是nginx配置的问题，这些地方不会出问题的。出问题的地方要么是你的代码，要么就是真的有很多请求。把广告请求屏蔽就好了。
