神刀安全网

ES-Spark连接ES后,ES Client节点流量打满分析

问题描述

前段时间用es-spark读取es数遇到了client节点流量打满的现象。es-spark配置的es.nodes是es的域名。由于其中一个client是master节点,然后普通查询变得特别慢,运行20多分钟后,主节点崩溃。

解决方法

临时解决方案:降低es-spark的并发,并重启主节点。

最终解决方案:设置es.nodes.wan.only为false,即不用域名访问。将es.nodes配置为client节点的IP。

原因分析

域名访问时必须配置参数es.nodes.wan.only为true,关于该参数的解释如下:

Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and onlyconnects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

es.nodes.wan.only设置为true时即只通过client节点进行读取操作,因此主节点负载会特别高,性能很差。长时间运行后,java gc回收一次要几十秒,慢慢的OOM,系统崩溃。

配置es.nodes为client节点的IP后,spark只通过data节点访问ES:

es.nodes.data.only (default true)
Whether to use Elasticsearch data nodes only. When enabled, elasticsearch-hadoop will route all its requests (after nodes discovery, if enabled) through the data nodes within the cluster. The purpose of this configuration setting is to avoid overwhelming non-data nodes as these tend to be “smaller” nodes. This is enabled by default.

es.nodes.data.only 默认为true,即spark所有的请求都会发到数据节点,不在通过client节点进行请求的转发,client节点只用来服务普通的查询。

源码角度分析

1、es-spark 读

其架构图如下所示:

ES-Spark连接ES后,ES Client节点流量打满分析

es_spark_read.png

我们知道spark能动态的发现节点,,但当我们配置wan.only为true的时候,整个集群的节点IP中只有从域名中解析出来的IP:

private static List<String> qualifyNodes(String nodes, int defaultPort, boolean resolveHostNames)  {    List<String> list = StringUtils.tokenize(nodes);    for (int i = 0; i < list.size(); i++)    {      String nodeIp = resolveHostNames ? resolveHostToIpIfNecessary((String)list.get(i)) : (String)list.get(i);      list.set(i, qualifyNode(nodeIp, defaultPort));    }    return list;  } 

从源码角度以scroll为例:

JavaEsSpark.esJsonRDD()–>JavaEsRDD.compute()–>JavaEsRDDIterator(继承AbstractEsRDDIterator).reader$lzycompute()
在lzycompute方法中我们可以看到,执行请求的是RestService:

private ScrollQuery reader$lzycompute()  {    synchronized (this)    {      if (!this.bitmap$0)      {        initialized_$eq(true);        Settings settings = this.partition.settings();         initReader(settings, log());         RestService.PartitionReader readr = RestService.createReader(settings, this.partition, log());this.reader =          readr.scrollQuery();this.bitmap$0 = true;      }      return this.reader;    }  } 

在createReader方法中会判断spark节点和当前请求请求的shard是否是同一个节点,如果是同一个节点,则将该IP写入Setting,用本地节点IP进行请求(执行请求的时候,从setting中读取该ip):

if ((!SettingsUtils.hasPinnedNode(settings)) && (partition.getLocations().length > 0)) {   String pinAddress = checkLocality(partition.getLocations(), log);   if (pinAddress != null)   {     if (log.isDebugEnabled()) {       log.debug(String.format("Partition reader instance [%s] assigned to [%s]:[%s]", new Object[] { partition, pinAddress }));     }     SettingsUtils.pinNode(settings, pinAddress);   } } 

通过PartitionReader.scrollQuery()–>SearchRequestBuilder.build()–>RestRepository.scanLimit()–>ScrollQuery.hasNext()–>RestRepository.scroll()–>RestClient.execute()–>NetWorkClient.execute()–>Transport.execute()

其实我们看到的最终要的执行是在NetWorkClient中,他会打乱所有的数据节点,并从中选出一个节点用来通信,如下:

public NetworkClient(Settings settings, TransportFactory transportFactory) {   this.settings = settings.copy();   this.nodes = SettingsUtils.discoveredOrDeclaredNodes(settings);   this.transportFactory = transportFactory;    Collections.shuffle(this.nodes);//打乱排序   if (SettingsUtils.hasPinnedNode(settings))   {     String pinnedNode = SettingsUtils.getPinnedNode(settings);     if (log.isDebugEnabled()) {       log.debug("Opening (pinned) network client to " + pinnedNode);     }     this.nodes.remove(pinnedNode);     this.nodes.add(0, pinnedNode);   }   selectNextNode();    Assert.notNull(this.currentTransport, "no node information provided"); }   private boolean selectNextNode() {   if (this.nextClient >= this.nodes.size()) {     return false;   }   if (this.currentTransport != null) {     this.stats.nodeRetries += 1;   }   closeTransport();   this.currentNode = ((String)this.nodes.get(this.nextClient++));   SettingsUtils.pinNode(this.settings, this.currentNode);   this.currentTransport = this.transportFactory.create(this.settings, this.currentNode);   return true; } 

2、es-spark 写

其架构图如下所示:

ES-Spark连接ES后,ES Client节点流量打满分析

es_spark_write.png

从源码角度来看:
写请求的时候,如果wan.only配置为true,则节点IP就是从域名解析出的IP中随机选择一个进行写操作。

if (settings.getNodesWANOnly()) {   return randomNodeWrite(settings, currentInstance, resource, log); } 

以bulk为例,其操作过程如下:

EsSpark.doSaveToEs()–>EsRDDWriter.write()–>RestService.createWriter()

在createWriter中首先随机或者按照split选择一个节点:

int selectedNode = currentSplit < 0 ? new Random().nextInt(nodes.size()) : currentSplit % nodes.size(); SettingsUtils.pinNode(settings, (String)nodes.get(selectedNode)); 

最终的改变是在RestService的initSingleIndex方法中,通过根据当前的split,找到对应的shard,然后获取到shard所在的IP,写入setting中(执行请求的时候,从setting中读取该ip)。

if (currentInstance <= 0) {    currentInstance = new Random().nextInt(targetShards.size()) + 1;  }  int bucket = currentInstance % targetShards.size();  ShardInfo chosenShard = (ShardInfo)orderedShards.get(bucket);  NodeInfo targetNode = (NodeInfo)targetShards.get(chosenShard);   SettingsUtils.pinNode(settings, targetNode.getPublishAddress());  String node = SettingsUtils.getPinnedNode(settings);  repository = new RestRepository(settings); 

接下来就是RestRepository.tryFlush()–>RestClient.bulk()–>NetWorkClient.execute()–>Transport.execute(),这一套流程和读差不多,这里就不再介绍。

3、shard-partition 对应关系

es-spark写的话就是就是一个partition对应一个shard,这里从上述的es-spark写代码中可以看出,不再过多介绍。

es-spark读的时候是按照shard的文档数来分的:

partition=numberOfDoc(shard)/100000

100000是默认的配置,这个可通过es.input.max.docs.per.partition配置。

假设一个shard有23w条doc,10w条一个partition,则分为3个partition。读操作时shard-partition 的架构图如下所示:

ES-Spark连接ES后,ES Client节点流量打满分析

partition_shard.png

从源码角度来说,如果是5.X版本,则用scrollSlice提高并发度。

if (version.onOrAfter(EsMajorVersion.V_5_X)) {   partitions = findSlicePartitions(client.getRestClient(), settings, mapping, nodesMap, shards); } else {   partitions = findShardPartitions(settings, mapping, nodesMap, shards); } 

在findSlicePartitions中给出了计算公式:

for (List<Map<String, Object>> group : shards) {   String index = null;   int shardId = -1;   List<String> locationList = new ArrayList();   for (Map<String, Object> replica : group)   {     ShardInfo shard = new ShardInfo(replica);     index = shard.getIndex();     shardId = shard.getName().intValue();     if (nodes.containsKey(shard.getNode())) {       locationList.add(((NodeInfo)nodes.get(shard.getNode())).getPublishAddress());     }   }   String[] locations = (String[])locationList.toArray(new String[0]);   StringBuilder indexAndType = new StringBuilder(index);   if (StringUtils.hasLength(types))   {     indexAndType.append("/");     indexAndType.append(types);   }   long numDocs = client.count(indexAndType.toString(), Integer.toString(shardId), query);   int numPartitions = (int)Math.max(1L, numDocs / maxDocsPerPartition);   for (int i = 0; i < numPartitions; i++)   {     PartitionDefinition.Slice slice = new PartitionDefinition.Slice(i, numPartitions);     partitions.add(new PartitionDefinition(settings, mapping, index, shardId, slice, locations));   } }  public int getMaxDocsPerPartition() {   return Integer.parseInt(getProperty("es.input.max.docs.per.partition", Integer.toString(100000))); } 

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » ES-Spark连接ES后,ES Client节点流量打满分析

分享到:更多 ()

评论 284

  1. #1

    Just what the doctor orrddee, thankity you!

    匿名3个月前 (05-23)
  2. #2

    This inoedtucrs a pleasingly rational point of view.

    匿名2个月前 (06-08)
  3. #3

    car insurance rates Spring Valley NY

    匿名2个月前 (06-09)
  4. #4

    auto owners insurance West Liberty KY

    匿名2个月前 (06-09)
  5. #5
  6. #6

    us agency car insurance Crossville TN

    匿名2个月前 (06-09)
  7. #7
  8. #8

    autoversicherung vergleich

    匿名2个月前 (06-10)
  9. #9

    kredite online berechnen

    匿名2个月前 (06-10)
  10. #10

    affordable auto insurance Lima OH

    匿名2个月前 (06-10)
  11. #11
  12. #12

    free car insurance quotes Wylie TX

    匿名2个月前 (06-11)
  13. #13

    list of auto insurances in Danbury CT

    匿名2个月前 (06-11)
  14. #14
  15. #15

    payless auto insurance Palm City FL

    匿名2个月前 (06-11)
  16. #16
  17. #17

    affordable car insurance Brighton CO

    匿名2个月前 (06-11)
  18. #18
  19. #19
  20. #20
  21. #21

    cheap auto insurance quotes Dublin CA

    匿名2个月前 (06-12)
  22. #22

    direct auto insurance Union NJ

    匿名2个月前 (06-12)
  23. #23

    auto insurance quotes Middletown NJ

    匿名2个月前 (06-12)
  24. #24

    finanzierungsrechner

    匿名2个月前 (06-12)
  25. #25

    cheap car insurance Cicero IL

    匿名2个月前 (06-12)
  26. #26

    auto insurance Traverse City MI

    匿名2个月前 (06-12)
  27. #27

    car insurance rates Williamstown NJ

    匿名2个月前 (06-12)
  28. #28

    car insurance in Port Saint Lucie FL

    匿名2个月前 (06-12)
  29. #29
  30. #30

    affordable auto insurance Savannah GA

    匿名2个月前 (06-13)
  31. #31

    car insurance Fairfield CA

    匿名2个月前 (06-13)
  32. #32
  33. #33
  34. #34
  35. #35

    look auto insurance Prairieville LA

    匿名2个月前 (06-14)
  36. #36

    us agency car insurance Reading PA

    匿名2个月前 (06-15)
  37. #37

    auto owners insurance Saint Paul MN

    匿名2个月前 (06-15)
  38. #38
  39. #39
  40. #40
  41. #41

    cheapest car insurance in Sandwich MA

    匿名2个月前 (06-16)
  42. #42

    car insurance rates West Jordan UT

    匿名2个月前 (06-16)
  43. #43
  44. #44

    car insurance quotes Bensalem PA

    匿名2个月前 (06-16)
  45. #45

    car insurance rates Winston Salem NC

    匿名2个月前 (06-16)
  46. #46

    direct auto insurance Lafayette IN

    匿名2个月前 (06-16)
  47. #47

    car insurance quotes Mooresville NC

    匿名2个月前 (06-16)
  48. #48

    car insurance

    匿名2个月前 (06-16)
  49. #49
  50. #50

    us agency car insurance Manhattan KS

    匿名2个月前 (06-17)
  51. #51
  52. #52

    car insurance in Rockville MD

    匿名2个月前 (06-17)
  53. #53
  54. #54

    list of car insurances in Elkton MD

    匿名2个月前 (06-17)
  55. #55

    cheapest auto insurance in Mason MI

    匿名2个月前 (06-17)
  56. #56
  57. #57
  58. #58

    free car insurance quotes Novi MI

    匿名2个月前 (06-18)
  59. #59

    low income car insurance Dundalk MD

    匿名2个月前 (06-18)
  60. #60
  61. #61

    affordable car insurance Franklin OH

    匿名2个月前 (06-18)
  62. #62

    direct auto insurance Greenville TX

    匿名2个月前 (06-19)
  63. #63

    cheap auto insurance Bethesda MD

    匿名2个月前 (06-19)
  64. #64
  65. #65
  66. #66
  67. #67

    payless auto insurance Kansas City MO

    匿名2个月前 (06-21)
  68. #68
  69. #69

    cheapest car insurance Sioux Falls SD

    匿名2个月前 (06-21)
  70. #70
  71. #71

    auto owners insurance Plano TX

    匿名2个月前 (06-21)
  72. #72

    list of car insurances in VA

    匿名2个月前 (06-21)
  73. #73
  74. #74

    auto owners insurance Escanaba MI

    匿名2个月前 (06-21)
  75. #75

    auto insurance rates Wichita Falls TX

    匿名2个月前 (06-22)
  76. #76
  77. #77

    auto insurance quotes Broken Arrow OK

    匿名2个月前 (06-22)
  78. #78

    cheapest car insurance Little Rock AR

    匿名2个月前 (06-22)
  79. #79

    auto insurance Sacramento CA

    匿名2个月前 (06-22)
  80. #80
  81. #81

    free auto insurance quotes Irwin PA

    匿名2个月前 (06-23)
  82. #82
  83. #83
  84. #84
  85. #85

    cheap auto insurance quotes Tucson AZ

    匿名1个月前 (06-23)
  86. #86
  87. #87
  88. #88

    payless auto insurance Gardena CA

    匿名1个月前 (06-24)
  89. #89
  90. #90

    affordable auto insurance Sun City CA

    匿名1个月前 (06-24)
  91. #91
  92. #92
  93. #93

    full coverage auto insurance Ocoee FL

    匿名1个月前 (06-24)
  94. #94

    car insurance quotes Fairfield OH

    匿名1个月前 (06-24)
  95. #95

    low income car insurance Lithonia GA

    匿名1个月前 (06-24)
  96. #96

    car insurance quotes Spring Hill FL

    匿名1个月前 (06-24)
  97. #97
  98. #98
  99. #99
  100. #100

    best auto insurance in Coatesville PA

    匿名1个月前 (06-24)
  101. #101
  102. #102
  103. #103

    low income auto insurance Pearl MS

    匿名1个月前 (06-25)
  104. #104
  105. #105

    best car insurance in Frisco TX

    匿名1个月前 (06-25)
  106. #106

    auto insurance Chagrin Falls OH

    匿名1个月前 (06-25)
  107. #107

    payless auto insurance Wichita KS

    匿名1个月前 (06-25)
  108. #108
  109. #109

    cheapest car insurance in Indio CA

    匿名1个月前 (06-25)
  110. #110

    direct auto insurance Cambridge MA

    匿名1个月前 (06-25)
  111. #111

    cheap car insurance Tulare CA

    匿名1个月前 (06-26)
  112. #112

    auto acceptance insurance Pacoima CA

    匿名1个月前 (06-26)
  113. #113

    look auto insurance Arlington VA

    匿名1个月前 (06-26)
  114. #114

    us agency car insurance Palm Bay FL

    匿名1个月前 (06-26)
  115. #115

    cheap auto insurance Oklahoma City OK

    匿名1个月前 (06-26)
  116. #116
  117. #117
  118. #118

    http://kreditrechnerco.info/

    匿名1个月前 (06-26)
  119. #119

    non owners auto insurance quotes DC

    匿名1个月前 (06-27)
  120. #120
  121. #121
  122. #122

    car insurance Zephyrhills FL

    匿名1个月前 (06-27)
  123. #123

    cheap auto insurance Sioux Falls SD

    匿名1个月前 (06-28)
  124. #124

    payless auto insurance Salisbury NC

    匿名1个月前 (06-28)
  125. #125
  126. #126
  127. #127
  128. #128
  129. #129

    auto insurance quotes Richmond VA

    匿名1个月前 (06-30)
  130. #130

    auto acceptance insurance Burbank CA

    匿名1个月前 (06-30)
  131. #131

    free car insurance quotes Olympia WA

    匿名1个月前 (06-30)
  132. #132

    cheapest auto insurance in KY

    匿名1个月前 (07-01)
  133. #133
  134. #134

    direct auto insurance Kyle TX

    匿名1个月前 (07-01)
  135. #135
  136. #136
  137. #137
  138. #138

    auto insurance Norman OK

    匿名1个月前 (07-03)
  139. #139

    auto insurance quotes Gloversville NY

    匿名1个月前 (07-03)
  140. #140
  141. #141
  142. #142

    http://immobilienkredit.club/

    匿名1个月前 (07-04)
  143. #143
  144. #144
  145. #145
  146. #146

    car insurance in North Las Vegas NV

    匿名1个月前 (07-05)
  147. #147

    immobilienfinanzierung

    匿名1个月前 (07-05)
  148. #148
  149. #149
  150. #150
  151. #151

    direct auto insurance Frankfort KY

    匿名1个月前 (07-06)
  152. #152

    direct auto insurance Pahoa HI

    匿名1个月前 (07-06)
  153. #153
  154. #154

    tadalafil en farmacias similares

    匿名1个月前 (07-07)
  155. #155

    köpa cialis flashback

    匿名1个月前 (07-07)
  156. #156

    online kredit deutschland

    匿名1个月前 (07-07)
  157. #157

    baufinanzierung zinsen

    匿名1个月前 (07-07)
  158. #158

    baufinanzierung zinsen

    匿名1个月前 (07-07)
  159. #159
  160. #160

    tilsvarer viagra over disk

    匿名1个月前 (07-08)
  161. #161
  162. #162

    billig cialis viagra

    匿名4周前 (07-08)
  163. #163

    viagra jeune fille

    匿名4周前 (07-09)
  164. #164
  165. #165
  166. #166

    viagra casera mujeres

    匿名4周前 (07-09)
  167. #167

    http://gesundermannliefert.info/

    匿名4周前 (07-10)
  168. #168

    viagra gevaarlijk euro

    匿名4周前 (07-10)
  169. #169

    Cialis yleinen hinta

    匿名4周前 (07-10)
  170. #170

    tadalafil cialis werking

    匿名4周前 (07-10)
  171. #171

    http://comprimespourhommes.top/

    匿名4周前 (07-11)
  172. #172

    venta de viagra punta arenas

    匿名4周前 (07-12)
  173. #173

    viagra patent danmark

    匿名4周前 (07-12)
  174. #174
  175. #175

    viagra kosten brief

    匿名4周前 (07-12)
  176. #176

    Cialis 5mg köpa online

    匿名4周前 (07-12)
  177. #177
  178. #178

    http://maanshaalsatabletter.top/

    匿名4周前 (07-12)
  179. #179

    http://baufinanzierung7x24.info/

    匿名4周前 (07-12)
  180. #180

    viagra cialis diferencia

    匿名4周前 (07-13)
  181. #181

    baufinanzierung vergleich

    匿名4周前 (07-13)
  182. #182

    erfaring med viagra natur

    匿名3周前 (07-14)
  183. #183

    http://pillerformenn.info/

    匿名3周前 (07-14)
  184. #184
  185. #185

    filmen der fyren selger viagra

    匿名3周前 (07-14)
  186. #186

    como tomar una viagra

    匿名3周前 (07-14)
  187. #187

    cialis online bestellen original

    匿名3周前 (07-14)
  188. #188

    partir pastilla de cialis

    匿名3周前 (07-15)
  189. #189

    http://pilloleperuomini.pro/

    匿名3周前 (07-15)
  190. #190

    viagra effet secondaire france

    匿名3周前 (07-15)
  191. #191

    con el viagra no eyaculas

    匿名3周前 (07-15)
  192. #192

    quand prendre viagra usa

    匿名3周前 (07-15)
  193. #193

    http://pillenvoormannen.info/

    匿名3周前 (07-16)
  194. #194

    viagra online free sample

    匿名3周前 (07-17)
  195. #195

    cialis pour une femme

    匿名3周前 (07-17)
  196. #196

    online kaufen soft viagra

    匿名3周前 (07-17)
  197. #197
  198. #198
  199. #199
  200. #200
  201. #201
  202. #202
  203. #203

    car insurance rates Columbia MD

    匿名3周前 (07-19)
  204. #204
  205. #205

    autoversicherung vergleich

    匿名3周前 (07-20)
  206. #206
  207. #207

    günstige kfz versicherung

    匿名3周前 (07-20)
  208. #208
  209. #209
  210. #210
  211. #211
  212. #212

    car insurance Rochester MI

    匿名2周前 (07-21)
  213. #213
  214. #214

    auto insurance Hialeah FL

    匿名2周前 (07-22)
  215. #215
  216. #216

    car insurance rates Cypress TX

    匿名2周前 (07-23)
  217. #217

    auto insurance quotes Mccomb MS

    匿名2周前 (07-23)
  218. #218

    rick hensley insurance

    匿名2周前 (07-23)
  219. #219

    rick hensley insurance

    匿名2周前 (07-23)
  220. #220

    auto insurance companies us

    匿名2周前 (07-24)
  221. #221
  222. #222

    classic car with cheap insurance

    匿名2周前 (07-26)
  223. #223

    aarp and budget car rental

    匿名2周前 (07-26)
  224. #224
  225. #225

    essurance quote

    匿名2周前 (07-27)
  226. #226
  227. #227

    trumble insurance

    匿名2周前 (07-27)
  228. #228

    motor vehicle insurance europe

    匿名1周前 (07-28)
  229. #229
  230. #230
  231. #231

    state workers insurance fund

    匿名1周前 (07-28)
  232. #232

    most popular car in the world

    匿名1周前 (07-28)
  233. #233

    insurance ireland car quote

    匿名1周前 (07-29)
  234. #234

    insurance auto insurance related items

    匿名1周前 (07-29)
  235. #235

    netherland insurance

    匿名1周前 (07-29)
  236. #236

    auto owners insurance howell mi

    匿名1周前 (07-29)
  237. #237
  238. #238

    car insurance oregon city

    匿名1周前 (07-29)
  239. #239

    insure a kid

    匿名1周前 (07-29)
  240. #240

    ofd car insurance

    匿名1周前 (07-29)
  241. #241

    quotes new york city

    匿名1周前 (07-30)
  242. #242
  243. #243

    new insurance online

    匿名1周前 (07-30)
  244. #244

    auto insurance washington pa

    匿名1周前 (07-30)
  245. #245
  246. #246

    the heritage insurance company

    匿名1周前 (07-31)
  247. #247
  248. #248
  249. #249
  250. #250

    auto insurance sylmar

    匿名7天前
  251. #251

    state wide insurance

    匿名6天前
  252. #252

    state wide insurance

    匿名6天前
  253. #253
  254. #254
  255. #255
  256. #256
  257. #257
  258. #258
  259. #259
  260. #260
  261. #261

    viagra ar cialis

    匿名3天前
  262. #262
  263. #263
  264. #264
  265. #265
  266. #266
  267. #267

    patente cialis auto

    匿名3天前
  268. #268

    wh0cd758372 Bupropion

    匿名2天前
  269. #269
  270. #270
  271. #271
  272. #272
  273. #273
  274. #274
  275. #275
  276. #276

    apotek viagra uden

    匿名2天前
  277. #277
  278. #278
  279. #279

    viagra stockist glasgow

    匿名24小时前
  280. #280
  281. #281

    generic viagra 50mg online

    匿名14小时前
  282. #282

    billig cialis viagra

    匿名12小时前
  283. #283

    wh0cd758372 BEST PRICE GENERIC TADALAFIL

    匿名5小时前
  284. #284

    wh0cd758372 strattera

    匿名5秒前