Details
-
Type:
Improvement
-
Status:
Closed
-
Priority:
Normal
-
Resolution: Fixed
-
Affects Version/s: 3.0.3
-
Fix Version/s: 3.1.0
-
Labels:None
-
Environment:
opush
-
Rank:6572
Description
I got this exception in opush.log
10:07:25.551 [ERROR] {o.o.p.h.SyncHandler$$EnhancerByGuice$$572f1bd9} [t20026/cSync/r3125///] Cassandra timeout during write query at consistency QUORUM (1 replica were requ$
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (1 replica were required but only 0 acknowledged the write)
at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:54) ~[cassandra-driver-core-2.1.2.jar:na]
at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:259) ~[cassandra-driver-core-2.1.2.jar:na]
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:175) ~[cassandra-driver-core-2.1.2.jar:na]
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) ~[cassandra-driver-core-2.1.2.jar:na]
at org.obm.push.cassandra.dao.SyncedCollectionDaoCassandraImpl.put(SyncedCollectionDaoCassandraImpl.java:82) ~[push-dao-cassandra-3.0.3.jar:na]
at org.obm.breakdownduration.BreakdownDurationInterceptor.invoke(BreakdownDurationInterceptor.java:58) ~[breakdown-duration-3.0.1-alpha3.jar:na]
at org.obm.push.protocol.data.SyncAnalyser.getCollection(SyncAnalyser.java:120) ~[push-core-3.0.3.jar:na]
at org.obm.push.protocol.data.SyncAnalyser.analyseSync(SyncAnalyser.java:92) ~[push-core-3.0.3.jar:na]
at org.obm.push.handler.SyncHandler.analyzeRequest(SyncHandler.java:220) ~[push-core-3.0.3.jar:na]
at org.obm.push.handler.SyncHandler.process(SyncHandler.java:163) ~[push-core-3.0.3.jar:na]
at org.obm.push.handler.WbxmlRequestHandler.process(WbxmlRequestHandler.java:128) ~[push-core-3.0.3.jar:na]
at org.obm.push.technicallog.TechnicalLoggingInterceptor.invoke(TechnicalLoggingInterceptor.java:70) ~[technical-log-3.0.3.jar:na]
at org.obm.push.ActiveSyncServlet.processActiveSyncMethod(ActiveSyncServlet.java:253) ~[push-core-3.0.3.jar:na]
at org.obm.push.ActiveSyncServlet.doPost(ActiveSyncServlet.java:176) ~[push-core-3.0.3.jar:na]
at org.obm.push.technicallog.TechnicalLoggingInterceptor.invoke(TechnicalLoggingInterceptor.java:70) ~[technical-log-3.0.3.jar:na]
at org.obm.annotations.transactional.TransactionalInterceptor.managerTransaction(TransactionalInterceptor.java:87) ~[annotations-3.0.1-alpha3.jar:na]
at org.obm.annotations.transactional.TransactionalInterceptor.invoke(TransactionalInterceptor.java:73) ~[annotations-3.0.1-alpha3.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) [guice-servlet-3.0.jar:na]
at org.obm.breakdownduration.BreakdownDurationFilter.doFilter(BreakdownDurationFilter.java:71) [breakdown-duration-3.0.1-alpha3.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at org.obm.servlet.filter.qos.QoSFilter.handleRequest(QoSFilter.java:113) [qos-filter-3.0.1-alpha3.jar:na]
at org.obm.servlet.filter.qos.QoSFilter.doFilter(QoSFilter.java:94) [qos-filter-3.0.1-alpha3.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at org.obm.push.ActiveSyncRequestFilter.doFilter(ActiveSyncRequestFilter.java:78) [push-core-3.0.3.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
a
How can i increase a little bit the write timeout ?
Issue Links
- relates to
-
OP-190 Failsoft capacity when too many nodes are down to get the QUORUM
-
Activity
Hi, this exception exist for me since Cassandra 2.1. 2 Times i have it, but when I have it, i need to restart cassandra and opush, after all request will exit in timeout.
Yes Cassandra and opush are in the same machine, 4gb of ram. What is surprising it's since dsc21 and this server only host 1 user (like QA for me)
We know that the default "retry policy" of the cassandra driver does not fit perfectly Opush's needs, we have to write our own.
But I think that you have another problem as it is weird that you need to restart your cassandra instance, regular timeouts do not bring down the server.
Please verify why your Cassandra does not reply anymore, your OS may kill it to free up memory or something like that ..
Right with you, Opush is not directly responsable for that, but will be good to be able to increse a bit the write timeout (as i don't even know it ).
Occurs since Cassandra 2.1, memory still free when this issue occurs. What seems strange is after that, all request lead to a timeout. And of course, no log in cassandra
QA: to be validate on a Cassandra cluster configuration
Two retry policy are now available on the configuration of the cassandra client (/etc/opush/cassandra.ini file).
cassandra.retry-policy:
- RETRY: retry the request when a read, write or unavailable timeout is raised (number of retry is parameterized thru the cassandra.max-retries entry)
- DOWNGRADE: such as RETRY on read and write timeout, downgrade the consistency level on unavailable
In order to validate this issue, you may stop one of the cassandra node, or degrade the network on a cassandra node.
tc qdisc add dev eth0 root netem loss 50%
The logs (/etc/opush/opush.log file) should looks like:
- RETRY:
[INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 0) [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 1) [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 2)
- DOWNGRADE:
[INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on unavailable exception at consistency ONE (initial consistency: QUORUM, required replica: 2, alive replica: 1, retries: 0)
UNSTABLE: Integrated in opush-master #213
OP-137 Add entries in cassandra configuration related to RetryPolicy (thilaire: 14b3ca0a0c0ede69f2328ac864591af69378c331)
- integration-testing/opush/src/main/java/org/obm/opush/env/OpushStaticConfiguration.java
- push-dao-cassandra/src/test/java/org/obm/push/configuration/CassandraConfigurationFileImplTest.java
- push-dao-cassandra/src/main/java/org/obm/push/configuration/CassandraConfigurationFileImpl.java
- push-dao-cassandra/src/test/java/org/obm/push/cassandra/TestCassandraConfiguration.java
- push-interfaces/src/main/java/org/obm/push/configuration/CassandraRetryPolicy.java
- integration-testing/opush/src/main/java/org/obm/opush/env/OpushConfigurationFixture.java
- push-interfaces/src/main/java/org/obm/push/configuration/CassandraConfiguration.java
- push-server/src/configuration/etc/cassandra.ini
OP-137Add RetryPolicy in the configuration of Cassandra client (thilaire: 808815f49f07f46e0f157b0d2b93c3de11cd8f2e) - push-dao-cassandra/src/main/java/org/obm/push/cassandra/RetryNTimesRetryPolicy.java
- push-dao-cassandra/src/main/java/org/obm/push/cassandra/CassandraSessionSupplierImpl.java
- push-dao-cassandra/src/main/java/org/obm/push/cassandra/RetryOrCLDowngradeRetryPolicy.java
- push-dao-cassandra/src/test/java/org/obm/push/cassandra/RetryNTimesRetryPolicyTest.java
- push-dao-cassandra/src/test/java/org/obm/push/cassandra/RetryOrCLDowngradeRetryPolicyTest.java
- push-server/src/main/resources/logback.xml
OP-137Documentation on Cassandra retry policy (thilaire: b6cf35665ae743dfaed12c9e2fdc3de823d1b055) - push-server/src/main/asciidoc/administration.adoc
OK tested with a centos 6.6 and three Cassandra nodes:
- it reties on write timeout until that 'cassandra.max-retries" is reached
- it reties on read timeout until that 'cassandra.max-retries" is reached
- it reties on unavailable until that 'cassandra.max-retries" is reached
chenlina20180320
http://www.canadagoosecoatsoutlet.us.com
http://www.burberryoutlet-canada.ca
http://www.truereligionsale.com.co
http://www.uggsstoreoutlet.us.com
http://www.newbalance.me.uk
http://www.burberrysale.us.org
http://www.airmax97.org
http://www.supremeuk.co.uk
http://www.ray-banssunglasses.org.uk
http://www.katespade-outlet.in.net
http://www.airmaxnike.me.uk
http://www.prada.us.org
http://www.vancleefarpelsjewelry.us.com
http://www.airmax2018.in.net
http://www.jacketmoncler.co.uk
http://www.oakleysungalsses.nom.co
http://www.north--face.us.com
http://www.pandorajewelry.us.org
http://www.coachfactoryoutletcom.us.com
http://www.pandorajewelrysoutlet.us.com
http://www.ray-bans-sunglasses.us.com
http://www.huarache.us.com
http://www.vibram-fivefingers.us.com
http://www.toryburchhandbagsoutlets.us.com
http://www.louboutinoutlets.us.com
http://www.monclernederland.nl
http://www.michaelkors--outlet.us.com
http://www.stuartweitzmanshoes.us.com
http://www.cheap-jordans.us.com
http://www.stussy.us.com
http://www.uggoutletstore.us.org
http://www.stansmithadidas.org.uk
http://www.michael-korshandbagsoutlet.us.org
http://www.nikeukstore.org.uk
http://www.oakleyssunglassescheap.com
http://www.nikekyrie3.us.com
http://www.katespadeoutletsale.us.com
http://www.uggbuys.com
http://www.hermesbeltoutlet.com
http://www.baseballbats.us.com
http://www.runningshoes.us.org
http://www.thenorthfacecom.us.com
http://www.salvatoreferragamo.in.net
http://www.mishka.us.com
http://www.supreme-newyork.com
http://www.poloshirtsoutlet.us.com
http://www.barcelonajersey.us.com
http://www.ralphlaurenoutletofficial.us.com
http://www.supremeclothing.us.com
http://www.canadagoosejacket.me.uk
http://www.adidasshoess.us.com
http://www.basketballshoesstore.us.com
http://www.swarovski.us.org
http://www.ugg.co.nl
http://www.pradahandbagsoutlet.us.com
http://www.coachoutletonlinestore.ca
http://www.uggscanadaboots.ca
http://www.bottega.in.net
http://www.nikeairmaxuk.co.uk
http://www.oakleysunglassescom.us.com
http://www.nikeairforce1.net
http://www.jordan1.us
http://www.coachoutlet-handbags.us.com
http://www.poloralphoutlet.us.com
http://www.bestnfljerseyswholesale.us.com
http://www.tods.us.com
http://www.juicycoutureoutlet.us.com
http://www.pradabags.in.net
http://www.converseshoesoutlet.us.com
http://www.raybansunglassesforwomen.us.com
http://www.airmaxnike.nl
http://www.thenorthfaceoutlet.ca
http://www.pandorajewelrycanada.ca
http://www.salecanadagooseoutlet.us.com
http://www.miumiu.us.com
http://www.nikefactoryoutletstores.us.com
http://www.tomsoutlet.in.net
http://www.michaelkorsonline.ca
http://www.poloralphlaurenpascher.fr
http://www.burberryscarves.us.com
http://www.beatsheadphones.us.com
http://www.nikesb.in.net
http://www.wholesaleoakleysunglasses.us.org
http://www.cheapnhljerseysshop.us.com
http://www.nikepresto.us.com
http://www.michaelkorsonline-outlet.us.com
http://www.uggpas-cher.fr
http://www.moncler.eu.com
http://www.katespade-outlet.us.com
http://www.poloralph.us.com
http://www.cheapjordansshoes.us.com
http://www.cheapraybansunglass.co.uk
http://www.nikeairmax.me.uk
http://www.jordans.in.net
http://www.nikecortez.net
http://www.coachonline.us.org
http://www.northfaceoutletstore.us.org
http://www.christianlouboutinshoessale.org.uk
http://www.toryburchoutletcom.us.com
http://www.air-jordanpascher.fr
http://www.nikelunarglide.us
http://www.superdry.me.uk
http://www.pandora-jewelry.name
http://www.kanyeyeezyshoes.us.com
http://www.mbtscarpe.it
http://www.fitflops.us.org
http://www.truereligionoutletjeans.us
http://www.timberland.us.org
http://www.niketrainers.org.uk
http://www.adidasflipflops.us.com
http://www.shoecarnival.us.com
http://www.coachoutlet70off.us.com
http://www.louboutin.uk
http://www.monclersitoufficiale.it
http://www.cheapnfljerseys.us.org
http://www.toryburchoutletsaleonline.us.com
http://www.louboutin.in.net
http://www.canadagoosestore.us.com
http://www.jordan5.us
http://www.manchesterunitedjersey.us.com
http://www.the-northfacecanada.ca
http://www.mizuno.us.org
http://www.airmax2017.us.com
http://www.adidasyeezy-boost.us.com
http://www.adidasfootballboots.org.uk
http://www.ferragamooutlet.us.com
http://www.nikeflipflops.org
http://www.coachfactoryoutlet-online.eu.com
http://www.christianlouboutinsale.uk
http://www.fitflopsoutlet.us.com
http://www.coachoutlet-online.eu.com
http://www.kobe11.in.net
http://www.mbt.us.com
http://www.fitflopsandals.org
http://www.canadagoosedownjackets.us.com
http://www.giuseppe-zanotti.us.com
http://www.uggstore.us.com
http://www.nikeblazers.us
http://www.coachhandbagsfactoryoutletonline.us.com
http://www.jordans-fusion.com
http://www.cheaprolexwatches.us.com
http://www.adidas.us.org
http://www.coachoutletstoreonlineofficial.us.com
http://www.pandoraukcharms.co.uk
http://www.christianlouboutinoutletuk.org.uk
http://www.nikejordans.co.uk
http://www.nikeblazer.fr
http://www.discountnike.us.com
http://www.raybansunglassesoutlet.net.co
http://www.coachwallets.us.com
http://www.cartierjewelry.us.com
http://www.longchamp-outlet.us.org
http://www.cheapmichaelkorshandbags.us.org
http://www.ralph-lauren.com.co
http://www.northfacethe.us.com
http://www.nikeoutletstoreonline.us.com
http://www.nikeshox.in.net
http://www.ukmoncler.org.uk
http://www.bcbg.us.com
http://www.northfacejackets.us.org
http://www.michael-korsoutletstores.us.org
http://www.mcmbackpack.us.com
http://www.airjordan11.us
http://www.dansko.us.com
http://www.monclerdunjacka.se
http://www.michaelkorsoutlet70off.us.com
http://www.birkenstockshoes.org.uk
http://www.coachoutlet-factory.us.com
http://www.redbottomsshoes.me.uk
http://www.airmaxplus.org
http://www.airforce1.in.net
http://www.adidasjeremyscott.in.net
http://www.airjordanretro.fr
http://www.oakleysunglasshut.us.com
http://www.ralphlauren-poloshirts.co.uk
http://www.redbottoms.us.org
http://www.michael--kors.me.uk
http://www.baseballjersey.us.com
http://www.moncler-outletonline.us.com
http://www.lebron15.in.net
http://www.asics.us.org
http://www.burberryoutletonline.us.org
http://www.nikeairmaxschweiz.ch
http://www.korsoutlet.us.org
http://www.jordanretro.us
http://www.ralphlauren.in.net
http://www.michael-korstaschen.ch
http://www.adidaswings.name
http://www.longchamp-outletstore.us.com
http://www.outletuggsale.us.com
http://www.flops.us.com
http://www.christian--louboutin.us.com
http://www.uggsitalia.it
http://www.katespadehandbagssale.us.org
http://www.cheapmlbjerseysshop.us.com
http://www.maccosmeticsofficial.us.com
http://www.adidasslides.us.com
http://www.pradaoutletonlinestore.us.com
http://www.balenciaga.us.org
http://www.swarovskijewelry.co.uk
http://www.adidasstoreuk.me.uk
http://www.coachfactoryoutlet-onlinestore.us.com
http://www.valentino.us.org
http://www.michael-kors-uk.me.uk
http://www.coach-bags.us.org
http://www.birkenstockoutlet.us.org
http://www.nikemercurial.us
http://www.montblanccom.us.com
http://www.oakley-sunglassesonsale.us.com
http://www.nikedunks.us
http://www.soccerjersey.us.com
http://www.authenticmlbjerseyscheap.us.com
http://www.coachfactory--outlet.us.com
http://www.outletsuggstores.us.com
http://www.weddingshoes.us.com
http://www.michaeljordan-shoes.com
http://www.rayban.net.co
http://www.canada-goosejassen.nl
http://www.ugg.com.co
http://www.korsmichaelkors.org.uk
http://www.niketennis.us.com
http://www.northfaceoutletonline.com
http://www.northfacecom.us.com
http://www.ralph-laurenpolouk.org.uk
http://www.fitflopssaleclearance.us.org
http://www.truereligionjeans-outlet.us.com
http://www.outletcanadagoosesale.us.com
http://www.uggstoresoutlet.us.com
http://www.canada--goose.fr
http://www.cheap-oakleysunglasses.com
http://www.pandorajewelry-canada.ca
http://www.outletralph-lauren.us.com
http://www.longchamp-handbags.us.org
http://www.nikeroshe.us.com
http://www.lightup-shoes.us.com
http://www.raysunglass.com
http://www.uggbootsclearance.net.co
http://www.nhlhockey-jerseys.us.com
http://www.converse.me.uk
http://www.airmax95.us.org
http://www.longchampoutletcom.us.com
http://www.dolceandgabbana.us.com
http://www.coachoutletonsale.us.com
http://www.northfacejacketsclearance.us.com
http://www.coachoutletsale.us.com
http://www.hervelegerdresses.us.com
http://www.katespadehandbagsoutlet.us.org
http://www.poloralphlaurenoutlet.org.uk
http://www.snapbacks-wholesale.us.com
http://www.uggaustralia.org.uk
http://www.jordanshoes.us.org
http://www.cheapairjordans.us.com
http://www.nikeairmax90.us.org
http://www.ralphlaurenpolo.in.net
http://www.oksunglasshut.com
http://www.uknorthface.org.uk
http://www.truereligion-jeans.us
http://www.uggfactoryoutletstore.us.com
http://www.tory-burch.org.uk
http://www.uggsoutletcom.us.com
http://www.airhuarache.fr
http://www.moncleroutletstore.us.com
http://www.thenorthfaceuk.co.uk
http://www.wwwnorthface.us.com
http://www.ralplaurenuk.org.uk
http://www.hermeskellybag.us.com
http://www.airjordan-uk.co.uk
http://www.michaelkorscom.us.com
http://www.redbottomshoes.org.uk
http://www.michaelkorsoutletmall.us.com
http://www.adidasshoesoutletstore.us.com
http://www.burberry-outlet-stores.us.com
http://www.pandoracharmjewelry.us.com
http://www.michaelkorsoutletoff.us.com
http://www.canadagooses.org
http://www.christian--louboutin.us
http://www.mulberryukhandbags.org.uk
http://www.tommyhilfigeroutlet.ca
http://www.nikeairmax90.com.de
http://www.nfl--jerseys.us.com
http://www.rayban.in.net
http://www.nikeoff.com
http://www.uggschweiz.ch
http://www.cheap-jordan-shoes.us.com
http://www.coach-factoryonline.us.org
http://www.nikefootballbootsuk.org.uk
http://www.coachcom.us.com
http://www.sunglassesoakleycheap.us.com
http://www.belstaffoutlet.us.com
http://www.fitflopssale.org.uk
http://www.newcoachoutlet.us.com
http://www.canada--goose.net
http://www.ralph-laurens.co.uk
http://www.bottesuggs.fr
http://www.linksoflondonuk.me.uk
http://www.longchampsacsoldes.fr
http://www.uggclearanceoutlet.us.com
http://www.ray-bansunglassessale.us.org
http://www.jordanxx9.in.net
http://www.jordan12.org
http://www.asics.org.uk
http://www.moncler-jacken.at
http://www.northfaceclearance.us.com
http://www.monclerdoudounepascher.fr
http://www.uggcanadaoutlet.ca
http://www.michael-korsoutletonline.us.org
http://www.airjordan-14.us
http://www.cheapjordansshoes.in.net
http://www.montblancpensstore.us.com
http://www.niketn.fr
http://www.marcjacobs.me.uk
http://www.nikefree.me.uk
http://www.katespadestoreoutlet.us.com
http://www.polooutletinc.us.com
http://www.moncler--paris.fr
http://www.coachsoutletstoreonline.us.com
http://www.jacketsmoncler.us.com
http://www.coachoutletstore.eu.com
http://www.wholesalenfl-jerseys.us.com
http://www.uggclearance.us.com
http://www.alexandermcqueen.us.com
http://www.goedkope-uggs.be
http://www.monclerjackenschweiz.ch
http://www.supremeparis.fr
http://www.coachoutletonline70off.us.com
http://www.cheapraybans.ca
http://www.undefeated.us.com
http://www.kevindurantshoes.us.com
http://www.birkenstocksandals.us.org
http://www.uggbootsoutlet.co.uk
http://www.hermesoutlets.us.com
http://www.rayban-sunglassesonsale.us.com
http://www.jerseys.us.org
http://www.airmax.us.org
http://www.cheapmlbjersey.us.com
http://www.oakleysunglasses-vault.us.com
http://www.hermesbags.me.uk
http://www.michaelkorsoutlet.us
http://www.louboutin.us.com
http://www.bvlgari.us.com
http://www.stephen-curryshoes.us.com
http://www.michaelkorshandbagsoutletsales.us.com
http://www.nikefreeruns.us.com
http://www.ralph-lauren.fr
http://www.underarmouroutletclearance.com
http://www.stansmithadidas.us.com
http://www.cheapnfljerseysstore.us.com
http://www.newbalance-shoes.us.com
http://www.uggscanadastore.ca
http://www.hardenvol1.us.com
http://www.suprashoesclearance.us.com
http://www.oakleysunglassescanada.ca
http://www.airmax1.us
http://www.nikesshoes.us.com
http://www.fingerlingsmonkey.org
http://www.nikerevolution.us
http://www.nikefree-run.fr
http://www.blackhawksjersey.us.com
http://www.adidassuperstar.name
http://www.nike-factoryoutlet.us.com
http://www.timberlandoutlet.us.org
http://www.nikefree.net
http://www.airmax270.org
http://www.cheapjerseys.in.net
http://www.nike-airhuarache.me.uk
http://www.ferragamoshoesoutlet.us.com
http://www.givenchyhandbags.us.com
http://www.truereligion-jeans.me.uk
http://www.north-face-outlet.in.net
http://www.monclerjassen.be
http://www.converseshoes.us.org
http://www.christian-louboutins.fr
http://www.ray-bansunglassesoutlet.com.co
http://www.timberlandoutlet.org.uk
http://www.manoloblahnik.us
http://www.chaussuresnikeairmax.fr
http://www.cheapbestnbajerseys.us.com
http://www.northfacejacket.us.com
http://www.christiandiorhandbags.us.com
http://www.cheapreplicawatches.us.com
http://www.nikefactory-outlet.us.com
http://www.longchampoutlet-online.com
http://www.jimmychoooutlet.us.com
http://www.timberlandpro.us.com
http://www.montblanc.com.co
http://www.moncleroutlet.us.org
http://www.adidasoutletstores.com
http://www.cheapraybansunglasses.net.co
http://www.hermesbirkinbags.us.com
chenlina20180320
http://www.ralphlauren.mex.com/, http://www.airhuarache-inc.co.uk/, http://www.scarpehogan-shoes.it/, http://www.tommys-hilfigers.net/, http://www.newbalance-canada.ca/, http://www.converse-shoes-outlet.com/, http://www.birkenstock.net.co/, http://www.cheapjerseys-outlet.com/, http://colts.nfl-jerseys.org/, http://www.thenorth-face.com.co/, http://www.airmax.com.se/, http://www.raybansbocco.it/, http://www.rolexswatchoutlet.com/, http://www.oakleyssunglasslrc.com/, http://www.mk-com.com/, http://www.hollister-clothings.us.com/, http://www.coachoutlet.fr/, http://www.soccers-shoes.us.com/, http://www.nhljerseys.com.co/, http://www.kate-spades.net/, http://www.australia-michaelkors.com/, http://lakers.nbajerseyssale.com/, http://www.michael-korsbags.com.co/, http://www.mcmbackpacks.com.co/, http://www.cheap-watches.com.co/, http://www.hollister.com.de/, http://clippers.nbajerseyssale.com/, http://www.ralphlaurens.us.com/, http://www.new-balanceshoes.com/, http://www.adidas-shoes.co.nl/, http://www.prada-outlet.net.co/, http://www.timberland-shoes.net.co/, http://www.supra-footwears.com/, http://www.ok-em.com/, http://www.katespades.com.co/, http://www.giuseppes-zanottishoes.com/, http://www.belstaff-jackets.us.com/, http://www.mlbjerseys.com.co/, http://www.oakley-outlet.fr/, http://www.salvatoreferragamo.us.com/, http://www.belstaff-outlet.com/, http://www.louboutinshoes.jp.net/, http://www.jackwolfskinoutlet.ca/, http://www.nike-shoes.com.co/, http://www.nikefree5.com.co/, http://www.replicabags.net.co/, http://www.jewelrys-swarovski.com/, http://www.gucci-shoesoutlet.com/, http://www.coach-factoryoutlet.net.co/, http://magic.nbajerseyssale.com/, http://www.raybans-cher.fr/, http://www.ed-hardy.us.com/, http://www.nike-factory.com.co/, http://www.kate-spadebags.com/, http://www.uggs-boots.com/, http://www.the-northfacejackets.net.co/, http://grizzlies.nbajerseyssale.com/, http://cavaliers.nbajerseyssale.com/, http://www.burberbags.com/, http://www.timberlandboots-outlet.com/, http://www.guccishoes.net.in/, http://www.swarovskionlineshop.com.de/, http://www.cheap-omegaswatches.com/, http://www.adidasschuhe-online.com.de/, http://www.coachblackfriday.com/, http://www.nikeroshe-run.com.es/, http://www.cheap-michaelkors.com.co/, http://www.oakleys-online.com.co/, http://www.nikeairmax-shoes.fr/, http://www.burberrys-outletonline.in.net/, http://www.airjordans-shoes.com/, http://www.jerseys-store.com/, http://www.poloralphlaurens.com.de/, http://www.michael-kors.com.es/, http://giants.nfl-jerseys.org/, http://www.nikeairmax-australia.com/, http://www.babylisspros.com/, http://www.coach-factorys.com/, http://www.new-iphonecases.com/, http://www.nikeair-max.es/, http://celtics.nbajerseyssale.com/, http://www.coach-factory.com.co/, http://www.versaces-outlet.com/, http://www.barbours-factory.com/, http://www.air-yeezyshoes.com/, http://coachoutlet.euro-us.net/, http://www.nikeairforce.com.de/, http://www.jerseys.net.in/, http://www.raybanoutlet.ca/, http://www.givenchy.com.co/, http://www.salomonshoes.com.de/, http://www.nikeair-max.ca/, http://www.airmax-90.com/, http://www.timberland-shoes.fr/, http://www.long-champ-outlet.com/, http://www.hermesbagsoutlet.com/, http://www.juicy-coutureoutlet.com/, http://www.rolex-watchescanada.ca/, http://www.fidgetspinner.com.co/, http://www.outletonline-michael-kors.co.uk/, http://www.salvatore-ferragamos.com/, http://www.polos-ralphlauren.com.co/, http://www.bcbg-dresses.net/, http://www.nikerosherun.com.co/, http://www.moncler-jackets.com.co/, http://www.michaelkors.so/, http://www.nfljerseys-sale.com/, http://www.converses.us.com/, http://www.nikefreerun-inc.co.uk/, http://www.uhren-stores.de/, http://www.vibram--fivefingers.com/, http://www.pandora-jewelry.top/, http://www.stone-islandestore.com/, http://www.newbalanceschuhe.com.de/, http://www.hermes-birkinoutlet.com/, http://www.woolrich-jackets.com/, http://www.christianlouboutin-shoes.ca/, http://www.hollister.com.se/, http://www.adidasshoes.com.se/, http://bears.nfl-jerseys.org/, http://www.ralphlaurens.co.uk/, http://www.christian-louboutinshoesoutlet.com/, http://www.the-northface.com.de/, http://www.burberry-outlets.ca/, http://www.nikeshoes.de/, http://www.supra-footwear.net/, http://www.true-religionsoutlet.com/, http://www.dsquared2-outlet.com/, http://www.nike-huarache.co.nl/, http://www.eyeglassesinsale.com/, http://www.longchamps-bags.us.com/, http://bucks.nbajerseyssale.com/, http://www.coachfactory.cc/, http://lions.nfl-jerseys.org/, http://www.christian-louboutins.us.com/, http://packers.nfl-jerseys.org/, http://www.cheap-montblancpens.com/, http://www.truereligions.net/, http://www.tommy-hilfigers.de/, http://www.fred-perrys.co.uk/, http://www.hoodiesoutlet.net/, http://www.michaelkorspurses.co.uk/, http://patriots.nfl-jerseys.org/, http://www.cheap-mlbjerseys.us.com/, http://76ers.nbajerseyssale.com/, http://www.mcm-bagsoutlet.com/, http://www.jordan-retros.com/, http://www.guci-heaven.com/, http://warriors.nbajerseyssale.com/, http://www.coachoutletonlinemall.com/, http://www.ralph-laurenoutlets.com/, http://www.designer-bags.us.com/, http://www.cheapshoes.net.co/, http://vikings.nfl-jerseys.org/, http://www.kate-spade-outlet.us.com/, http://www.nike-paschers.fr/, http://www.nike-free-runs.de/, http://www.nikefrees-shoes.co.uk/, http://saints.nfl-jerseys.org/, http://www.barboursjackets.com/, http://texans.nfl-jerseys.org/, http://www.linksoflondonvip.com/, http://www.lacosteoutlet.com.co/, http://www.hugo-boss.com.co/, http://www.swarovskijewellery-au.com/, http://www.canada-michaelkors.ca/, http://www.rosherun-shoes.us/, http://www.armaniexchange.us.com/, http://www.tracksuits.com.co/, http://www.jeans-truereligion.com/, http://www.oakley-sbocco.it/, http://www.tomsshoes-outlet.us.org/, http://www.nike-freeruns.com/, http://seahawks.nfl-jerseys.org/, http://www.skechers-outlet.com/, http://www.adidas-shoes.com.co/, http://www.rayban-lunettes.fr/, http://www.lonchamp.us.com/, http://www.fredperrypolos.co.uk/, http://spurs.nbajerseyssale.com/, http://www.reeboks.com.de/, http://www.tods-shoesoutlet.com/, http://www.rayban-sunglasses.co/, http://www.oakleysunglasses.org.es/, http://www.skechers-shoesoutlet.com/, http://www.oakleys-outlets.net/, http://www.cheap-juicycouture.com/, http://www.coachoutlet-online.com.co/, http://www.burberry-storevip.com/, http://www.ferragamos.us.com/, http://www.nike-store.com.de/, http://www.oakleys.com.de/, http://www.adidassuper-star.com.de/, http://www.converse-schuhe.com.de/, http://www.raybans-glasses.com.co/, http://ravens.nfl-jerseys.org/, http://www.nikeshoes.mex.com/, http://www.horlogesrolexs.co.nl/, http://www.ferragamo-inc.co.uk/, http://www.omegaswatches.com/, http://www.barbours.us.com/, http://www.belstaffjacketsoutlet.co.uk/, http://www.michael-kors.net.co/, http://titans.nfl-jerseys.org/, http://www.versaceoutlet.us.com/, http://www.the-north-face-jackets.com/, http://www.raybansasses.com/, http://www.marcjacobs-outlet.com/, http://www.ray-bans.net.co/, http://www.converse-shoes.net/, http://www.outlet-michaelkors.com/, http://bulls.nbajerseyssale.com/, http://www.under-armouroutlet.com/, http://jazz.nbajerseyssale.com/, http://www.burberryhandbagsvip.com/, http://www.cheap-jordans.com.co/, http://www.crystals-swarovskis.com/, http://www.oakleyframeslrc.com/, http://nuggets.nbajerseyssale.com/, http://jets.nfl-jerseys.org/, http://www.oakley-outletonline.com.co/, http://www.nike-shoesoutlet.com/, http://www.coachoutletonline.vip/, http://www.pandora-jewellery.com.es/, http://www.barbour-jackets.us.com/, http://www.tommyhilfiger-outlet.net/, http://chiefs.nfl-jerseys.org/, http://www.christian-louboutin.jp.net/, http://eagles.nfl-jerseys.org/, http://www.adidas-shoesoutlet.com/, http://www.burberoutlet.com/, http://www.toms-shoe.us.org/, http://falcons.nfl-jerseys.org/, http://www.cheap-pandoracharms.org.uk/, http://cowboys.nfl-jerseys.org/, http://www.fitflop-outlet.com/, http://www.philipppleins.us.com/, http://www.tommyhilfigerca.ca/, http://www.ralphslauren-polo.com/, http://www.nfl-jerseys.org/, http://www.asicsoutlet.net/, http://www.jordan.com.de/, http://www.outlet-michaelskors.com/, http://hawks.nbajerseyssale.com/, http://www.ralph-laurenpolos.com/, http://www.burberroutlet.co.uk/, http://www.handbags-outlet.in.net/, http://www.nikeairmax-isc.co.uk/, http://hornets.nbajerseyssale.com/, http://www.reebok-shoes.co.uk/, http://www.burberry-store.com/, http://www.vansschuhe.com.de/, http://www.new-balanceoutlet.org/, http://www.thomas-sabo.us.com/, http://www.outlet-true-religion.com/, http://www.nikerosherun.com.de/, http://www.montblancoutlet.com/, http://www.outlet-pradabags.com/, http://www.long-champ.com.co/, http://www.michaelkors-ins.com/, http://www.the-northface.in.net/, http://buccaneers.nfl-jerseys.org/, http://www.oakleys-lrc.com/, http://www.the-north-face.net.co/, http://www.nikeshoesoutlet.co.nl/, http://www.outlet-pumashoes.com/, http://www.northface-jackets.co.uk/, http://www.raybansoutlet.co.uk/, http://www.eccoshoesoutlet.com/, http://www.fendi-outlet.us.com/, http://www.cc-beanies.us.com/, http://www.nbajerseyssale.com/, http://www.tommyhilfigers.co.nl/, http://www.mbt-shoesoutlet.com/, http://dolphins.nfl-jerseys.org/, http://www.okglassesale.com/, http://coach.euro-us.net/, http://panthers.nfl-jerseys.org/, http://www.outlet-pradas.com/, http://www.oakley-sunglassessale.com.co/, http://www.nike-maxshoes.fr/, http://www.chiflatironhair.us.com/, http://azcardinals.nfl-jerseys.org/, http://www.bcbg-canada.ca/, http://www.birkenstocks.com.de/, http://pelicans.nbajerseyssale.com/, http://www.toms-shoesoutlets.com/, http://www.cheapnhljerseys.us.com/, http://www.outlet-ralphlaurens.com/, http://www.pandorajewellery.com.es/, http://www.co-aol.com/, http://www.pandora-jewelry.com.de/, http://www.marc-jacobs.us.com/, http://www.thomas-sabos.com.de/, http://www.chrome-heartsoutlet.com/, http://www.outlet-toryburchshoes.com/, http://www.mizuno-running.net/, http://www.ralphlauren-au.com/, http://www.monclers-outlet.us.com/, http://raptors.nbajerseyssale.com/, http://www.skechers-canada.ca/, http://www.dsquared2s.com/, http://www.cheap-raybansoutlet.com.co/, http://www.nike-shoesoutlet.us.org/, http://www.rolexwatchesforsale.us.com/, http://www.ralph-laurenspolos.com/, http://www.rl-ralphlaurenpolos.co.uk/, http://www.michael-kors-bags.com/, http://www.oakleys.mex.com/, http://www.nike-shoes.org.es/, http://www.michael-kors.co.nl/, http://www.christian-louboutin-shoes.com/, http://www.the-north-face.com.co/, http://www.michael-korsoutletonline.com.co/, http://www.vansshoes-outlet.com/, http://www.nike-mercurial.com/, http://www.bottega-outlet.com/, http://www.prada-shoes.us.org/, http://www.levis-jeansoutlet.com/, http://www.outletburberry-bags.com/, http://www.guessfactorys.com/, http://www.toms-shoesstore.com.co/, http://www.hollister-outlet.com/, http://www.jack-wolfskin.us.com/, http://pistons.nbajerseyssale.com/, http://www.mkoutlet.com.co/, http://www.outlet-tommyhilfigers.com/, http://www.adidas-shoes.ca/, http://kings.nbajerseyssale.com/, http://chargers.nfl-jerseys.org/, http://www.raybans-outlet.co.nl/, http://www.new-balanceshoes.es/, http://thunder.nbajerseyssale.com/, http://www.airmax-shoes.com/, http://www.ralphslauren-polo.com.co/, http://trailblazers.nbajerseyssale.com/, http://www.polosoutlets-store.com/, http://www.nike-airmax.in.net/, http://www.cheap-baseballbats.us/, http://www.softballbats-sale.com/, http://browns.nfl-jerseys.org/, http://www.ralphlauren-canada.ca/, http://www.long-champhandbags.com/, http://www.basketballshoes.com.co/, http://www.pandora-charmscanada.ca/, http://www.hollisterclothingcanada.ca/, http://www.hollister-clothings-store.com/, http://www.northfacejackets.fr/, http://www.valentino-shoesoutlet.us.com/, http://www.retrojordan-shoes.com/, http://www.birkenstockshoes.com.de/, http://www.toryburchs.us.com/, http://pacers.nbajerseyssale.com/, http://www.christianlouboutinoutlet.net.co/, http://www.nikeshoes-canada.ca/, http://www.oakleysunglasses-canada.ca/, http://www.coachoutlet-coachoutletonline.com/, http://www.sac-hermes.fr/, http://www.celinefactory.com/, http://timberwolves.nbajerseyssale.com/, http://www.christian-louboutins.com/, http://rockets.nbajerseyssale.com/, http://www.adidas-shoes.es/, http://www.swarovski-jewelrys.co.uk/, http://www.outlet-michaelkors-online.com/, http://www.raybans.com.de/, http://www.nbateamonline.com/, http://www.soccershoes-outlet.com/, http://www.gbyguess.us.com/, http://wizards.nbajerseyssale.com/, http://knicks.nbajerseyssale.com/, http://michaelkors.euro-us.net/, http://www.oakleys-sunglasses.com.co/, http://broncos.nfl-jerseys.org/, http://www.coach-outlets.net.co/, http://www.lacoste-polos.us.com/, http://raiders.nfl-jerseys.org/, http://www.mcmshandbags.com.co/, http://www.air-maxschoenen.co.nl/, http://www.nike-airmax.com.de/, http://www.thomassabos-jewelrys.org.uk/, http://www.vans-shoesoutlet.co.uk/, http://www.barbour-jackets.in.net/, http://steelers.nfl-jerseys.org/, http://www.nike-rosherun.co.nl/, http://www.ferragamo.net.co/, http://www.ralph-laurens-polo.co.uk/, http://www.ray-bansoutlet.net.co/, http://www.longchamp-outlet.com.co/, http://www.calvin-kleinoutlet.com/, http://www.nike-skos.com/, http://www.oakestore.com/, http://www.mcm-handbags.com.co/, http://www.polo-ralphlaurens.com.co/, http://www.ray-bans.org.es/, http://www.pradas.com.de/, http://www.long-champ.com.de/, http://www.mcm-backpacks.com.co/, http://www.under-armour.us.com/, http://www.burberrbags.co.uk/, http://www.true-religions.com/, http://www.burberryoutletonline.com.de/, http://www.burbagsoutlet.com/, http://www.rayban-wear.co.uk/, http://www.cheap-jerseys.mex.com/, http://www.polo-ralphlaurens.us.com/, http://www.watch-rolex.co.uk/, http://www.nba-shoes.com/, http://www.michaelkorsoutlet.se/, http://redskins.nfl-jerseys.org/, http://www.airmax2015.net/, http://nets.nbajerseyssale.com/, http://www.chiflatiron-hair.com/, http://www.ralphlaurens.com.de/, http://www.nike-shoes-outlet.co.uk/, http://www.michaels-kors.co.uk/, http://www.soccershoes.us.com/, http://www.cheap-iphone-cases.com/, http://www.philipp-pleins.org/, http://www.airhuarache-shoes.co.uk/, http://rams.nfl-jerseys.org/, http://www.coachoutlet-usa.com/, http://www.michaelkors.com.de/, http://www.the-northface.ca/, http://www.pandora-jewelrys.com/, http://bills.nfl-jerseys.org/, http://www.bottega-venetasoutlet.com/, http://www.nikeairmax.com.de/, http://www.giuseppezanotti.com.co/, http://www.hollister-store.com.co/, http://www.rolex-watches.us.com/, http://www.swarovski-canada.ca/, http://www.coach-outletonline.ca/, http://mavericks.nbajerseyssale.com/, http://www.jimmychoos-shoes.com/, http://www.oakleysoutlet.com.co/, http://heat.nbajerseyssale.com/, http://www.nike-shoes.com.se/, http://www.michaelkorsoutlet.online/, http://www.outlet-ralphlauren.co.uk/, http://www.mizuno.us.com/, http://www.danskoshoes.in.net/, http://www.asics-shoesoutlet.net/, http://www.outlet-true-religion.us.org/, http://www.raybans-outlet.net.co/, http://www.hogans.com.de/, http://jaguars.nfl-jerseys.org/, http://www.replica-bags.us.com/, http://www.burberryoutlet-sale.net.co/, http://www.hoganshoesoutlet.co.uk/, http://www.jimmychoo-shoes.com/, http://www.omegas-relojes.es/, http://www.australia-uggs.com/, http://www.hermesbags-outlet.com/, http://www.puma-shoes.com.de/, http://www.tommyhilfigers.de/, http://www.ray-bans.com.co/, http://49ers.nfl-jerseys.org/, http://www.rosherun-shoes.co.uk/, http://www.toms-outletonline.com.co/, http://www.toms--outlet.com.co/, http://suns.nbajerseyssale.com/, http://bengals.nfl-jerseys.org/, http://www.zccoachoutlet.com/, http://www.raybans-outlet.cc/, http://www.montrespascher.fr/, http://www.asicsshoes-outlet.com/, http://www.parajumpersjacketsoutlet.com/, http://www.lauren-ralphs-outlet.co.uk/, http://www.airjordanshoes.com.co/, http://www.michaelkorsoutletvip.com/, http://www.oakleyslrc.com/, http://www.timberland-boots.com.co/
Thanks you for reporting your issue, can I know if this exception appears only once, sometimes or often ?
Can you also tell us some informations about your Cassandra host ? What's the machine configuration ? Does it hold all your OBM services ?