[OP-137] Customize RetryPolicy to avoid Cassandra timeouts Created: 23/Nov/14  Updated: 09/May/18  Resolved: 18/Mar/15

Status: Closed
Project: Opush
Component/s: None
Affects Version/s: 3.0.3
Fix Version/s: 3.1.0

Type: Improvement Priority: Normal
Reporter: Carnus Jérémy Assignee: Antoine DUPRAT
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

opush


Issue Links:
Relates
relates to OP-190 Failsoft capacity when too many nodes... Closed
Rank: 6572

 Description   

I got this exception in opush.log

 10:07:25.551 [ERROR] {o.o.p.h.SyncHandler$$EnhancerByGuice$$572f1bd9} [t20026/cSync/r3125///] Cassandra timeout during write query at consistency QUORUM (1 replica were requ$
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (1 replica were required but only 0 acknowledged the write)
        at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:54) ~[cassandra-driver-core-2.1.2.jar:na]
        at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:259) ~[cassandra-driver-core-2.1.2.jar:na]
        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:175) ~[cassandra-driver-core-2.1.2.jar:na]
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) ~[cassandra-driver-core-2.1.2.jar:na]
        at org.obm.push.cassandra.dao.SyncedCollectionDaoCassandraImpl.put(SyncedCollectionDaoCassandraImpl.java:82) ~[push-dao-cassandra-3.0.3.jar:na]
        at org.obm.breakdownduration.BreakdownDurationInterceptor.invoke(BreakdownDurationInterceptor.java:58) ~[breakdown-duration-3.0.1-alpha3.jar:na]
        at org.obm.push.protocol.data.SyncAnalyser.getCollection(SyncAnalyser.java:120) ~[push-core-3.0.3.jar:na]
        at org.obm.push.protocol.data.SyncAnalyser.analyseSync(SyncAnalyser.java:92) ~[push-core-3.0.3.jar:na]
        at org.obm.push.handler.SyncHandler.analyzeRequest(SyncHandler.java:220) ~[push-core-3.0.3.jar:na]
        at org.obm.push.handler.SyncHandler.process(SyncHandler.java:163) ~[push-core-3.0.3.jar:na]
        at org.obm.push.handler.WbxmlRequestHandler.process(WbxmlRequestHandler.java:128) ~[push-core-3.0.3.jar:na]
        at org.obm.push.technicallog.TechnicalLoggingInterceptor.invoke(TechnicalLoggingInterceptor.java:70) ~[technical-log-3.0.3.jar:na]
        at org.obm.push.ActiveSyncServlet.processActiveSyncMethod(ActiveSyncServlet.java:253) ~[push-core-3.0.3.jar:na]
        at org.obm.push.ActiveSyncServlet.doPost(ActiveSyncServlet.java:176) ~[push-core-3.0.3.jar:na]
        at org.obm.push.technicallog.TechnicalLoggingInterceptor.invoke(TechnicalLoggingInterceptor.java:70) ~[technical-log-3.0.3.jar:na]
        at org.obm.annotations.transactional.TransactionalInterceptor.managerTransaction(TransactionalInterceptor.java:87) ~[annotations-3.0.1-alpha3.jar:na]
        at org.obm.annotations.transactional.TransactionalInterceptor.invoke(TransactionalInterceptor.java:73) ~[annotations-3.0.1-alpha3.jar:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) [guice-servlet-3.0.jar:na]
        at org.obm.breakdownduration.BreakdownDurationFilter.doFilter(BreakdownDurationFilter.java:71) [breakdown-duration-3.0.1-alpha3.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        at org.obm.servlet.filter.qos.QoSFilter.handleRequest(QoSFilter.java:113) [qos-filter-3.0.1-alpha3.jar:na]
        at org.obm.servlet.filter.qos.QoSFilter.doFilter(QoSFilter.java:94) [qos-filter-3.0.1-alpha3.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        at org.obm.push.ActiveSyncRequestFilter.doFilter(ActiveSyncRequestFilter.java:78) [push-core-3.0.3.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        a

How can i increase a little bit the write timeout ?



 Comments   
Comment by Thomas HILAIRE [ 24/Nov/14 ]

Thanks you for reporting your issue, can I know if this exception appears only once, sometimes or often ?
Can you also tell us some informations about your Cassandra host ? What's the machine configuration ? Does it hold all your OBM services ?

Comment by Carnus Jérémy [ 24/Nov/14 ]

Hi, this exception exist for me since Cassandra 2.1. 2 Times i have it, but when I have it, i need to restart cassandra and opush, after all request will exit in timeout.

Yes Cassandra and opush are in the same machine, 4gb of ram. What is surprising it's since dsc21 and this server only host 1 user (like QA for me)

Comment by Thomas HILAIRE [ 24/Nov/14 ]

We know that the default "retry policy" of the cassandra driver does not fit perfectly Opush's needs, we have to write our own.

But I think that you have another problem as it is weird that you need to restart your cassandra instance, regular timeouts do not bring down the server.
Please verify why your Cassandra does not reply anymore, your OS may kill it to free up memory or something like that ..

Comment by Carnus Jérémy [ 24/Nov/14 ]

Right with you, Opush is not directly responsable for that, but will be good to be able to increse a bit the write timeout (as i don't even know it ).
Occurs since Cassandra 2.1, memory still free when this issue occurs. What seems strange is after that, all request lead to a timeout. And of course, no log in cassandra

Comment by Antoine DUPRAT [ 12/Mar/15 ]

QA: to be validate on a Cassandra cluster configuration

Two retry policy are now available on the configuration of the cassandra client (/etc/opush/cassandra.ini file).

cassandra.retry-policy:

  • RETRY: retry the request when a read, write or unavailable timeout is raised (number of retry is parameterized thru the cassandra.max-retries entry)
  • DOWNGRADE: such as RETRY on read and write timeout, downgrade the consistency level on unavailable

In order to validate this issue, you may stop one of the cassandra node, or degrade the network on a cassandra node.

tc qdisc add dev eth0 root netem loss 50%

The logs (/etc/opush/opush.log file) should looks like:

  • RETRY:
    [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 0)
    [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 1)
    [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on write timeout at consistency QUORUM(initial consistency: QUORUM, write type: SIMPLE, required acknowledgments: 2, received acknowledgments: 1, retries: 2)
    
  • DOWNGRADE:
    [INFO ] {c.d.d.c.p.LoggingRetryPolicy} [t/c/r/] Retrying on unavailable exception at consistency ONE (initial consistency: QUORUM, required replica: 2, alive replica: 1, retries: 0)
    
Comment by Jenkins Continuous Integration Server [ 18/Mar/15 ]

UNSTABLE: Integrated in opush-master #213
OP-137 Add entries in cassandra configuration related to RetryPolicy (thilaire: 14b3ca0a0c0ede69f2328ac864591af69378c331)

  • integration-testing/opush/src/main/java/org/obm/opush/env/OpushStaticConfiguration.java
  • push-dao-cassandra/src/test/java/org/obm/push/configuration/CassandraConfigurationFileImplTest.java
  • push-dao-cassandra/src/main/java/org/obm/push/configuration/CassandraConfigurationFileImpl.java
  • push-dao-cassandra/src/test/java/org/obm/push/cassandra/TestCassandraConfiguration.java
  • push-interfaces/src/main/java/org/obm/push/configuration/CassandraRetryPolicy.java
  • integration-testing/opush/src/main/java/org/obm/opush/env/OpushConfigurationFixture.java
  • push-interfaces/src/main/java/org/obm/push/configuration/CassandraConfiguration.java
  • push-server/src/configuration/etc/cassandra.ini
    OP-137 Add RetryPolicy in the configuration of Cassandra client (thilaire: 808815f49f07f46e0f157b0d2b93c3de11cd8f2e)
  • push-dao-cassandra/src/main/java/org/obm/push/cassandra/RetryNTimesRetryPolicy.java
  • push-dao-cassandra/src/main/java/org/obm/push/cassandra/CassandraSessionSupplierImpl.java
  • push-dao-cassandra/src/main/java/org/obm/push/cassandra/RetryOrCLDowngradeRetryPolicy.java
  • push-dao-cassandra/src/test/java/org/obm/push/cassandra/RetryNTimesRetryPolicyTest.java
  • push-dao-cassandra/src/test/java/org/obm/push/cassandra/RetryOrCLDowngradeRetryPolicyTest.java
  • push-server/src/main/resources/logback.xml
    OP-137 Documentation on Cassandra retry policy (thilaire: b6cf35665ae743dfaed12c9e2fdc3de823d1b055)
  • push-server/src/main/asciidoc/administration.adoc
Comment by Thomas HILAIRE [ 06/May/15 ]

OK tested with a centos 6.6 and three Cassandra nodes:

  • it reties on write timeout until that 'cassandra.max-retries" is reached
  • it reties on read timeout until that 'cassandra.max-retries" is reached
  • it reties on unavailable until that 'cassandra.max-retries" is reached
Comment by chenlixiang [ 20/Mar/18 ]

chenlina20180320
http://www.canadagoosecoatsoutlet.us.com
http://www.burberryoutlet-canada.ca
http://www.truereligionsale.com.co
http://www.uggsstoreoutlet.us.com
http://www.newbalance.me.uk
http://www.burberrysale.us.org
http://www.airmax97.org
http://www.supremeuk.co.uk
http://www.ray-banssunglasses.org.uk
http://www.katespade-outlet.in.net
http://www.airmaxnike.me.uk
http://www.prada.us.org
http://www.vancleefarpelsjewelry.us.com
http://www.airmax2018.in.net
http://www.jacketmoncler.co.uk
http://www.oakleysungalsses.nom.co
http://www.north--face.us.com
http://www.pandorajewelry.us.org
http://www.coachfactoryoutletcom.us.com
http://www.pandorajewelrysoutlet.us.com
http://www.ray-bans-sunglasses.us.com
http://www.huarache.us.com
http://www.vibram-fivefingers.us.com
http://www.toryburchhandbagsoutlets.us.com
http://www.louboutinoutlets.us.com
http://www.monclernederland.nl
http://www.michaelkors--outlet.us.com
http://www.stuartweitzmanshoes.us.com
http://www.cheap-jordans.us.com
http://www.stussy.us.com
http://www.uggoutletstore.us.org
http://www.stansmithadidas.org.uk
http://www.michael-korshandbagsoutlet.us.org
http://www.nikeukstore.org.uk
http://www.oakleyssunglassescheap.com
http://www.nikekyrie3.us.com
http://www.katespadeoutletsale.us.com
http://www.uggbuys.com
http://www.hermesbeltoutlet.com
http://www.baseballbats.us.com
http://www.runningshoes.us.org
http://www.thenorthfacecom.us.com
http://www.salvatoreferragamo.in.net
http://www.mishka.us.com
http://www.supreme-newyork.com
http://www.poloshirtsoutlet.us.com
http://www.barcelonajersey.us.com
http://www.ralphlaurenoutletofficial.us.com
http://www.supremeclothing.us.com
http://www.canadagoosejacket.me.uk
http://www.adidasshoess.us.com
http://www.basketballshoesstore.us.com
http://www.swarovski.us.org
http://www.ugg.co.nl
http://www.pradahandbagsoutlet.us.com
http://www.coachoutletonlinestore.ca
http://www.uggscanadaboots.ca
http://www.bottega.in.net
http://www.nikeairmaxuk.co.uk
http://www.oakleysunglassescom.us.com
http://www.nikeairforce1.net
http://www.jordan1.us
http://www.coachoutlet-handbags.us.com
http://www.poloralphoutlet.us.com
http://www.bestnfljerseyswholesale.us.com
http://www.tods.us.com
http://www.juicycoutureoutlet.us.com
http://www.pradabags.in.net
http://www.converseshoesoutlet.us.com
http://www.raybansunglassesforwomen.us.com
http://www.airmaxnike.nl
http://www.thenorthfaceoutlet.ca
http://www.pandorajewelrycanada.ca
http://www.salecanadagooseoutlet.us.com
http://www.miumiu.us.com
http://www.nikefactoryoutletstores.us.com
http://www.tomsoutlet.in.net
http://www.michaelkorsonline.ca
http://www.poloralphlaurenpascher.fr
http://www.burberryscarves.us.com
http://www.beatsheadphones.us.com
http://www.nikesb.in.net
http://www.wholesaleoakleysunglasses.us.org
http://www.cheapnhljerseysshop.us.com
http://www.nikepresto.us.com
http://www.michaelkorsonline-outlet.us.com
http://www.uggpas-cher.fr
http://www.moncler.eu.com
http://www.katespade-outlet.us.com
http://www.poloralph.us.com
http://www.cheapjordansshoes.us.com
http://www.cheapraybansunglass.co.uk
http://www.nikeairmax.me.uk
http://www.jordans.in.net
http://www.nikecortez.net
http://www.coachonline.us.org
http://www.northfaceoutletstore.us.org
http://www.christianlouboutinshoessale.org.uk
http://www.toryburchoutletcom.us.com
http://www.air-jordanpascher.fr
http://www.nikelunarglide.us
http://www.superdry.me.uk
http://www.pandora-jewelry.name
http://www.kanyeyeezyshoes.us.com
http://www.mbtscarpe.it
http://www.fitflops.us.org
http://www.truereligionoutletjeans.us
http://www.timberland.us.org
http://www.niketrainers.org.uk
http://www.adidasflipflops.us.com
http://www.shoecarnival.us.com
http://www.coachoutlet70off.us.com
http://www.louboutin.uk
http://www.monclersitoufficiale.it
http://www.cheapnfljerseys.us.org
http://www.toryburchoutletsaleonline.us.com
http://www.louboutin.in.net
http://www.canadagoosestore.us.com
http://www.jordan5.us
http://www.manchesterunitedjersey.us.com
http://www.the-northfacecanada.ca
http://www.mizuno.us.org
http://www.airmax2017.us.com
http://www.adidasyeezy-boost.us.com
http://www.adidasfootballboots.org.uk
http://www.ferragamooutlet.us.com
http://www.nikeflipflops.org
http://www.coachfactoryoutlet-online.eu.com
http://www.christianlouboutinsale.uk
http://www.fitflopsoutlet.us.com
http://www.coachoutlet-online.eu.com
http://www.kobe11.in.net
http://www.mbt.us.com
http://www.fitflopsandals.org
http://www.canadagoosedownjackets.us.com
http://www.giuseppe-zanotti.us.com
http://www.uggstore.us.com
http://www.nikeblazers.us
http://www.coachhandbagsfactoryoutletonline.us.com
http://www.jordans-fusion.com
http://www.cheaprolexwatches.us.com
http://www.adidas.us.org
http://www.coachoutletstoreonlineofficial.us.com
http://www.pandoraukcharms.co.uk
http://www.christianlouboutinoutletuk.org.uk
http://www.nikejordans.co.uk
http://www.nikeblazer.fr
http://www.discountnike.us.com
http://www.raybansunglassesoutlet.net.co
http://www.coachwallets.us.com
http://www.cartierjewelry.us.com
http://www.longchamp-outlet.us.org
http://www.cheapmichaelkorshandbags.us.org
http://www.ralph-lauren.com.co
http://www.northfacethe.us.com
http://www.nikeoutletstoreonline.us.com
http://www.nikeshox.in.net
http://www.ukmoncler.org.uk
http://www.bcbg.us.com
http://www.northfacejackets.us.org
http://www.michael-korsoutletstores.us.org
http://www.mcmbackpack.us.com
http://www.airjordan11.us
http://www.dansko.us.com
http://www.monclerdunjacka.se
http://www.michaelkorsoutlet70off.us.com
http://www.birkenstockshoes.org.uk
http://www.coachoutlet-factory.us.com
http://www.redbottomsshoes.me.uk
http://www.airmaxplus.org
http://www.airforce1.in.net
http://www.adidasjeremyscott.in.net
http://www.airjordanretro.fr
http://www.oakleysunglasshut.us.com
http://www.ralphlauren-poloshirts.co.uk
http://www.redbottoms.us.org
http://www.michael--kors.me.uk
http://www.baseballjersey.us.com
http://www.moncler-outletonline.us.com
http://www.lebron15.in.net
http://www.asics.us.org
http://www.burberryoutletonline.us.org
http://www.nikeairmaxschweiz.ch
http://www.korsoutlet.us.org
http://www.jordanretro.us
http://www.ralphlauren.in.net
http://www.michael-korstaschen.ch
http://www.adidaswings.name
http://www.longchamp-outletstore.us.com
http://www.outletuggsale.us.com
http://www.flops.us.com
http://www.christian--louboutin.us.com
http://www.uggsitalia.it
http://www.katespadehandbagssale.us.org
http://www.cheapmlbjerseysshop.us.com
http://www.maccosmeticsofficial.us.com
http://www.adidasslides.us.com
http://www.pradaoutletonlinestore.us.com
http://www.balenciaga.us.org
http://www.swarovskijewelry.co.uk
http://www.adidasstoreuk.me.uk
http://www.coachfactoryoutlet-onlinestore.us.com
http://www.valentino.us.org
http://www.michael-kors-uk.me.uk
http://www.coach-bags.us.org
http://www.birkenstockoutlet.us.org
http://www.nikemercurial.us
http://www.montblanccom.us.com
http://www.oakley-sunglassesonsale.us.com
http://www.nikedunks.us
http://www.soccerjersey.us.com
http://www.authenticmlbjerseyscheap.us.com
http://www.coachfactory--outlet.us.com
http://www.outletsuggstores.us.com
http://www.weddingshoes.us.com
http://www.michaeljordan-shoes.com
http://www.rayban.net.co
http://www.canada-goosejassen.nl
http://www.ugg.com.co
http://www.korsmichaelkors.org.uk
http://www.niketennis.us.com
http://www.northfaceoutletonline.com
http://www.northfacecom.us.com
http://www.ralph-laurenpolouk.org.uk
http://www.fitflopssaleclearance.us.org
http://www.truereligionjeans-outlet.us.com
http://www.outletcanadagoosesale.us.com
http://www.uggstoresoutlet.us.com
http://www.canada--goose.fr
http://www.cheap-oakleysunglasses.com
http://www.pandorajewelry-canada.ca
http://www.outletralph-lauren.us.com
http://www.longchamp-handbags.us.org
http://www.nikeroshe.us.com
http://www.lightup-shoes.us.com
http://www.raysunglass.com
http://www.uggbootsclearance.net.co
http://www.nhlhockey-jerseys.us.com
http://www.converse.me.uk
http://www.airmax95.us.org
http://www.longchampoutletcom.us.com
http://www.dolceandgabbana.us.com
http://www.coachoutletonsale.us.com
http://www.northfacejacketsclearance.us.com
http://www.coachoutletsale.us.com
http://www.hervelegerdresses.us.com
http://www.katespadehandbagsoutlet.us.org
http://www.poloralphlaurenoutlet.org.uk
http://www.snapbacks-wholesale.us.com
http://www.uggaustralia.org.uk
http://www.jordanshoes.us.org
http://www.cheapairjordans.us.com
http://www.nikeairmax90.us.org
http://www.ralphlaurenpolo.in.net
http://www.oksunglasshut.com
http://www.uknorthface.org.uk
http://www.truereligion-jeans.us
http://www.uggfactoryoutletstore.us.com
http://www.tory-burch.org.uk
http://www.uggsoutletcom.us.com
http://www.airhuarache.fr
http://www.moncleroutletstore.us.com
http://www.thenorthfaceuk.co.uk
http://www.wwwnorthface.us.com
http://www.ralplaurenuk.org.uk
http://www.hermeskellybag.us.com
http://www.airjordan-uk.co.uk
http://www.michaelkorscom.us.com
http://www.redbottomshoes.org.uk
http://www.michaelkorsoutletmall.us.com
http://www.adidasshoesoutletstore.us.com
http://www.burberry-outlet-stores.us.com
http://www.pandoracharmjewelry.us.com
http://www.michaelkorsoutletoff.us.com
http://www.canadagooses.org
http://www.christian--louboutin.us
http://www.mulberryukhandbags.org.uk
http://www.tommyhilfigeroutlet.ca
http://www.nikeairmax90.com.de
http://www.nfl--jerseys.us.com
http://www.rayban.in.net
http://www.nikeoff.com
http://www.uggschweiz.ch
http://www.cheap-jordan-shoes.us.com
http://www.coach-factoryonline.us.org
http://www.nikefootballbootsuk.org.uk
http://www.coachcom.us.com
http://www.sunglassesoakleycheap.us.com
http://www.belstaffoutlet.us.com
http://www.fitflopssale.org.uk
http://www.newcoachoutlet.us.com
http://www.canada--goose.net
http://www.ralph-laurens.co.uk
http://www.bottesuggs.fr
http://www.linksoflondonuk.me.uk
http://www.longchampsacsoldes.fr
http://www.uggclearanceoutlet.us.com
http://www.ray-bansunglassessale.us.org
http://www.jordanxx9.in.net
http://www.jordan12.org
http://www.asics.org.uk
http://www.moncler-jacken.at
http://www.northfaceclearance.us.com
http://www.monclerdoudounepascher.fr
http://www.uggcanadaoutlet.ca
http://www.michael-korsoutletonline.us.org
http://www.airjordan-14.us
http://www.cheapjordansshoes.in.net
http://www.montblancpensstore.us.com
http://www.niketn.fr
http://www.marcjacobs.me.uk
http://www.nikefree.me.uk
http://www.katespadestoreoutlet.us.com
http://www.polooutletinc.us.com
http://www.moncler--paris.fr
http://www.coachsoutletstoreonline.us.com
http://www.jacketsmoncler.us.com
http://www.coachoutletstore.eu.com
http://www.wholesalenfl-jerseys.us.com
http://www.uggclearance.us.com
http://www.alexandermcqueen.us.com
http://www.goedkope-uggs.be
http://www.monclerjackenschweiz.ch
http://www.supremeparis.fr
http://www.coachoutletonline70off.us.com
http://www.cheapraybans.ca
http://www.undefeated.us.com
http://www.kevindurantshoes.us.com
http://www.birkenstocksandals.us.org
http://www.uggbootsoutlet.co.uk
http://www.hermesoutlets.us.com
http://www.rayban-sunglassesonsale.us.com
http://www.jerseys.us.org
http://www.airmax.us.org
http://www.cheapmlbjersey.us.com
http://www.oakleysunglasses-vault.us.com
http://www.hermesbags.me.uk
http://www.michaelkorsoutlet.us
http://www.louboutin.us.com
http://www.bvlgari.us.com
http://www.stephen-curryshoes.us.com
http://www.michaelkorshandbagsoutletsales.us.com
http://www.nikefreeruns.us.com
http://www.ralph-lauren.fr
http://www.underarmouroutletclearance.com
http://www.stansmithadidas.us.com
http://www.cheapnfljerseysstore.us.com
http://www.newbalance-shoes.us.com
http://www.uggscanadastore.ca
http://www.hardenvol1.us.com
http://www.suprashoesclearance.us.com
http://www.oakleysunglassescanada.ca
http://www.airmax1.us
http://www.nikesshoes.us.com
http://www.fingerlingsmonkey.org
http://www.nikerevolution.us
http://www.nikefree-run.fr
http://www.blackhawksjersey.us.com
http://www.adidassuperstar.name
http://www.nike-factoryoutlet.us.com
http://www.timberlandoutlet.us.org
http://www.nikefree.net
http://www.airmax270.org
http://www.cheapjerseys.in.net
http://www.nike-airhuarache.me.uk
http://www.ferragamoshoesoutlet.us.com
http://www.givenchyhandbags.us.com
http://www.truereligion-jeans.me.uk
http://www.north-face-outlet.in.net
http://www.monclerjassen.be
http://www.converseshoes.us.org
http://www.christian-louboutins.fr
http://www.ray-bansunglassesoutlet.com.co
http://www.timberlandoutlet.org.uk
http://www.manoloblahnik.us
http://www.chaussuresnikeairmax.fr
http://www.cheapbestnbajerseys.us.com
http://www.northfacejacket.us.com
http://www.christiandiorhandbags.us.com
http://www.cheapreplicawatches.us.com
http://www.nikefactory-outlet.us.com
http://www.longchampoutlet-online.com
http://www.jimmychoooutlet.us.com
http://www.timberlandpro.us.com
http://www.montblanc.com.co
http://www.moncleroutlet.us.org
http://www.adidasoutletstores.com
http://www.cheapraybansunglasses.net.co
http://www.hermesbirkinbags.us.com
chenlina20180320

Comment by myy (Inactive) [ 09/May/18 ]

http://www.ralphlauren.mex.com/, http://www.airhuarache-inc.co.uk/, http://www.scarpehogan-shoes.it/, http://www.tommys-hilfigers.net/, http://www.newbalance-canada.ca/, http://www.converse-shoes-outlet.com/, http://www.birkenstock.net.co/, http://www.cheapjerseys-outlet.com/, http://colts.nfl-jerseys.org/, http://www.thenorth-face.com.co/, http://www.airmax.com.se/, http://www.raybansbocco.it/, http://www.rolexswatchoutlet.com/, http://www.oakleyssunglasslrc.com/, http://www.mk-com.com/, http://www.hollister-clothings.us.com/, http://www.coachoutlet.fr/, http://www.soccers-shoes.us.com/, http://www.nhljerseys.com.co/, http://www.kate-spades.net/, http://www.australia-michaelkors.com/, http://lakers.nbajerseyssale.com/, http://www.michael-korsbags.com.co/, http://www.mcmbackpacks.com.co/, http://www.cheap-watches.com.co/, http://www.hollister.com.de/, http://clippers.nbajerseyssale.com/, http://www.ralphlaurens.us.com/, http://www.new-balanceshoes.com/, http://www.adidas-shoes.co.nl/, http://www.prada-outlet.net.co/, http://www.timberland-shoes.net.co/, http://www.supra-footwears.com/, http://www.ok-em.com/, http://www.katespades.com.co/, http://www.giuseppes-zanottishoes.com/, http://www.belstaff-jackets.us.com/, http://www.mlbjerseys.com.co/, http://www.oakley-outlet.fr/, http://www.salvatoreferragamo.us.com/, http://www.belstaff-outlet.com/, http://www.louboutinshoes.jp.net/, http://www.jackwolfskinoutlet.ca/, http://www.nike-shoes.com.co/, http://www.nikefree5.com.co/, http://www.replicabags.net.co/, http://www.jewelrys-swarovski.com/, http://www.gucci-shoesoutlet.com/, http://www.coach-factoryoutlet.net.co/, http://magic.nbajerseyssale.com/, http://www.raybans-cher.fr/, http://www.ed-hardy.us.com/, http://www.nike-factory.com.co/, http://www.kate-spadebags.com/, http://www.uggs-boots.com/, http://www.the-northfacejackets.net.co/, http://grizzlies.nbajerseyssale.com/, http://cavaliers.nbajerseyssale.com/, http://www.burberbags.com/, http://www.timberlandboots-outlet.com/, http://www.guccishoes.net.in/, http://www.swarovskionlineshop.com.de/, http://www.cheap-omegaswatches.com/, http://www.adidasschuhe-online.com.de/, http://www.coachblackfriday.com/, http://www.nikeroshe-run.com.es/, http://www.cheap-michaelkors.com.co/, http://www.oakleys-online.com.co/, http://www.nikeairmax-shoes.fr/, http://www.burberrys-outletonline.in.net/, http://www.airjordans-shoes.com/, http://www.jerseys-store.com/, http://www.poloralphlaurens.com.de/, http://www.michael-kors.com.es/, http://giants.nfl-jerseys.org/, http://www.nikeairmax-australia.com/, http://www.babylisspros.com/, http://www.coach-factorys.com/, http://www.new-iphonecases.com/, http://www.nikeair-max.es/, http://celtics.nbajerseyssale.com/, http://www.coach-factory.com.co/, http://www.versaces-outlet.com/, http://www.barbours-factory.com/, http://www.air-yeezyshoes.com/, http://coachoutlet.euro-us.net/, http://www.nikeairforce.com.de/, http://www.jerseys.net.in/, http://www.raybanoutlet.ca/, http://www.givenchy.com.co/, http://www.salomonshoes.com.de/, http://www.nikeair-max.ca/, http://www.airmax-90.com/, http://www.timberland-shoes.fr/, http://www.long-champ-outlet.com/, http://www.hermesbagsoutlet.com/, http://www.juicy-coutureoutlet.com/, http://www.rolex-watchescanada.ca/, http://www.fidgetspinner.com.co/, http://www.outletonline-michael-kors.co.uk/, http://www.salvatore-ferragamos.com/, http://www.polos-ralphlauren.com.co/, http://www.bcbg-dresses.net/, http://www.nikerosherun.com.co/, http://www.moncler-jackets.com.co/, http://www.michaelkors.so/, http://www.nfljerseys-sale.com/, http://www.converses.us.com/, http://www.nikefreerun-inc.co.uk/, http://www.uhren-stores.de/, http://www.vibram--fivefingers.com/, http://www.pandora-jewelry.top/, http://www.stone-islandestore.com/, http://www.newbalanceschuhe.com.de/, http://www.hermes-birkinoutlet.com/, http://www.woolrich-jackets.com/, http://www.christianlouboutin-shoes.ca/, http://www.hollister.com.se/, http://www.adidasshoes.com.se/, http://bears.nfl-jerseys.org/, http://www.ralphlaurens.co.uk/, http://www.christian-louboutinshoesoutlet.com/, http://www.the-northface.com.de/, http://www.burberry-outlets.ca/, http://www.nikeshoes.de/, http://www.supra-footwear.net/, http://www.true-religionsoutlet.com/, http://www.dsquared2-outlet.com/, http://www.nike-huarache.co.nl/, http://www.eyeglassesinsale.com/, http://www.longchamps-bags.us.com/, http://bucks.nbajerseyssale.com/, http://www.coachfactory.cc/, http://lions.nfl-jerseys.org/, http://www.christian-louboutins.us.com/, http://packers.nfl-jerseys.org/, http://www.cheap-montblancpens.com/, http://www.truereligions.net/, http://www.tommy-hilfigers.de/, http://www.fred-perrys.co.uk/, http://www.hoodiesoutlet.net/, http://www.michaelkorspurses.co.uk/, http://patriots.nfl-jerseys.org/, http://www.cheap-mlbjerseys.us.com/, http://76ers.nbajerseyssale.com/, http://www.mcm-bagsoutlet.com/, http://www.jordan-retros.com/, http://www.guci-heaven.com/, http://warriors.nbajerseyssale.com/, http://www.coachoutletonlinemall.com/, http://www.ralph-laurenoutlets.com/, http://www.designer-bags.us.com/, http://www.cheapshoes.net.co/, http://vikings.nfl-jerseys.org/, http://www.kate-spade-outlet.us.com/, http://www.nike-paschers.fr/, http://www.nike-free-runs.de/, http://www.nikefrees-shoes.co.uk/, http://saints.nfl-jerseys.org/, http://www.barboursjackets.com/, http://texans.nfl-jerseys.org/, http://www.linksoflondonvip.com/, http://www.lacosteoutlet.com.co/, http://www.hugo-boss.com.co/, http://www.swarovskijewellery-au.com/, http://www.canada-michaelkors.ca/, http://www.rosherun-shoes.us/, http://www.armaniexchange.us.com/, http://www.tracksuits.com.co/, http://www.jeans-truereligion.com/, http://www.oakley-sbocco.it/, http://www.tomsshoes-outlet.us.org/, http://www.nike-freeruns.com/, http://seahawks.nfl-jerseys.org/, http://www.skechers-outlet.com/, http://www.adidas-shoes.com.co/, http://www.rayban-lunettes.fr/, http://www.lonchamp.us.com/, http://www.fredperrypolos.co.uk/, http://spurs.nbajerseyssale.com/, http://www.reeboks.com.de/, http://www.tods-shoesoutlet.com/, http://www.rayban-sunglasses.co/, http://www.oakleysunglasses.org.es/, http://www.skechers-shoesoutlet.com/, http://www.oakleys-outlets.net/, http://www.cheap-juicycouture.com/, http://www.coachoutlet-online.com.co/, http://www.burberry-storevip.com/, http://www.ferragamos.us.com/, http://www.nike-store.com.de/, http://www.oakleys.com.de/, http://www.adidassuper-star.com.de/, http://www.converse-schuhe.com.de/, http://www.raybans-glasses.com.co/, http://ravens.nfl-jerseys.org/, http://www.nikeshoes.mex.com/, http://www.horlogesrolexs.co.nl/, http://www.ferragamo-inc.co.uk/, http://www.omegaswatches.com/, http://www.barbours.us.com/, http://www.belstaffjacketsoutlet.co.uk/, http://www.michael-kors.net.co/, http://titans.nfl-jerseys.org/, http://www.versaceoutlet.us.com/, http://www.the-north-face-jackets.com/, http://www.raybansasses.com/, http://www.marcjacobs-outlet.com/, http://www.ray-bans.net.co/, http://www.converse-shoes.net/, http://www.outlet-michaelkors.com/, http://bulls.nbajerseyssale.com/, http://www.under-armouroutlet.com/, http://jazz.nbajerseyssale.com/, http://www.burberryhandbagsvip.com/, http://www.cheap-jordans.com.co/, http://www.crystals-swarovskis.com/, http://www.oakleyframeslrc.com/, http://nuggets.nbajerseyssale.com/, http://jets.nfl-jerseys.org/, http://www.oakley-outletonline.com.co/, http://www.nike-shoesoutlet.com/, http://www.coachoutletonline.vip/, http://www.pandora-jewellery.com.es/, http://www.barbour-jackets.us.com/, http://www.tommyhilfiger-outlet.net/, http://chiefs.nfl-jerseys.org/, http://www.christian-louboutin.jp.net/, http://eagles.nfl-jerseys.org/, http://www.adidas-shoesoutlet.com/, http://www.burberoutlet.com/, http://www.toms-shoe.us.org/, http://falcons.nfl-jerseys.org/, http://www.cheap-pandoracharms.org.uk/, http://cowboys.nfl-jerseys.org/, http://www.fitflop-outlet.com/, http://www.philipppleins.us.com/, http://www.tommyhilfigerca.ca/, http://www.ralphslauren-polo.com/, http://www.nfl-jerseys.org/, http://www.asicsoutlet.net/, http://www.jordan.com.de/, http://www.outlet-michaelskors.com/, http://hawks.nbajerseyssale.com/, http://www.ralph-laurenpolos.com/, http://www.burberroutlet.co.uk/, http://www.handbags-outlet.in.net/, http://www.nikeairmax-isc.co.uk/, http://hornets.nbajerseyssale.com/, http://www.reebok-shoes.co.uk/, http://www.burberry-store.com/, http://www.vansschuhe.com.de/, http://www.new-balanceoutlet.org/, http://www.thomas-sabo.us.com/, http://www.outlet-true-religion.com/, http://www.nikerosherun.com.de/, http://www.montblancoutlet.com/, http://www.outlet-pradabags.com/, http://www.long-champ.com.co/, http://www.michaelkors-ins.com/, http://www.the-northface.in.net/, http://buccaneers.nfl-jerseys.org/, http://www.oakleys-lrc.com/, http://www.the-north-face.net.co/, http://www.nikeshoesoutlet.co.nl/, http://www.outlet-pumashoes.com/, http://www.northface-jackets.co.uk/, http://www.raybansoutlet.co.uk/, http://www.eccoshoesoutlet.com/, http://www.fendi-outlet.us.com/, http://www.cc-beanies.us.com/, http://www.nbajerseyssale.com/, http://www.tommyhilfigers.co.nl/, http://www.mbt-shoesoutlet.com/, http://dolphins.nfl-jerseys.org/, http://www.okglassesale.com/, http://coach.euro-us.net/, http://panthers.nfl-jerseys.org/, http://www.outlet-pradas.com/, http://www.oakley-sunglassessale.com.co/, http://www.nike-maxshoes.fr/, http://www.chiflatironhair.us.com/, http://azcardinals.nfl-jerseys.org/, http://www.bcbg-canada.ca/, http://www.birkenstocks.com.de/, http://pelicans.nbajerseyssale.com/, http://www.toms-shoesoutlets.com/, http://www.cheapnhljerseys.us.com/, http://www.outlet-ralphlaurens.com/, http://www.pandorajewellery.com.es/, http://www.co-aol.com/, http://www.pandora-jewelry.com.de/, http://www.marc-jacobs.us.com/, http://www.thomas-sabos.com.de/, http://www.chrome-heartsoutlet.com/, http://www.outlet-toryburchshoes.com/, http://www.mizuno-running.net/, http://www.ralphlauren-au.com/, http://www.monclers-outlet.us.com/, http://raptors.nbajerseyssale.com/, http://www.skechers-canada.ca/, http://www.dsquared2s.com/, http://www.cheap-raybansoutlet.com.co/, http://www.nike-shoesoutlet.us.org/, http://www.rolexwatchesforsale.us.com/, http://www.ralph-laurenspolos.com/, http://www.rl-ralphlaurenpolos.co.uk/, http://www.michael-kors-bags.com/, http://www.oakleys.mex.com/, http://www.nike-shoes.org.es/, http://www.michael-kors.co.nl/, http://www.christian-louboutin-shoes.com/, http://www.the-north-face.com.co/, http://www.michael-korsoutletonline.com.co/, http://www.vansshoes-outlet.com/, http://www.nike-mercurial.com/, http://www.bottega-outlet.com/, http://www.prada-shoes.us.org/, http://www.levis-jeansoutlet.com/, http://www.outletburberry-bags.com/, http://www.guessfactorys.com/, http://www.toms-shoesstore.com.co/, http://www.hollister-outlet.com/, http://www.jack-wolfskin.us.com/, http://pistons.nbajerseyssale.com/, http://www.mkoutlet.com.co/, http://www.outlet-tommyhilfigers.com/, http://www.adidas-shoes.ca/, http://kings.nbajerseyssale.com/, http://chargers.nfl-jerseys.org/, http://www.raybans-outlet.co.nl/, http://www.new-balanceshoes.es/, http://thunder.nbajerseyssale.com/, http://www.airmax-shoes.com/, http://www.ralphslauren-polo.com.co/, http://trailblazers.nbajerseyssale.com/, http://www.polosoutlets-store.com/, http://www.nike-airmax.in.net/, http://www.cheap-baseballbats.us/, http://www.softballbats-sale.com/, http://browns.nfl-jerseys.org/, http://www.ralphlauren-canada.ca/, http://www.long-champhandbags.com/, http://www.basketballshoes.com.co/, http://www.pandora-charmscanada.ca/, http://www.hollisterclothingcanada.ca/, http://www.hollister-clothings-store.com/, http://www.northfacejackets.fr/, http://www.valentino-shoesoutlet.us.com/, http://www.retrojordan-shoes.com/, http://www.birkenstockshoes.com.de/, http://www.toryburchs.us.com/, http://pacers.nbajerseyssale.com/, http://www.christianlouboutinoutlet.net.co/, http://www.nikeshoes-canada.ca/, http://www.oakleysunglasses-canada.ca/, http://www.coachoutlet-coachoutletonline.com/, http://www.sac-hermes.fr/, http://www.celinefactory.com/, http://timberwolves.nbajerseyssale.com/, http://www.christian-louboutins.com/, http://rockets.nbajerseyssale.com/, http://www.adidas-shoes.es/, http://www.swarovski-jewelrys.co.uk/, http://www.outlet-michaelkors-online.com/, http://www.raybans.com.de/, http://www.nbateamonline.com/, http://www.soccershoes-outlet.com/, http://www.gbyguess.us.com/, http://wizards.nbajerseyssale.com/, http://knicks.nbajerseyssale.com/, http://michaelkors.euro-us.net/, http://www.oakleys-sunglasses.com.co/, http://broncos.nfl-jerseys.org/, http://www.coach-outlets.net.co/, http://www.lacoste-polos.us.com/, http://raiders.nfl-jerseys.org/, http://www.mcmshandbags.com.co/, http://www.air-maxschoenen.co.nl/, http://www.nike-airmax.com.de/, http://www.thomassabos-jewelrys.org.uk/, http://www.vans-shoesoutlet.co.uk/, http://www.barbour-jackets.in.net/, http://steelers.nfl-jerseys.org/, http://www.nike-rosherun.co.nl/, http://www.ferragamo.net.co/, http://www.ralph-laurens-polo.co.uk/, http://www.ray-bansoutlet.net.co/, http://www.longchamp-outlet.com.co/, http://www.calvin-kleinoutlet.com/, http://www.nike-skos.com/, http://www.oakestore.com/, http://www.mcm-handbags.com.co/, http://www.polo-ralphlaurens.com.co/, http://www.ray-bans.org.es/, http://www.pradas.com.de/, http://www.long-champ.com.de/, http://www.mcm-backpacks.com.co/, http://www.under-armour.us.com/, http://www.burberrbags.co.uk/, http://www.true-religions.com/, http://www.burberryoutletonline.com.de/, http://www.burbagsoutlet.com/, http://www.rayban-wear.co.uk/, http://www.cheap-jerseys.mex.com/, http://www.polo-ralphlaurens.us.com/, http://www.watch-rolex.co.uk/, http://www.nba-shoes.com/, http://www.michaelkorsoutlet.se/, http://redskins.nfl-jerseys.org/, http://www.airmax2015.net/, http://nets.nbajerseyssale.com/, http://www.chiflatiron-hair.com/, http://www.ralphlaurens.com.de/, http://www.nike-shoes-outlet.co.uk/, http://www.michaels-kors.co.uk/, http://www.soccershoes.us.com/, http://www.cheap-iphone-cases.com/, http://www.philipp-pleins.org/, http://www.airhuarache-shoes.co.uk/, http://rams.nfl-jerseys.org/, http://www.coachoutlet-usa.com/, http://www.michaelkors.com.de/, http://www.the-northface.ca/, http://www.pandora-jewelrys.com/, http://bills.nfl-jerseys.org/, http://www.bottega-venetasoutlet.com/, http://www.nikeairmax.com.de/, http://www.giuseppezanotti.com.co/, http://www.hollister-store.com.co/, http://www.rolex-watches.us.com/, http://www.swarovski-canada.ca/, http://www.coach-outletonline.ca/, http://mavericks.nbajerseyssale.com/, http://www.jimmychoos-shoes.com/, http://www.oakleysoutlet.com.co/, http://heat.nbajerseyssale.com/, http://www.nike-shoes.com.se/, http://www.michaelkorsoutlet.online/, http://www.outlet-ralphlauren.co.uk/, http://www.mizuno.us.com/, http://www.danskoshoes.in.net/, http://www.asics-shoesoutlet.net/, http://www.outlet-true-religion.us.org/, http://www.raybans-outlet.net.co/, http://www.hogans.com.de/, http://jaguars.nfl-jerseys.org/, http://www.replica-bags.us.com/, http://www.burberryoutlet-sale.net.co/, http://www.hoganshoesoutlet.co.uk/, http://www.jimmychoo-shoes.com/, http://www.omegas-relojes.es/, http://www.australia-uggs.com/, http://www.hermesbags-outlet.com/, http://www.puma-shoes.com.de/, http://www.tommyhilfigers.de/, http://www.ray-bans.com.co/, http://49ers.nfl-jerseys.org/, http://www.rosherun-shoes.co.uk/, http://www.toms-outletonline.com.co/, http://www.toms--outlet.com.co/, http://suns.nbajerseyssale.com/, http://bengals.nfl-jerseys.org/, http://www.zccoachoutlet.com/, http://www.raybans-outlet.cc/, http://www.montrespascher.fr/, http://www.asicsshoes-outlet.com/, http://www.parajumpersjacketsoutlet.com/, http://www.lauren-ralphs-outlet.co.uk/, http://www.airjordanshoes.com.co/, http://www.michaelkorsoutletvip.com/, http://www.oakleyslrc.com/, http://www.timberland-boots.com.co/

Generated at Wed Aug 21 13:25:20 CEST 2019 using JIRA 6.1.1#6155-sha1:7188aeec9a6b57d61ea04c52f235f15f55c105e2.