QA: check that the client truncation size is respected on huge mails

Client request:

<Sync>
<Collections>
<Collection>
<SyncKey>ecc038bd-1d4a-4dc1-97b5-79e2848af50e</SyncKey>
<CollectionId>12</CollectionId>
<DeletesAsMoves>1</DeletesAsMoves>
<GetChanges/>
<WindowSize>10</WindowSize>
<Options>
<FilterType>0</FilterType>
<BodyPreference>
<Type>2</Type>
<TruncationSize>200000</TruncationSize>
</BodyPreference>
</Options>
</Collection>
</Collections>
</Sync>

IMAP logs:

09:29:39.474 [INFO ] {IMAP.COMMAND} [t36/cSync/r6/usera@adupratvm.lyon.lan] C: A11 UID FETCH 5 (UID BODY.PEEK[1]<0.200000>)
09:29:39.477 [INFO ] {IMAP.COMMAND} [t36/cSync/r6/usera@adupratvm.lyon.lan] Command status SUCCESS

ABORTED: Integrated in opush-master #206
OP-180 Use truncation when fetching mail (thilaire: bc95128598e2d1b893f8cfa0788f666ede0757ca)

• push-mail/src/test/java/org/obm/push/mail/imap/MailboxStoreAPITest.java
• push-mail/src/main/java/org/obm/push/mail/MailboxService.java
• pom.xml
• client/src/main/java/org/obm/sync/push/client/commands/Sync.java
• push-mail/src/test/java/org/obm/push/mail/MailboxBackendTest.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
• push-mail/src/test/java/org/obm/push/mail/imap/MailboxServiceTest.java
• integration-testing/opush/src/test/resources/eml/only_attachments.eml
• push-mail/src/main/java/org/obm/push/mail/imap/LinagoraMailboxService.java
• push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
• push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
• integration-testing/opush/src/main/java/org/obm/opush/IntegrationTestUtils.java

NOK many emails display are broken on iOS7 and were OK with opush 3.0.6. We really should mimic the exchange behavior for this kind of stuff to let this kind of issue behind us.

I create some eml to tests that and I made every required capture on Exchange. I won't attach them to this ticket because their are too heavy.

Twice less object (when > 1kib) allocations are done with improvements done in this issue, see before.png and end.png attachments

QA & Impacts:

• all kind of emails synchronization
  • text only
  • html only
  • text and html
  • with or without attachment
  • etc
• such emails synchronization with all kinds of devices
• replying and forwarding

FAILURE: Integrated in opush-master #229
OP-180 Use linagora-imap version with fewer memory usage (thilaire: d661578af0edca3d9005379d9891d7b0fc6efdee)

• pom.xml
  OP-180 Limit the log size (thilaire: 2218d3dc368be955099efbb87863a4dfc819d552)
• push-core/src/test/java/org/obm/push/utils/LimitedOutputStreamTest.java
• push-core/src/main/java/org/obm/push/utils/LimitedOutputStream.java
• push-core/src/main/java/org/obm/push/impl/DOMDumper.java
  OP-180 pipe streams to avoid ByteArray and String allocation (thilaire: effc7253ebc24d5efaf6d9de0f8dc007cb8310e9)
• wbxml/src/main/java/org/obm/push/wbxml/parsers/WbxmlEncoder.java
• wbxml/src/main/java/org/obm/push/wbxml/WBXMLTools.java
  OP-180 Use a writer to avoid array and String allocations (thilaire: 3d40f4147ba47224fd157423e638c59c36cec147)
• wbxml/src/main/java/org/obm/push/wbxml/parsers/WbxmlEncoder.java
• wbxml/src/main/java/org/obm/push/wbxml/parsers/EncoderHandler.java
  OP-180 Use the regular protocol encoder for sync test (thilaire: fe5bd4b0498c977e0a3169d85618ac21309f373d)
• integration-testing/opush/src/test/java/org/obm/opush/MailBackendGetChangedTest.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerOnCalendarsTest.java
• client/src/main/java/org/obm/sync/push/client/commands/Sync.java
• protocol/src/main/java/org/obm/push/protocol/data/ActiveSyncFields.java
• protocol/src/main/java/org/obm/push/protocol/data/SyncEncoder.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerTest.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerOnContactsTest.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  OP-180 Strong typing of MimeSupport (thilaire: 0832aa453b0be3651ea5856fc9381e62767ec91a)
• protocol/src/test/java/org/obm/push/protocol/bean/SyncCollectionTest.java
• push-bean/src/test/java/org/obm/push/bean/SyncCollectionOptionsTest.java
• push-dao-cassandra/src/main/java/org/obm/push/json/JSONService.java
• push-dao-cassandra/src/main/java/org/obm/push/json/mixin/MimeSupportMixIn.java
• push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceCompatibilityTest.java
• push-bean/src/test/java/org/obm/push/bean/BeansTest.java
• protocol/src/test/java/org/obm/push/protocol/SyncProtocolTest.java
• protocol/src/test/java/org/obm/push/protocol/data/SyncDecoderTest.java
• push-bean/pom.xml
• push-bean/src/main/java/org/obm/push/bean/MimeSupport.java
• push-bean/src/main/java/org/obm/push/bean/SyncCollectionOptions.java
• push-dao-cassandra/src/test/resources/compatibility/org.obm.push.bean.SyncCollectionOptions.json
• push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceTest.java
• protocol/src/main/java/org/obm/push/protocol/data/SyncEncoder.java
• protocol/src/main/java/org/obm/push/protocol/data/SyncDecoder.java
• push-dao-cassandra/src/test/resources/compatibility/org.obm.push.bean.AnalysedSyncCollection.json
  OP-180 Make non-message related values as fields of MimePartSelector (thilaire: a0c1f421286eedd578b7056b4989c7aae69e216b)
• push-mail/src/test/java/org/obm/push/mail/imap/MailboxFetchAPITest.java
• push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
• push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
• push-mail/src/main/java/org/obm/push/mail/MimePartSelector.java
  OP-180 Make the email body optional (thilaire: e9dd4ad85936c0e5005b955f1effe3b58f9e5202)
• push-mail/src/main/java/org/obm/push/mail/conversation/EmailView.java
• integration-testing/opush/src/test/java/org/obm/opush/command/meeting/MeetingResponseHandlerTest.java
• push-bean/src/test/java/org/obm/push/bean/SerializableTest.java
• push-mail/src/test/java/org/obm/push/mail/MSMailTestsUtils.java
• push-bean/src/test/java/org/obm/push/bean/BeansTest.java
• push-mail/src/test/java/org/obm/push/mail/conversion/EmailViewTest.java
• push-mail/src/test/java/org/obm/push/mail/ReplyEmailTest.java
• push-mail/src/test/java/org/obm/push/mail/MailboxBackendTest.java
• protocol/src/main/java/org/obm/push/protocol/data/ms/MSEmailDecoder.java
• integration-testing/opush/src/test/java/org/obm/opush/WBXmlBeanSerialization.java
• integration-testing/opush/src/test/java/org/obm/opush/MailBackendHandlerTest.java
• protocol/src/main/java/org/obm/push/protocol/data/MSEmailEncoder.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerTest.java
• protocol/src/test/java/org/obm/push/protocol/data/MSEmailHeaderSerializingTest.java
• push-mail/src/test/java/org/obm/push/mail/EmailViewTestsUtils.java
• protocol/src/test/java/org/obm/push/protocol/data/MSEmailEncoderTest.java
• protocol/src/test/java/org/obm/push/protocol/ItemOperationsProtocolTest.java
• push-mail/src/main/java/org/obm/push/mail/ReplyEmail.java
• push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceCompatibilityTest.java
• push-bean/src/main/java/org/obm/push/bean/ms/MSEmailBody.java
• protocol/src/test/java/org/obm/push/protocol/data/ms/MSEmailDecoderTest.java
• push-mail/pom.xml
• push-mail/src/test/java/org/obm/push/mail/EmailChangesFetcherImplTest.java
• push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
• push-bean/src/test/java/org/obm/push/bean/ms/MSEmailTest.java
• push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceTest.java
• push-mail/src/main/java/org/obm/push/mail/MailViewToMSEmailConverterImpl.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  OP-180 Decoding email body take too much context (thilaire: 51bbac12e719cfd9dbf25add5992c6402f01e689)
• protocol/src/test/java/org/obm/push/protocol/data/ms/MSEmailDecoderTest.java
• protocol/src/main/java/org/obm/push/protocol/data/ms/MSEmailDecoder.java
  OP-180 Give iOS expected response when mime only (thilaire: 90e85134870ba8a7e850413b67767209ff2060ab)
• push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
• push-mail/src/main/java/org/obm/push/mail/EmailChangesFetcherImpl.java
• push-mail/src/main/java/org/obm/push/mail/MimePartSelector.java
• push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
• push-mail/src/test/java/org/obm/push/mail/MailBackendImplTest.java
• push-mail/src/main/java/org/obm/push/mail/MSEmailFetcher.java
• push-mail/src/test/java/org/obm/push/mail/EmailChangesFetcherImplTest.java
• push-mail/src/main/java/org/obm/push/mail/EmailChangesFetcher.java
• push-mail/src/main/java/org/obm/push/mail/MailBackendImpl.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
• push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
  OP-180 Give iOS expected response when text or html (thilaire: 01e92aa969191e873d29f6298e5b5c6f16cacad7)
• integration-testing/opush/src/test/resources/eml/textAndAttachments.eml
• integration-testing/opush/src/test/resources/eml/htmlAndAttachments.eml
• push-mail/src/main/java/org/obm/push/mail/AnyMatchBodyPreferencePolicy.java
• push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
• push-mail/src/test/java/org/obm/push/mail/AnyMatchBodyPreferencePolicyTest.java
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
• push-mail/src/main/java/org/obm/push/mail/transformer/HtmlToText.java
• push-mail/src/main/java/org/obm/push/mail/MSEmailFetcher.java
• push-mail/src/test/java/org/obm/push/mail/transformer/HtmlToTextTest.java
• push-mail/src/main/java/org/obm/push/mail/StrictMatchBodyPreferencePolicy.java
• push-mail/src/main/java/org/obm/push/mail/BodyPreferencePolicy.java
  OP-180 Test Android expected responses (thilaire: 5d175c40809b857def60ed6061eec6df6dfd2de3)
• integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java

OK with Opush 3.1.0~rc1-1

A kernel Out Of Memory reproduced with a fairphone + Opush 3.0.6 and eml trying to be retrieved 5 times:

May 19 14:10:22 obm24 kernel: [12481.812801] Out of memory: Kill process 16540 (java) score 368 or sacrifice child
May 19 14:10:22 obm24 kernel: [12481.813019] Killed process 16540 (java) total-vm:1261572kB, anon-rss:710504kB, file-rss:4kB

And not reproduced with Opush 3.1.0

About the impacts of the change, all cases will be tested with non-regression testing with several smartphones (iOS8, Android 4 and Android 5)