[OP-180] Limit the amount of data retreived by IMAP on emails with attachments only Created: 09/Mar/15  Updated: 19/May/15  Resolved: 18/May/15

Status: Closed
Project: Opush
Component/s: None
Affects Version/s: 3.1.0
Fix Version/s: 3.1.0

Type: Bug Priority: Normal
Reporter: Antoine DUPRAT Assignee: Thomas HILAIRE
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File before.png     PNG File end.png    
Issue Links:
Dependency
depends on OBMFULL-6390 linagora-imap - Less memory comsuptio... Closed
Relates
relates to OBMFULL-6375 UIDFetchMessageCommand should support... Closed
Rank: 6789
Sprint: OBM Sprint 2015#3

 Description   

About

1 - The memory consumption has been improved to prevent OutOfMemory error when synchronizing a window of huge emails
2 - Many captures from an exchange has been done. Opush now mimic it.

Reproduction steps

As usera imports 5 times OP-180.eml (there is private pictures in this email, ask me if you need it)
As usera synchronize with his gs3 phone

Current behavior

  1. NOK an OutOfMemory exception is triggered

Expected behavior

  1. OK no exception is triggered


 Comments   
Comment by Antoine DUPRAT [ 12/Mar/15 ]

QA: check that the client truncation size is respected on huge mails

Client request:

<Sync>
<Collections>
<Collection>
<SyncKey>ecc038bd-1d4a-4dc1-97b5-79e2848af50e</SyncKey>
<CollectionId>12</CollectionId>
<DeletesAsMoves>1</DeletesAsMoves>
<GetChanges/>
<WindowSize>10</WindowSize>
<Options>
<FilterType>0</FilterType>
<BodyPreference>
<Type>2</Type>
<TruncationSize>200000</TruncationSize>
</BodyPreference>
</Options>
</Collection>
</Collections>
</Sync>

IMAP logs:

09:29:39.474 [INFO ] {IMAP.COMMAND} [t36/cSync/r6/usera@adupratvm.lyon.lan] C: A11 UID FETCH 5 (UID BODY.PEEK[1]<0.200000>)
09:29:39.477 [INFO ] {IMAP.COMMAND} [t36/cSync/r6/usera@adupratvm.lyon.lan] Command status SUCCESS
Comment by Jenkins Continuous Integration Server [ 13/Mar/15 ]

ABORTED: Integrated in opush-master #205
OP-180 Use truncation when fetching mail (thilaire: 62e69d98bc021dad36153fb4d7f5f16086d356df)

  • push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
  • pom.xml
  • push-mail/src/test/java/org/obm/push/mail/imap/MailboxServiceTest.java
  • client/src/main/java/org/obm/sync/push/client/commands/Sync.java
  • integration-testing/opush/src/test/resources/eml/OP-180.eml
  • push-mail/src/test/java/org/obm/push/mail/imap/MailboxStoreAPITest.java
  • push-mail/src/main/java/org/obm/push/mail/MailboxService.java
  • push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
  • push-mail/src/main/java/org/obm/push/mail/imap/LinagoraMailboxService.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  • integration-testing/opush/src/main/java/org/obm/opush/IntegrationTestUtils.java
  • push-mail/src/test/java/org/obm/push/mail/MailboxBackendTest.java
Comment by Jenkins Continuous Integration Server [ 13/Mar/15 ]

ABORTED: Integrated in opush-master #206
OP-180 Use truncation when fetching mail (thilaire: bc95128598e2d1b893f8cfa0788f666ede0757ca)

  • push-mail/src/test/java/org/obm/push/mail/imap/MailboxStoreAPITest.java
  • push-mail/src/main/java/org/obm/push/mail/MailboxService.java
  • pom.xml
  • client/src/main/java/org/obm/sync/push/client/commands/Sync.java
  • push-mail/src/test/java/org/obm/push/mail/MailboxBackendTest.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  • push-mail/src/test/java/org/obm/push/mail/imap/MailboxServiceTest.java
  • integration-testing/opush/src/test/resources/eml/only_attachments.eml
  • push-mail/src/main/java/org/obm/push/mail/imap/LinagoraMailboxService.java
  • push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
  • push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
  • integration-testing/opush/src/main/java/org/obm/opush/IntegrationTestUtils.java
Comment by Thomas HILAIRE [ 02/Apr/15 ]

NOK many emails display are broken on iOS7 and were OK with opush 3.0.6. We really should mimic the exchange behavior for this kind of stuff to let this kind of issue behind us.

I create some eml to tests that and I made every required capture on Exchange. I won't attach them to this ticket because their are too heavy.

Comment by Thomas HILAIRE [ 24/Apr/15 ]

Twice less object (when > 1kib) allocations are done with improvements done in this issue, see before.png and end.png attachments

Comment by Thomas HILAIRE [ 05/May/15 ]

QA & Impacts:

  • all kind of emails synchronization
    • text only
    • html only
    • text and html
    • with or without attachment
    • etc
  • such emails synchronization with all kinds of devices
  • replying and forwarding
Comment by Jenkins Continuous Integration Server [ 18/May/15 ]

FAILURE: Integrated in opush-master #229
OP-180 Use linagora-imap version with fewer memory usage (thilaire: d661578af0edca3d9005379d9891d7b0fc6efdee)

  • pom.xml
    OP-180 Limit the log size (thilaire: 2218d3dc368be955099efbb87863a4dfc819d552)
  • push-core/src/test/java/org/obm/push/utils/LimitedOutputStreamTest.java
  • push-core/src/main/java/org/obm/push/utils/LimitedOutputStream.java
  • push-core/src/main/java/org/obm/push/impl/DOMDumper.java
    OP-180 pipe streams to avoid ByteArray and String allocation (thilaire: effc7253ebc24d5efaf6d9de0f8dc007cb8310e9)
  • wbxml/src/main/java/org/obm/push/wbxml/parsers/WbxmlEncoder.java
  • wbxml/src/main/java/org/obm/push/wbxml/WBXMLTools.java
    OP-180 Use a writer to avoid array and String allocations (thilaire: 3d40f4147ba47224fd157423e638c59c36cec147)
  • wbxml/src/main/java/org/obm/push/wbxml/parsers/WbxmlEncoder.java
  • wbxml/src/main/java/org/obm/push/wbxml/parsers/EncoderHandler.java
    OP-180 Use the regular protocol encoder for sync test (thilaire: fe5bd4b0498c977e0a3169d85618ac21309f373d)
  • integration-testing/opush/src/test/java/org/obm/opush/MailBackendGetChangedTest.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerOnCalendarsTest.java
  • client/src/main/java/org/obm/sync/push/client/commands/Sync.java
  • protocol/src/main/java/org/obm/push/protocol/data/ActiveSyncFields.java
  • protocol/src/main/java/org/obm/push/protocol/data/SyncEncoder.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerTest.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerOnContactsTest.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
    OP-180 Strong typing of MimeSupport (thilaire: 0832aa453b0be3651ea5856fc9381e62767ec91a)
  • protocol/src/test/java/org/obm/push/protocol/bean/SyncCollectionTest.java
  • push-bean/src/test/java/org/obm/push/bean/SyncCollectionOptionsTest.java
  • push-dao-cassandra/src/main/java/org/obm/push/json/JSONService.java
  • push-dao-cassandra/src/main/java/org/obm/push/json/mixin/MimeSupportMixIn.java
  • push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceCompatibilityTest.java
  • push-bean/src/test/java/org/obm/push/bean/BeansTest.java
  • protocol/src/test/java/org/obm/push/protocol/SyncProtocolTest.java
  • protocol/src/test/java/org/obm/push/protocol/data/SyncDecoderTest.java
  • push-bean/pom.xml
  • push-bean/src/main/java/org/obm/push/bean/MimeSupport.java
  • push-bean/src/main/java/org/obm/push/bean/SyncCollectionOptions.java
  • push-dao-cassandra/src/test/resources/compatibility/org.obm.push.bean.SyncCollectionOptions.json
  • push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceTest.java
  • protocol/src/main/java/org/obm/push/protocol/data/SyncEncoder.java
  • protocol/src/main/java/org/obm/push/protocol/data/SyncDecoder.java
  • push-dao-cassandra/src/test/resources/compatibility/org.obm.push.bean.AnalysedSyncCollection.json
    OP-180 Make non-message related values as fields of MimePartSelector (thilaire: a0c1f421286eedd578b7056b4989c7aae69e216b)
  • push-mail/src/test/java/org/obm/push/mail/imap/MailboxFetchAPITest.java
  • push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
  • push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
  • push-mail/src/main/java/org/obm/push/mail/MimePartSelector.java
    OP-180 Make the email body optional (thilaire: e9dd4ad85936c0e5005b955f1effe3b58f9e5202)
  • push-mail/src/main/java/org/obm/push/mail/conversation/EmailView.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/meeting/MeetingResponseHandlerTest.java
  • push-bean/src/test/java/org/obm/push/bean/SerializableTest.java
  • push-mail/src/test/java/org/obm/push/mail/MSMailTestsUtils.java
  • push-bean/src/test/java/org/obm/push/bean/BeansTest.java
  • push-mail/src/test/java/org/obm/push/mail/conversion/EmailViewTest.java
  • push-mail/src/test/java/org/obm/push/mail/ReplyEmailTest.java
  • push-mail/src/test/java/org/obm/push/mail/MailboxBackendTest.java
  • protocol/src/main/java/org/obm/push/protocol/data/ms/MSEmailDecoder.java
  • integration-testing/opush/src/test/java/org/obm/opush/WBXmlBeanSerialization.java
  • integration-testing/opush/src/test/java/org/obm/opush/MailBackendHandlerTest.java
  • protocol/src/main/java/org/obm/push/protocol/data/MSEmailEncoder.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerTest.java
  • protocol/src/test/java/org/obm/push/protocol/data/MSEmailHeaderSerializingTest.java
  • push-mail/src/test/java/org/obm/push/mail/EmailViewTestsUtils.java
  • protocol/src/test/java/org/obm/push/protocol/data/MSEmailEncoderTest.java
  • protocol/src/test/java/org/obm/push/protocol/ItemOperationsProtocolTest.java
  • push-mail/src/main/java/org/obm/push/mail/ReplyEmail.java
  • push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceCompatibilityTest.java
  • push-bean/src/main/java/org/obm/push/bean/ms/MSEmailBody.java
  • protocol/src/test/java/org/obm/push/protocol/data/ms/MSEmailDecoderTest.java
  • push-mail/pom.xml
  • push-mail/src/test/java/org/obm/push/mail/EmailChangesFetcherImplTest.java
  • push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
  • push-bean/src/test/java/org/obm/push/bean/ms/MSEmailTest.java
  • push-dao-cassandra/src/test/java/org/obm/push/json/JSONServiceTest.java
  • push-mail/src/main/java/org/obm/push/mail/MailViewToMSEmailConverterImpl.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
    OP-180 Decoding email body take too much context (thilaire: 51bbac12e719cfd9dbf25add5992c6402f01e689)
  • protocol/src/test/java/org/obm/push/protocol/data/ms/MSEmailDecoderTest.java
  • protocol/src/main/java/org/obm/push/protocol/data/ms/MSEmailDecoder.java
    OP-180 Give iOS expected response when mime only (thilaire: 90e85134870ba8a7e850413b67767209ff2060ab)
  • push-mail/src/test/java/org/obm/push/mail/EmailViewPartsFetcherImplTest.java
  • push-mail/src/main/java/org/obm/push/mail/EmailChangesFetcherImpl.java
  • push-mail/src/main/java/org/obm/push/mail/MimePartSelector.java
  • push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
  • push-mail/src/test/java/org/obm/push/mail/MailBackendImplTest.java
  • push-mail/src/main/java/org/obm/push/mail/MSEmailFetcher.java
  • push-mail/src/test/java/org/obm/push/mail/EmailChangesFetcherImplTest.java
  • push-mail/src/main/java/org/obm/push/mail/EmailChangesFetcher.java
  • push-mail/src/main/java/org/obm/push/mail/MailBackendImpl.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  • push-mail/src/main/java/org/obm/push/mail/EmailViewPartsFetcherImpl.java
    OP-180 Give iOS expected response when text or html (thilaire: 01e92aa969191e873d29f6298e5b5c6f16cacad7)
  • integration-testing/opush/src/test/resources/eml/textAndAttachments.eml
  • integration-testing/opush/src/test/resources/eml/htmlAndAttachments.eml
  • push-mail/src/main/java/org/obm/push/mail/AnyMatchBodyPreferencePolicy.java
  • push-mail/src/test/java/org/obm/push/mail/MimePartSelectorTest.java
  • push-mail/src/test/java/org/obm/push/mail/AnyMatchBodyPreferencePolicyTest.java
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
  • push-mail/src/main/java/org/obm/push/mail/transformer/HtmlToText.java
  • push-mail/src/main/java/org/obm/push/mail/MSEmailFetcher.java
  • push-mail/src/test/java/org/obm/push/mail/transformer/HtmlToTextTest.java
  • push-mail/src/main/java/org/obm/push/mail/StrictMatchBodyPreferencePolicy.java
  • push-mail/src/main/java/org/obm/push/mail/BodyPreferencePolicy.java
    OP-180 Test Android expected responses (thilaire: 5d175c40809b857def60ed6061eec6df6dfd2de3)
  • integration-testing/opush/src/test/java/org/obm/opush/command/sync/SyncHandlerWithBackendTest.java
Comment by Stephane COLSON [ 19/May/15 ]

OK with Opush 3.1.0~rc1-1

A kernel Out Of Memory reproduced with a fairphone + Opush 3.0.6 and eml trying to be retrieved 5 times:

May 19 14:10:22 obm24 kernel: [12481.812801] Out of memory: Kill process 16540 (java) score 368 or sacrifice child
May 19 14:10:22 obm24 kernel: [12481.813019] Killed process 16540 (java) total-vm:1261572kB, anon-rss:710504kB, file-rss:4kB

And not reproduced with Opush 3.1.0

About the impacts of the change, all cases will be tested with non-regression testing with several smartphones (iOS8, Android 4 and Android 5)

Generated at Mon Oct 14 00:45:40 CEST 2019 using JIRA 6.1.1#6155-sha1:7188aeec9a6b57d61ea04c52f235f15f55c105e2.