{"action":"closed","issue":{"url":"https://api.github.com/repos/pravega/pravega/issues/1505","repository_url":"https://api.github.com/repos/pravega/pravega","labels_url":"https://api.github.com/repos/pravega/pravega/issues/1505/labels{/name}","comments_url":"https://api.github.com/repos/pravega/pravega/issues/1505/comments","events_url":"https://api.github.com/repos/pravega/pravega/issues/1505/events","html_url":"https://github.com/pravega/pravega/issues/1505","id":239578407,"number":1505,"title":"Sporadic Build Failure: Storage Writer vs Read Index on shutdown","user":{"login":"andreipaduroiu","id":14354377,"avatar_url":"https://avatars1.githubusercontent.com/u/14354377?v=3","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"andreipaduroiu","id":14354377,"avatar_url":"https://avatars1.githubusercontent.com/u/14354377?v=3","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false},"assignees":[{"login":"andreipaduroiu","id":14354377,"avatar_url":"https://avatars1.githubusercontent.com/u/14354377?v=3","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2017-06-29T18:55:45Z","updated_at":"2017-06-30T09:51:48Z","closed_at":"2017-06-30T09:51:48Z","body":"**Problem description**\r\nhttps://travis-ci.org/pravega/pravega/builds/247945863\r\n\r\nObserved symptom:\r\n```\r\nio.pravega.segmentstore.server.containers.StreamSegmentContainerTests > testMetadataCleanupRecovery FAILED\r\n    java.lang.IllegalStateException: Expected the service MetadataCleanupContainer [FAILED] to be TERMINATED, but the service has FAILED\r\n        at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:328)\r\n        at com.google.common.util.concurrent.AbstractService.awaitTerminated(AbstractService.java:293)\r\n        at io.pravega.segmentstore.server.containers.StreamSegmentContainerTests.testMetadataCleanupRecovery(StreamSegmentContainerTests.java:911)\r\n        Caused by:\r\n        io.pravega.segmentstore.server.DataCorruptionException: Unable to retrieve CacheContents for 'AggregatedAppend: SegmentId = 3, Offsets = [4806-4810), SeqNo = 4903'.\r\n```\r\n\r\n**Problem location**\r\n* `StreamSegmentReadIndex` and `StorageWriter/SegmentAggregator`:\r\n* `The StorageWriter` takes some time to shut down (due to pending IO or something else), and in the meantime we close the Read Index, hence the cache is cleared\r\n* The `StorageWriter` (`SegmentAggregator`) tries to get some data out of the read index (since it's in the middle of an operation), but the latter wrongfully does not throw `ObjectClosedException`, instead returns null. This is interpreted as a `DataCorruptionException`\r\n\r\n**Suggestions for an improvement**\r\n* Add a check for `closed` in the relevant method in `StreamSegmentReadIndex`\r\n* Update `StorageWriter` to ignore `ObjectClosedException`s when shutting down - since they are likely triggered by the shutdown.\r\n* This is virtually impossible to detect and/or reproduce reliably via a unit test."}}
{"actor":{"display_login":"fpj"}}