{"action":"created","issue":{"url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/4","repository_url":"https://api.github.com/repos/teambition/kafka-connect-mongo","labels_url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/4/comments","events_url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/4/events","html_url":"https://github.com/teambition/kafka-connect-mongo/issues/4","id":249624690,"number":4,"title":"record loss because of maxMessageSize?","user":{"login":"sabinthomas","id":797044,"avatar_url":"https://avatars2.githubusercontent.com/u/797044?v=4","gravatar_id":"","url":"https://api.github.com/users/sabinthomas","html_url":"https://github.com/sabinthomas","followers_url":"https://api.github.com/users/sabinthomas/followers","following_url":"https://api.github.com/users/sabinthomas/following{/other_user}","gists_url":"https://api.github.com/users/sabinthomas/gists{/gist_id}","starred_url":"https://api.github.com/users/sabinthomas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sabinthomas/subscriptions","organizations_url":"https://api.github.com/users/sabinthomas/orgs","repos_url":"https://api.github.com/users/sabinthomas/repos","events_url":"https://api.github.com/users/sabinthomas/events{/privacy}","received_events_url":"https://api.github.com/users/sabinthomas/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-08-11T12:45:05Z","updated_at":"2017-11-09T03:10:11Z","closed_at":"2017-08-23T01:38:51Z","author_association":"NONE","body":"EDIT: Editing my earlier question, after I reviewed the code some more, and understood the purpose of the TailableAwait cursor\r\n\r\nHi,\r\nI believe I am losing records when using Mongo as a source. \r\n\r\nI reviewed the code in [DatabaseReader.kt](https://github.com/teambition/kafka-connect-mongo/blob/17abbcae41f4386897e2618b65990e30b907c1af/src/main/kotlin/org/apache/kafka/connect/mongo/DatabaseReader.kt#L61)\r\n\r\n```kotlin\r\n                // Stop pulling data when length of message is too large!\r\n                while (messages.size > maxMessageSize) {\r\n                    log.warn(\"Message overwhelm! database {}, docs {}, messages {}\",\r\n                            db,\r\n                            count,\r\n                            messages.size)\r\n                    Thread.sleep(500)\r\n                }\r\n```\r\nI believe the section of code where the Thread is put to sleep if it exceeds the maxMessageSize that is currently set to `2000` causes loss of record insertion into Mongo. Is there any alternative besides Thread.sleep()?\r\n"},"comment":{"url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/comments/343034357","html_url":"https://github.com/teambition/kafka-connect-mongo/issues/4#issuecomment-343034357","issue_url":"https://api.github.com/repos/teambition/kafka-connect-mongo/issues/4","id":343034357,"user":{"login":"sabinthomas","id":797044,"avatar_url":"https://avatars2.githubusercontent.com/u/797044?v=4","gravatar_id":"","url":"https://api.github.com/users/sabinthomas","html_url":"https://github.com/sabinthomas","followers_url":"https://api.github.com/users/sabinthomas/followers","following_url":"https://api.github.com/users/sabinthomas/following{/other_user}","gists_url":"https://api.github.com/users/sabinthomas/gists{/gist_id}","starred_url":"https://api.github.com/users/sabinthomas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sabinthomas/subscriptions","organizations_url":"https://api.github.com/users/sabinthomas/orgs","repos_url":"https://api.github.com/users/sabinthomas/repos","events_url":"https://api.github.com/users/sabinthomas/events{/privacy}","received_events_url":"https://api.github.com/users/sabinthomas/received_events","type":"User","site_admin":false},"created_at":"2017-11-09T03:10:11Z","updated_at":"2017-11-09T03:10:11Z","author_association":"NONE","body":"ah sorry, I edited the topic before I posted it here, it should be \r\n```\r\n$ kafka-topics --describe --zookeeper localhost:2181 --topic in_collectionA\r\nTopic:in_collectionA\tPartitionCount:1\tReplicationFactor:1\tConfigs:\r\n\tTopic: in_collectionA\tPartition: 0\tLeader: 0\tReplicas: 0\tIsr: 0\r\n```"}}
{"actor":{"display_login":"sabinthomas"}}