{"action":"created","issue":{"url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/189","repository_url":"https://api.github.com/repos/Blizzard/node-rdkafka","labels_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/189/labels{/name}","comments_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/189/comments","events_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/189/events","html_url":"https://github.com/Blizzard/node-rdkafka/issues/189","id":234065573,"number":189,"title":"Unable to produce message with an AVRO Key.","user":{"login":"joelnet","id":742630,"avatar_url":"https://avatars0.githubusercontent.com/u/742630?v=3","gravatar_id":"","url":"https://api.github.com/users/joelnet","html_url":"https://github.com/joelnet","followers_url":"https://api.github.com/users/joelnet/followers","following_url":"https://api.github.com/users/joelnet/following{/other_user}","gists_url":"https://api.github.com/users/joelnet/gists{/gist_id}","starred_url":"https://api.github.com/users/joelnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joelnet/subscriptions","organizations_url":"https://api.github.com/users/joelnet/orgs","repos_url":"https://api.github.com/users/joelnet/repos","events_url":"https://api.github.com/users/joelnet/events{/privacy}","received_events_url":"https://api.github.com/users/joelnet/received_events","type":"User","site_admin":false},"labels":[{"id":423346290,"url":"https://api.github.com/repos/Blizzard/node-rdkafka/labels/bug","name":"bug","color":"ee0701","default":true},{"id":468083305,"url":"https://api.github.com/repos/Blizzard/node-rdkafka/labels/waiting%20next%20minor","name":"waiting next minor","color":"e99695","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-06-07T01:12:48Z","updated_at":"2017-06-18T14:46:40Z","closed_at":null,"body":"I have created a publisher and consumer using the kafka-rest service found here: http://docs.confluent.io/current/kafka-rest/docs/intro.html. This is working fine.\r\n\r\nI am attempting to convert the producer to use node-rdkafka, but **I am finding the AVRO keys are being truncated**.\r\n\r\nExample of a good message (using kafka-rest):\r\n```\r\n{\"key\":\"AAAAAAEGbm93\",\"value\":\"AAAAAALAy6+TCw==\",\"partition\":0,\"offset\":46500,\"topic\":null}\r\n```\r\n\r\nExample of a bad message, missing key (using node-rdkafka):\r\n```\r\n{\"key\":\"\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n```\r\n\r\nA couple experiments:\r\n```\r\n// Buffer's\r\nconst encodedKey = encode(schema.key, keySchemaId, key);\r\nconst encodedValue = encode(schema.value, valueSchemaId, value);\r\n\r\n// key is a Buffer and will be truncated\r\nproducer.produce('topic', null, encodedValue, encodedKey);\r\n> {\"key\":\"\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n\r\n// key is a string and will be truncated\r\nproducer.produce('topic', null, encodedValue, encodedKey.toString('utf8'));\r\n> {\"key\":\"\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n\r\n// still truncated\r\nproducer.produce('topic', null, encodedValue, '\\x00\\x00\\x00\\x00\\x01\\x06now');\r\n> {\"key\":\"\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n\r\n// key is double base64 encoded (and invalid)\r\nproducer.produce('topic', null, encodedValue, encodedKey.toString('base64'));\r\n> {\"key\":\"QUFBQUFBRUdibTkz\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n\r\n// key will be set to TEST\r\nproducer.produce('topic', null, encodedValue, 'TEST\\x00\\x00\\x00\\x00\\x01\\x06now');\r\n> {\"key\":\"TEST\",\"value\":\"AAAAAALy/LiTCw==\",\"partition\":0,\"offset\":46567,\"topic\":null}\r\n```\r\n\r\nbased on my last test, it seems that the key is being truncated at the first `\\x00`, most-likely due to null-termination.\r\n\r\nI am guessing that node-rdkafka is calling a C++ `produce` method that is calling `key->size()` instead of another overload that will allow you to set the size of the key. (I'm looking in ProducerImpl.cpp at RdKafka::ProducerImpl::produce).\r\n\r\nI'm not that familiar with calling C++ from node, so this is as far as I can take this issue.\r\n\r\nThank you for taking the time to read this :)"},"comment":{"url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/comments/309281930","html_url":"https://github.com/Blizzard/node-rdkafka/issues/189#issuecomment-309281930","issue_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/189","id":309281930,"user":{"login":"smastous","id":12286898,"avatar_url":"https://avatars1.githubusercontent.com/u/12286898?v=3","gravatar_id":"","url":"https://api.github.com/users/smastous","html_url":"https://github.com/smastous","followers_url":"https://api.github.com/users/smastous/followers","following_url":"https://api.github.com/users/smastous/following{/other_user}","gists_url":"https://api.github.com/users/smastous/gists{/gist_id}","starred_url":"https://api.github.com/users/smastous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smastous/subscriptions","organizations_url":"https://api.github.com/users/smastous/orgs","repos_url":"https://api.github.com/users/smastous/repos","events_url":"https://api.github.com/users/smastous/events{/privacy}","received_events_url":"https://api.github.com/users/smastous/received_events","type":"User","site_admin":false},"created_at":"2017-06-18T14:46:40Z","updated_at":"2017-06-18T14:46:40Z","body":"Hey guys, we're also running into this problem.  Is there anything we can do to help cut a release?  We'd be happy to do some PRs or write unit tests etc if more work is needed."}}
{"actor":{"display_login":"smastous"}}