{"action":"created","issue":{"url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/269","repository_url":"https://api.github.com/repos/Mailtrain-org/mailtrain","labels_url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/269/labels{/name}","comments_url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/269/comments","events_url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/269/events","html_url":"https://github.com/Mailtrain-org/mailtrain/pull/269","id":238915596,"number":269,"title":"Check for locally requeued messages in postfix","user":{"login":"losacno","id":22334130,"avatar_url":"https://avatars3.githubusercontent.com/u/22334130?v=3","gravatar_id":"","url":"https://api.github.com/users/losacno","html_url":"https://github.com/losacno","followers_url":"https://api.github.com/users/losacno/followers","following_url":"https://api.github.com/users/losacno/following{/other_user}","gists_url":"https://api.github.com/users/losacno/gists{/gist_id}","starred_url":"https://api.github.com/users/losacno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/losacno/subscriptions","organizations_url":"https://api.github.com/users/losacno/orgs","repos_url":"https://api.github.com/users/losacno/repos","events_url":"https://api.github.com/users/losacno/events{/privacy}","received_events_url":"https://api.github.com/users/losacno/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-06-27T16:49:49Z","updated_at":"2017-07-06T12:28:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Mailtrain-org/mailtrain/pulls/269","html_url":"https://github.com/Mailtrain-org/mailtrain/pull/269","diff_url":"https://github.com/Mailtrain-org/mailtrain/pull/269.diff","patch_url":"https://github.com/Mailtrain-org/mailtrain/pull/269.patch"},"body":"When using Postfix + Amavis received mail first goes to Amavisd and, if not spam, is queued to Postfix Smtp service so queue id for bounce check is not the first one but the second as you can see:\r\n\r\n    Jun 27 14:30:11 server-mx postfix/smtpd[8270]: connect from localhost[127.0.0.1]\r\n    Jun 27 14:30:11 server-mx postfix/smtpd[8270]: NOQUEUE: filter: RCPT from localhost[127.0.0.1]: <mail@sender.tld>: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=<mail@sender.tld> to=<mail@receiver.tld> proto=ESMTP helo=<[127.0.0.1]>\r\n    Jun 27 14:30:11 server-mx postfix/smtpd[8270]: B2BBE8414B5: client=localhost[127.0.0.1], sasl_method=PLAIN, sasl_username=mail@sender.tld\r\n    Jun 27 14:30:11 server-mx postfix/cleanup[8313]: B2BBE8414B5: message-id=<mail-id@sender.tld>\r\n    Jun 27 14:30:11 server-mx postfix/qmgr[1694]: B2BBE8414B5: from=<mail@sender.tld>, size=123137, nrcpt=1 (queue active)\r\n    Jun 27 14:30:12 server-mx postfix/smtpd[8318]: connect from localhost[127.0.0.1]\r\n    Jun 27 14:30:12 server-mx postfix/smtpd[8318]: C642C8414CE: client=localhost[127.0.0.1]\r\n    Jun 27 14:30:12 server-mx postfix/cleanup[8313]: C642C8414CE: message-id=<mail-id@sender.tld>\r\n    Jun 27 14:30:12 server-mx postfix/smtpd[8318]: disconnect from localhost[127.0.0.1]\r\n    Jun 27 14:30:12 server-mx postfix/qmgr[1694]: C642C8414CE: from=<mail@sender.tld>, size=124068, nrcpt=1 (queue active)\r\n    Jun 27 14:30:12 server-mx amavis[5085]: (05085-15) Passed CLEAN {RelayedOutbound}, ORIGINATING LOCAL [127.0.0.1]:45421 <mail@sender.tld> -> <mail@receiver.tld>, Queue-ID: B2BBE8414B5, Message-ID: <mail-id@sender.tld>, mail_id: KFebp5elSvWC, Hits: -0.997, size: 123137, queued_as: C642C8414CE, dkim_new=dkim:sender.tld, 1048 ms\r\n    Jun 27 14:30:12 server-mx postfix/smtp[8314]: B2BBE8414B5: to=<mail@receiver.tld>, relay=127.0.0.1[127.0.0.1]:10026, delay=1.2, delays=0.11/0.01/0/1.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10027): 250 2.0.0 Ok: queued as C642C8414CE)\r\n    Jun 27 14:30:12 server-mx postfix/qmgr[1694]: B2BBE8414B5: removed\r\n    Jun 27 14:30:15 server-mx postfix/smtp[8320]: C642C8414CE: to=<mail@receiver.tld>, relay=remote-mx.tld[2.2.2.2]:25, delay=2.5, delays=0.07/0.01/2.1/0.26, dsn=5.7.1, status=bounced (host remote-mx.tld[2.2.2.2] said: 550 5.7.1 <mail@receiver.tld>: Recipient address rejected: User unknown. (in reply to RCPT TO command))\r\n    Jun 27 14:30:15 server-mx postfix/cleanup[8313]: 6DE8F841563: message-id=<new-mail-id@server.sender.tld>\r\n    Jun 27 14:30:15 server-mx postfix/bounce[8324]:C642C8414CE: sender non-delivery notification: 6DE8F841563\r\n\r\nThis PR try to solve this problem saving locally requeued messages in an array and checking bounces against it"},"comment":{"url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/comments/313381882","html_url":"https://github.com/Mailtrain-org/mailtrain/pull/269#issuecomment-313381882","issue_url":"https://api.github.com/repos/Mailtrain-org/mailtrain/issues/269","id":313381882,"user":{"login":"witzig","id":4999980,"avatar_url":"https://avatars2.githubusercontent.com/u/4999980?v=3","gravatar_id":"","url":"https://api.github.com/users/witzig","html_url":"https://github.com/witzig","followers_url":"https://api.github.com/users/witzig/followers","following_url":"https://api.github.com/users/witzig/following{/other_user}","gists_url":"https://api.github.com/users/witzig/gists{/gist_id}","starred_url":"https://api.github.com/users/witzig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/witzig/subscriptions","organizations_url":"https://api.github.com/users/witzig/orgs","repos_url":"https://api.github.com/users/witzig/repos","events_url":"https://api.github.com/users/witzig/events{/privacy}","received_events_url":"https://api.github.com/users/witzig/received_events","type":"User","site_admin":false},"created_at":"2017-07-06T12:28:39Z","updated_at":"2017-07-06T12:28:39Z","body":"@losacno, with your PR, processing the postfix logs of 1 million successfully sent emails will raise memory usage quite a bit. Testing with:\r\n\r\n```javascript\r\nlet seenIds = new Set();\r\nlet queueIds = {};\r\n\r\nconsole.log(process.memoryUsage().rss / 1048576 + ' MB');\r\n\r\nfor (let i = 0; i < 1000000; i++) {\r\n    let id = '#' + (1000000000 + i);\r\n    seenIds.add(id);\r\n    queueIds[id] = id;\r\n}\r\n\r\nconsole.log(process.memoryUsage().rss / 1048576 + ' MB');\r\n```\r\n\r\nLogs:\r\n\r\n```\r\n27.16796875 MB\r\n361.02734375 MB\r\n```\r\n\r\nI think the `seenIds` and `queueIds` will have to expire somehow. Perhaps [memory-cache](https://github.com/ptarjan/node-cache) could be an easy fix for this problem, yet there might be a better solution around."}}
{"actor":{"display_login":"witzig"}}