{"action":"created","comment":{"url":"https://api.github.com/repos/citusdata/citus/pulls/comments/132504322","pull_request_review_id":55588989,"id":132504322,"diff_hunk":"@@ -24,12 +26,352 @@\n #include \"utils/hsearch.h\"\n #include \"utils/timestamp.h\"\n \n+/* GUC, determining whether debug messages for deadlock detection sent to LOG */\n+bool LogDistributedDeadlockDetection = false;\n+\n+/* we access local group quite heavily, thus cache it via a local variable */\n+static int localGroupId = -1;\n \n+static bool CheckDeadlockForTransactionNode(TransactionNode *startingTransactionNode,\n+\t\t\t\t\t\t\t\t\t\t\tTransactionNode **transactionNodeStack,\n+\t\t\t\t\t\t\t\t\t\t\tList **deadlockPath);\n+static void PrependOutgoingNodesToQueue(TransactionNode *queuedTransactionNode,\n+\t\t\t\t\t\t\t\t\t\tint currentStackDepth,\n+\t\t\t\t\t\t\t\t\t\tList **toBeVisitedNodes);\n+static void BuildDeadlockPathList(QueuedTransactionNode *cycledTransactionNode,\n+\t\t\t\t\t\t\t\t  TransactionNode **transactionNodeStack,\n+\t\t\t\t\t\t\t\t  List **deadlockPath);\n+static void ResetVisitedFields(HTAB *adjacencyList);\n+static void AssociateDistributedTransactionWithBackendProc(TransactionNode *\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t   transactionNode);\n static TransactionNode * GetOrCreateTransactionNode(HTAB *adjacencyList,\n \t\t\t\t\t\t\t\t\t\t\t\t\tDistributedTransactionId *\n \t\t\t\t\t\t\t\t\t\t\t\t\ttransactionId);\n static uint32 DistributedTransactionIdHash(const void *key, Size keysize);\n-static int DistributedTransactionIdCompare(const void *a, const void *b, Size keysize);\n+static int DistributedTransactionIdCompareHash(const void *a, const void *b, Size\n+\t\t\t\t\t\t\t\t\t\t\t   keysize);\n+static int DistributedTransactionIdCompare(const void *a, const void *b);\n+static void LogCancellingBackend(TransactionNode *transactionNode);\n+static void LogTransactionNode(TransactionNode *transactionNode);\n+static void LogDistributedDeadlockDebugMessage(const char *errorMessage);\n+\n+\n+PG_FUNCTION_INFO_V1(check_distributed_deadlocks);\n+\n+\n+/*\n+ * check_distributed_deadlocks is the external API for manually\n+ * checking for distributed deadlocks. For the details, see\n+ * CheckForDistributedDeadlocks().\n+ */\n+Datum\n+check_distributed_deadlocks(PG_FUNCTION_ARGS)\n+{\n+\tbool deadlockFound = CheckForDistributedDeadlocks();\n+\n+\treturn BoolGetDatum(deadlockFound);\n+}\n+\n+\n+/*\n+ * CheckForDistributedDeadlocks is the entry point for detecing\n+ * distributed deadlocks.\n+ *\n+ * In plain words, the function first builds a wait graph by\n+ * adding the wait edges from the local node and then adding the\n+ * remote wait edges to form a global wait graph. Later, the wait\n+ * graph is converted into another graph representation (adjacency\n+ * lists) for more efficient searches. Finally, a DFS is done on\n+ * the adjacency lists. Finding a cycle in the graph unveils a\n+ * distributed deadlock. Upon finding a deadlock, the youngest\n+ * participant backend is cancelled.\n+ *\n+ * The function returns true if a deadlock is found. Otherwise, returns\n+ * false.\n+ */\n+bool\n+CheckForDistributedDeadlocks(void)\n+{\n+\tWaitGraph *waitGraph = BuildGlobalWaitGraph();\n+\tHTAB *adjacencyLists = BuildAdjacencyListsForWaitGraph(waitGraph);\n+\tHASH_SEQ_STATUS status;\n+\tTransactionNode *transactionNode = NULL;\n+\tint edgeCount = waitGraph->edgeCount;\n+\n+\t/* avoid calling GetLocalGroup() many times by caching for the session */\n+\tif (localGroupId == -1)","path":"src/backend/distributed/transaction/distributed_deadlock_detection.c","position":88,"original_position":88,"commit_id":"92613c9525d20946207e352c6d35201566e9ff39","original_commit_id":"c79aac6360d596ecbe0b17c57eab14a2e6c550bc","user":{"login":"anarazel","id":1598211,"avatar_url":"https://avatars1.githubusercontent.com/u/1598211?v=4","gravatar_id":"","url":"https://api.github.com/users/anarazel","html_url":"https://github.com/anarazel","followers_url":"https://api.github.com/users/anarazel/followers","following_url":"https://api.github.com/users/anarazel/following{/other_user}","gists_url":"https://api.github.com/users/anarazel/gists{/gist_id}","starred_url":"https://api.github.com/users/anarazel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anarazel/subscriptions","organizations_url":"https://api.github.com/users/anarazel/orgs","repos_url":"https://api.github.com/users/anarazel/repos","events_url":"https://api.github.com/users/anarazel/events{/privacy}","received_events_url":"https://api.github.com/users/anarazel/received_events","type":"User","site_admin":false},"body":"Only store on function local level.","created_at":"2017-08-10T16:33:05Z","updated_at":"2017-08-10T17:01:52Z","html_url":"https://github.com/citusdata/citus/pull/1529#discussion_r132504322","pull_request_url":"https://api.github.com/repos/citusdata/citus/pulls/1529","_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus/pulls/comments/132504322"},"html":{"href":"https://github.com/citusdata/citus/pull/1529#discussion_r132504322"},"pull_request":{"href":"https://api.github.com/repos/citusdata/citus/pulls/1529"}}},"pull_request":{"url":"https://api.github.com/repos/citusdata/citus/pulls/1529","id":133491643,"html_url":"https://github.com/citusdata/citus/pull/1529","diff_url":"https://github.com/citusdata/citus/pull/1529.diff","patch_url":"https://github.com/citusdata/citus/pull/1529.patch","issue_url":"https://api.github.com/repos/citusdata/citus/issues/1529","number":1529,"state":"open","locked":false,"title":"Deadlock detection main","user":{"login":"onderkalaci","id":7829470,"avatar_url":"https://avatars2.githubusercontent.com/u/7829470?v=4","gravatar_id":"","url":"https://api.github.com/users/onderkalaci","html_url":"https://github.com/onderkalaci","followers_url":"https://api.github.com/users/onderkalaci/followers","following_url":"https://api.github.com/users/onderkalaci/following{/other_user}","gists_url":"https://api.github.com/users/onderkalaci/gists{/gist_id}","starred_url":"https://api.github.com/users/onderkalaci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/onderkalaci/subscriptions","organizations_url":"https://api.github.com/users/onderkalaci/orgs","repos_url":"https://api.github.com/users/onderkalaci/repos","events_url":"https://api.github.com/users/onderkalaci/events{/privacy}","received_events_url":"https://api.github.com/users/onderkalaci/received_events","type":"User","site_admin":false},"body":"This is a WIP, opened to get some early feedback. The PR detects distributed deadlocks and kills one of the participating backends on the coordinator.\r\n\r\nThings to complete before merge:\r\n\r\n* [x] Add citus.deadlock_timeout GUC, as 1+ float factor of postgres' deadlock timeout\r\n* [x] Discuss removing citus.enable_deadlock_prevention (I couldn't see any discussions on removing the GUC, are we planning to keep it?)\r\n* [x] Add more regression tests, currently there is only one.\r\n* [x] Add more meaningful error messages, currently the only error message is `ERROR:  canceling statement due to user request` (See Andres' prototype on how to implement it)\r\n* [x] Add some debug messages as Postgres does\r\n* [x] Make sure that deadlock detection works on MX as expected (See Marco's `deadlock_detection` branch)\r\n* [x] Lessen the debug messages\r\n* [ ] Do some benchmarks\r\n* [ ] Consider failure cases a bit more. Be creative to generate test cases.\r\n* [ ] Split into smaller commits","created_at":"2017-08-01T14:45:02Z","updated_at":"2017-08-10T17:01:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cef468ba8ef3b6bb320f5789242589c467d7a967","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/citusdata/citus/pulls/1529/commits","review_comments_url":"https://api.github.com/repos/citusdata/citus/pulls/1529/comments","review_comment_url":"https://api.github.com/repos/citusdata/citus/pulls/comments{/number}","comments_url":"https://api.github.com/repos/citusdata/citus/issues/1529/comments","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/92613c9525d20946207e352c6d35201566e9ff39","head":{"label":"citusdata:deadlock_detection_main","ref":"deadlock_detection_main","sha":"92613c9525d20946207e352c6d35201566e9ff39","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":50874442,"name":"citus","full_name":"citusdata/citus","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus","description":"Scalable PostgreSQL for multi-tenant and real-time workloads","fork":false,"url":"https://api.github.com/repos/citusdata/citus","forks_url":"https://api.github.com/repos/citusdata/citus/forks","keys_url":"https://api.github.com/repos/citusdata/citus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus/teams","hooks_url":"https://api.github.com/repos/citusdata/citus/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus/events","assignees_url":"https://api.github.com/repos/citusdata/citus/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus/tags","blobs_url":"https://api.github.com/repos/citusdata/citus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus/subscription","commits_url":"https://api.github.com/repos/citusdata/citus/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus/merges","archive_url":"https://api.github.com/repos/citusdata/citus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus/downloads","issues_url":"https://api.github.com/repos/citusdata/citus/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus/deployments","created_at":"2016-02-01T21:58:39Z","updated_at":"2017-08-10T14:25:10Z","pushed_at":"2017-08-10T16:48:54Z","git_url":"git://github.com/citusdata/citus.git","ssh_url":"git@github.com:citusdata/citus.git","clone_url":"https://github.com/citusdata/citus.git","svn_url":"https://github.com/citusdata/citus","homepage":"https://www.citusdata.com","size":7796,"stargazers_count":2153,"watchers_count":2153,"language":"C","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":156,"mirror_url":null,"open_issues_count":431,"forks":156,"open_issues":431,"watchers":2153,"default_branch":"master"}},"base":{"label":"citusdata:master","ref":"master","sha":"9d93fb5551fc8cda33a8811241c2a212af1e3979","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":50874442,"name":"citus","full_name":"citusdata/citus","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus","description":"Scalable PostgreSQL for multi-tenant and real-time workloads","fork":false,"url":"https://api.github.com/repos/citusdata/citus","forks_url":"https://api.github.com/repos/citusdata/citus/forks","keys_url":"https://api.github.com/repos/citusdata/citus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus/teams","hooks_url":"https://api.github.com/repos/citusdata/citus/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus/events","assignees_url":"https://api.github.com/repos/citusdata/citus/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus/tags","blobs_url":"https://api.github.com/repos/citusdata/citus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus/subscription","commits_url":"https://api.github.com/repos/citusdata/citus/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus/merges","archive_url":"https://api.github.com/repos/citusdata/citus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus/downloads","issues_url":"https://api.github.com/repos/citusdata/citus/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus/deployments","created_at":"2016-02-01T21:58:39Z","updated_at":"2017-08-10T14:25:10Z","pushed_at":"2017-08-10T16:48:54Z","git_url":"git://github.com/citusdata/citus.git","ssh_url":"git@github.com:citusdata/citus.git","clone_url":"https://github.com/citusdata/citus.git","svn_url":"https://github.com/citusdata/citus","homepage":"https://www.citusdata.com","size":7796,"stargazers_count":2153,"watchers_count":2153,"language":"C","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":156,"mirror_url":null,"open_issues_count":431,"forks":156,"open_issues":431,"watchers":2153,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus/pulls/1529"},"html":{"href":"https://github.com/citusdata/citus/pull/1529"},"issue":{"href":"https://api.github.com/repos/citusdata/citus/issues/1529"},"comments":{"href":"https://api.github.com/repos/citusdata/citus/issues/1529/comments"},"review_comments":{"href":"https://api.github.com/repos/citusdata/citus/pulls/1529/comments"},"review_comment":{"href":"https://api.github.com/repos/citusdata/citus/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/citusdata/citus/pulls/1529/commits"},"statuses":{"href":"https://api.github.com/repos/citusdata/citus/statuses/92613c9525d20946207e352c6d35201566e9ff39"}}}}
{"actor":{"display_login":"anarazel"}}