{"action":"created","issue":{"url":"https://api.github.com/repos/NaluCFD/Nalu/issues/73","repository_url":"https://api.github.com/repos/NaluCFD/Nalu","labels_url":"https://api.github.com/repos/NaluCFD/Nalu/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/NaluCFD/Nalu/issues/73/comments","events_url":"https://api.github.com/repos/NaluCFD/Nalu/issues/73/events","html_url":"https://github.com/NaluCFD/Nalu/issues/73","id":207371957,"number":73,"title":"Linear system imbalance in ABL strong-scaling simulations","user":{"login":"jhux2","id":12970120,"avatar_url":"https://avatars.githubusercontent.com/u/12970120?v=3","gravatar_id":"","url":"https://api.github.com/users/jhux2","html_url":"https://github.com/jhux2","followers_url":"https://api.github.com/users/jhux2/followers","following_url":"https://api.github.com/users/jhux2/following{/other_user}","gists_url":"https://api.github.com/users/jhux2/gists{/gist_id}","starred_url":"https://api.github.com/users/jhux2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhux2/subscriptions","organizations_url":"https://api.github.com/users/jhux2/orgs","repos_url":"https://api.github.com/users/jhux2/repos","events_url":"https://api.github.com/users/jhux2/events{/privacy}","received_events_url":"https://api.github.com/users/jhux2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-02-13T23:31:12Z","updated_at":"2017-02-14T15:34:06Z","closed_at":null,"body":"@spdomin @alanw0 @aprokop @mbarone81 \r\nIn looking at the ABL strong scaling simulations on 256, 512, and 1024 cores, we have observed that the linear system passed into MueLu becomes more imbalanced as the core count grows.  (See data below).  This will very likely affect communication in the preconditioner setup.\r\n\r\nI have some initial questions:\r\n\r\n- How is the distribution of the fine grid matrix determined?  Is it primarily from the decomposition of the underlying grid?\r\n- Are there options available in the Nalu input deck that can change/influence the matrix distribution?\r\n\r\n### 256 cores\r\n```\r\nA0 size =  2113536 x 2113536, nnz = 15155200\r\nA0 Load balancing info\r\nA0   # active processes: 256/256\r\nA0   # rows per proc   : avg = 8.26e+03,  dev =   3.4%,  min =   -7.0%,  max =   +8.7%\r\nA0   #  nnz per proc   : avg = 5.92e+04,  dev =   5.1%,  min =   -9.2%,  max =  +13.5%\r\nA0 Communication info\r\nA0   # num export send : avg =     0.00,  dev =   0.00,  min =    0.0 ,  max =    0.0\r\nA0   # num import send : avg = 2.51e+03,  dev =   5.6%,  min =  -11.0%,  max =  +10.6%\r\nA0   # num msgs        : avg =     9.50,  dev =   1.39,  min =    6.0 ,  max =   13.0\r\nA0   # min msg size    : avg = 2.56e+00,  dev = 157.3%,  min =  -61.0%,  max = +485.4%\r\nA0   # max msg size    : avg = 5.20e+02,  dev =   2.5%,  min =   -1.5%,  max =   +7.9%\r\n```\r\n\r\n### 512 cores\r\n```\r\nA0 size =  2113536 x 2113536, nnz = 15155200\r\nA0 Load balancing info\r\nA0   # active processes: 512/512\r\nA0   # rows per proc   : avg = 4.13e+03,  dev =   3.7%,  min =   -7.0%,  max =  +12.0%\r\nA0   #  nnz per proc   : avg = 2.96e+04,  dev =   7.3%,  min =   -9.2%,  max =  +24.1%\r\nA0 Communication info\r\nA0   # num export send : avg =     0.00,  dev =   0.00,  min =    0.0 ,  max =    0.0\r\nA0   # num import send : avg = 1.53e+03,  dev =   7.5%,  min =  -18.7%,  max =  +10.3%\r\nA0   # num msgs        : avg =     9.75,  dev =   1.47,  min =    6.0 ,  max =   13.0\r\nA0   # min msg size    : avg = 2.66e+00,  dev = 163.5%,  min =  -62.4%,  max = +464.7%\r\nA0   # max msg size    : avg = 2.60e+02,  dev =   2.9%,  min =   -1.5%,  max =  +11.1%\r\n```\r\n\r\n### 1024 cores\r\n```\r\nA0 size =  2113536 x 2113536, nnz = 15155200\r\nA0 Load balancing info\r\nA0   # active processes: 1024/1024\r\nA0   # rows per proc   : avg = 2.06e+03,  dev =   4.9%,  min =  -13.2%,  max =  +18.6%\r\nA0   #  nnz per proc   : avg = 1.48e+04,  dev =   7.9%,  min =  -15.2%,  max =  +31.4%\r\nA0 Communication info\r\nA0   # num export send : avg =     0.00,  dev =   0.00,  min =    0.0 ,  max =    0.0\r\nA0   # num import send : avg = 1.04e+03,  dev =   5.9%,  min =  -17.4%,  max =  +12.2%\r\nA0   # num msgs        : avg =     9.75,  dev =   1.08,  min =    6.0 ,  max =   13.0\r\nA0   # min msg size    : avg = 1.45e+00,  dev =  94.6%,  min =  -31.2%,  max = +381.7%\r\nA0   # max msg size    : avg = 2.58e+02,  dev =   2.1%,  min =   -0.8%,  max =   +5.4%\r\n```"},"comment":{"url":"https://api.github.com/repos/NaluCFD/Nalu/issues/comments/279740786","html_url":"https://github.com/NaluCFD/Nalu/issues/73#issuecomment-279740786","issue_url":"https://api.github.com/repos/NaluCFD/Nalu/issues/73","id":279740786,"user":{"login":"alanw0","id":18755013,"avatar_url":"https://avatars.githubusercontent.com/u/18755013?v=3","gravatar_id":"","url":"https://api.github.com/users/alanw0","html_url":"https://github.com/alanw0","followers_url":"https://api.github.com/users/alanw0/followers","following_url":"https://api.github.com/users/alanw0/following{/other_user}","gists_url":"https://api.github.com/users/alanw0/gists{/gist_id}","starred_url":"https://api.github.com/users/alanw0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanw0/subscriptions","organizations_url":"https://api.github.com/users/alanw0/orgs","repos_url":"https://api.github.com/users/alanw0/repos","events_url":"https://api.github.com/users/alanw0/events{/privacy}","received_events_url":"https://api.github.com/users/alanw0/received_events","type":"User","site_admin":false},"created_at":"2017-02-14T15:34:06Z","updated_at":"2017-02-14T15:34:06Z","body":"In the cases I've looked at in nalu, the element balance has been perfect, and the node balance is the issue. The node balance is what determines the linear system distribution. Currently stk-mesh doesn't do a great job of choosing processor-owners for the nodes that are shared by multiple procs (MPI ranks). The STK team has Karen Devine working on some improved load-balancing approaches that will hopefully make that better. I've tried some simple approaches to balancing the shared nodes, and it turns out to be non-trivial."}}
{"actor":{"display_login":"alanw0"}}