{"action":"created","issue":{"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/553","repository_url":"https://api.github.com/repos/SubPointSolutions/spmeta2","labels_url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/553/labels{/name}","comments_url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/553/comments","events_url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/553/events","html_url":"https://github.com/SubPointSolutions/spmeta2/issues/553","id":88344200,"number":553,"title":"Implement 'incremental' provision support","user":{"login":"SubPointSupport","id":8142869,"avatar_url":"https://avatars1.githubusercontent.com/u/8142869?v=3","gravatar_id":"","url":"https://api.github.com/users/SubPointSupport","html_url":"https://github.com/SubPointSupport","followers_url":"https://api.github.com/users/SubPointSupport/followers","following_url":"https://api.github.com/users/SubPointSupport/following{/other_user}","gists_url":"https://api.github.com/users/SubPointSupport/gists{/gist_id}","starred_url":"https://api.github.com/users/SubPointSupport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SubPointSupport/subscriptions","organizations_url":"https://api.github.com/users/SubPointSupport/orgs","repos_url":"https://api.github.com/users/SubPointSupport/repos","events_url":"https://api.github.com/users/SubPointSupport/events{/privacy}","received_events_url":"https://api.github.com/users/SubPointSupport/received_events","type":"User","site_admin":false},"labels":[{"id":103243733,"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/labels/community-feedback","name":"community-feedback","color":"fbca04","default":false},{"id":132352960,"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/labels/m2-research","name":"m2-research","color":"eb6420","default":false},{"id":255821895,"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/labels/plugin","name":"plugin","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":{"login":"SubPointSupport","id":8142869,"avatar_url":"https://avatars1.githubusercontent.com/u/8142869?v=3","gravatar_id":"","url":"https://api.github.com/users/SubPointSupport","html_url":"https://github.com/SubPointSupport","followers_url":"https://api.github.com/users/SubPointSupport/followers","following_url":"https://api.github.com/users/SubPointSupport/following{/other_user}","gists_url":"https://api.github.com/users/SubPointSupport/gists{/gist_id}","starred_url":"https://api.github.com/users/SubPointSupport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SubPointSupport/subscriptions","organizations_url":"https://api.github.com/users/SubPointSupport/orgs","repos_url":"https://api.github.com/users/SubPointSupport/repos","events_url":"https://api.github.com/users/SubPointSupport/events{/privacy}","received_events_url":"https://api.github.com/users/SubPointSupport/received_events","type":"User","site_admin":false},"assignees":[{"login":"SubPointSupport","id":8142869,"avatar_url":"https://avatars1.githubusercontent.com/u/8142869?v=3","gravatar_id":"","url":"https://api.github.com/users/SubPointSupport","html_url":"https://github.com/SubPointSupport","followers_url":"https://api.github.com/users/SubPointSupport/followers","following_url":"https://api.github.com/users/SubPointSupport/following{/other_user}","gists_url":"https://api.github.com/users/SubPointSupport/gists{/gist_id}","starred_url":"https://api.github.com/users/SubPointSupport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SubPointSupport/subscriptions","organizations_url":"https://api.github.com/users/SubPointSupport/orgs","repos_url":"https://api.github.com/users/SubPointSupport/repos","events_url":"https://api.github.com/users/SubPointSupport/events{/privacy}","received_events_url":"https://api.github.com/users/SubPointSupport/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/milestones/81","html_url":"https://github.com/SubPointSolutions/spmeta2/milestone/81","labels_url":"https://api.github.com/repos/SubPointSolutions/spmeta2/milestones/81/labels","id":1309573,"number":81,"title":"SPMeta2 1.3.0-alpha","description":"","creator":{"login":"SubPointSupport","id":8142869,"avatar_url":"https://avatars1.githubusercontent.com/u/8142869?v=3","gravatar_id":"","url":"https://api.github.com/users/SubPointSupport","html_url":"https://github.com/SubPointSupport","followers_url":"https://api.github.com/users/SubPointSupport/followers","following_url":"https://api.github.com/users/SubPointSupport/following{/other_user}","gists_url":"https://api.github.com/users/SubPointSupport/gists{/gist_id}","starred_url":"https://api.github.com/users/SubPointSupport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SubPointSupport/subscriptions","organizations_url":"https://api.github.com/users/SubPointSupport/orgs","repos_url":"https://api.github.com/users/SubPointSupport/repos","events_url":"https://api.github.com/users/SubPointSupport/events{/privacy}","received_events_url":"https://api.github.com/users/SubPointSupport/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":0,"state":"open","created_at":"2015-09-17T01:53:22Z","updated_at":"2017-01-31T08:40:15Z","due_on":"2017-12-31T08:00:00Z","closed_at":null},"comments":9,"created_at":"2015-06-15T07:21:06Z","updated_at":"2017-03-03T01:13:28Z","closed_at":null,"body":"### Work in progress\r\n* [x] Implement DefaultIncrementalModelTreeTraverseService\r\n* [x] Regression tests: basic API\r\n* [x] Regression tests: all definitions + double deployment\r\n* [x] Regression tests: all definitions + double deployment and random updates\r\n* [x] Implement special provision services (both CSOM/SSOM) \r\n* [x] Implement fluent API to make existing provision services working with incremental updates\r\n* [x] Implement fluent API totoggle provision services mode - from defalt to incremental\r\n* [x] Regression tests for newly created provision services and fluent API\r\n* [x] Implement IncrementalModelPrettyPrintService (to output model state)\r\n* [x] model hash classes / api\r\n* [ ] Persistence storage base service\r\n* [ ] Persistence storage 'file system' implementation, tests\r\n* [ ] Persistence storage CSOM/SSOM implementation, tests\r\n\r\n### Problem\r\nSPMeta2 might take unreasonable amount of time on large deployments. For instance, thousands of definitions and/or terabytes of content in the farm cause provision time to be unreasonable huge.\r\n\r\n### Solution \r\nSPMeta2 needs to offer a concept of \"incremental\" or \"conditional\" provision. In nutshell, SPMeta2 should be able to calculate changes and make a derision on \"should/should not\" provision the current artifact.\r\n\r\nBenefits of incremental provision should enable robust, fast provision on extremely large SharePoint deployments - terabytes of data and thousands of artifacts.\r\n\r\nPotential implementation should work smoothly with already existing code, legacy code and current deployments. We should not spend tons of time adding some code or troubleshooting stuff.\r\n\r\n### Implementation\r\nTwo potential implementation to calculate differences are seen:\r\n* Model based - difference between current deployed model and previously deployed model\r\n* SharePoint based - difference between current deployed model  and actual SharePoint objects\r\n\r\nTo simplify things, the following terminology is used:\r\n\r\n* \"model hash\" - represents model hash (model + model nodes + definitions)\r\n* \"node hash\"  - represents model node hash (node + definition)\r\n* \"definition hash\"  - represents model node hash (node + definition)\r\n\r\nWith two strategies to calculate the difference, the easiest way would be to implement diff calculation between models. Calculations between model and SharePoint object is complex and require a huge amount of code to be written for both CSOM, SSOM and O365 APIs.\r\n\r\nFirst draft of the \"model based\" diff calculation should work as following:\r\n* calculate model hash of the current model\r\n* fetch model hash of previously deployed model (via extensible provider: from local folder, sharepoint or other storage)\r\n* calculate the diff between models and mark nodes as \"should provision\" or \"should not provision\"\r\n* provision model\r\n* provide hash of both current and previous model\r\n\r\nA set of additional services are to be provided:\r\n* a service to calculate model hashes\r\n* a service to persist model hash (ootb impl for folders, SharePoint CSOM, SharePoint SSOM)\r\n* a service to output \"final model\" with marked nodes [will / will not be provisioned]\r\n\r\nSuggested implementation has to be implemented for both CSOM/SSOM APIs, covered by unit tests and regression tests, offer traceability to understand which nodes are/aren't provisioned and why.\r\n\r\nBenefits of incremental provision should enable robust, fast provision on extremely large SharePoint deployments - terabytes of data and thousands of artifacts."},"comment":{"url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/comments/283835176","html_url":"https://github.com/SubPointSolutions/spmeta2/issues/553#issuecomment-283835176","issue_url":"https://api.github.com/repos/SubPointSolutions/spmeta2/issues/553","id":283835176,"user":{"login":"SubPointSupport","id":8142869,"avatar_url":"https://avatars1.githubusercontent.com/u/8142869?v=3","gravatar_id":"","url":"https://api.github.com/users/SubPointSupport","html_url":"https://github.com/SubPointSupport","followers_url":"https://api.github.com/users/SubPointSupport/followers","following_url":"https://api.github.com/users/SubPointSupport/following{/other_user}","gists_url":"https://api.github.com/users/SubPointSupport/gists{/gist_id}","starred_url":"https://api.github.com/users/SubPointSupport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SubPointSupport/subscriptions","organizations_url":"https://api.github.com/users/SubPointSupport/orgs","repos_url":"https://api.github.com/users/SubPointSupport/repos","events_url":"https://api.github.com/users/SubPointSupport/events{/privacy}","received_events_url":"https://api.github.com/users/SubPointSupport/received_events","type":"User","site_admin":false},"created_at":"2017-03-03T01:13:28Z","updated_at":"2017-03-03T01:13:28Z","body":"Okay, first beta implementation looks as following:\r\n```cs\r\n// create you service as usual\r\nvar provisionService = ...;\r\n\r\n// set provision model via extension method\r\nprovisionService.SetIncrementalProvisionMode();\r\n// set previous model hash if you've got it (see below on how to obtain it)\r\nprovisionService.SetIncrementalProvisionModelHash(previousModelHash);\r\n\r\n// deploy model as usual\r\nprovisionService.DeployModel()\r\n\r\n// this is the model hash to be saved somewhere, it stores information on provisioned model\r\nvar modelHash = provisionService.GetIncrementalProvisionModelHash();\r\n\r\n// optionally, switch back to default provision\r\nprovisionService.SetDefaultProvisionMode();\r\n```\r\n\r\nOnce model is deployed, we get two things such as model hash and some userful trace:\r\n```cs\r\n// print model state\r\nvar tracer = new DefaultIncrementalModelPrettyPrintService();\r\n\r\nConsole.WriteLine(string.Format(\"Deployed model with incremental updates:\"));\r\nConsole.WriteLine(Environment.NewLine + tracer.PrintModel(model));\r\n```\r\n\r\nOutcome with first provision:\r\n```\r\n[-] [WebDefinition] - Title:[] Description:[] Convert(p.LCID):[1033]\r\n[+]     [ListDefinition] - Title: [662fecf933624f81934bef9a18e6343b] \r\n[+]         [FolderDefinition] - Name:[ce56514be250472fa8d44c9aede36fda]\r\n[+]         [FolderDefinition] - Name:[c416cebc8f8b4f2abe8491102d0108ee]\r\n[+]         [ListItemDefinition] - Title:[04c00dbf66ea4d2fbb23d32fae87d07f]\r\n```\r\n\r\nOutcome with second and further provision:\r\n```\r\n[-] [WebDefinition] - Title:[] Description:[] Convert(p.LCID):[1033]\r\n[-]     [ListDefinition] - Title: [662fecf933624f81934bef9a18e6343b] \r\n[-]         [FolderDefinition] - Name:[ce56514be250472fa8d44c9aede36fda]\r\n[-]         [FolderDefinition] - Name:[c416cebc8f8b4f2abe8491102d0108ee]\r\n[-]         [ListItemDefinition] - Title:[04c00dbf66ea4d2fbb23d32fae87d07f]\r\n```\r\n\r\nPlus indicates provision and minus - skipping actual provision of the artifact. \r\n\r\nMost of the ground work has been done, we already re-engineered out regression testing to support both normal and incremental provision. Working hard on two things right now:\r\n* More regression testing, adding tests on randomized changes and incremental provision checks\r\n* Implementing default persistence storage for model hash to work with local file system AND SharePoint\r\n\r\nIdeally, you wouldn't have to work with model hash directly as it would always be automatically persist into SharePoint. You would have to call .SetIncrementalProvisionMode() and then also set \"unique ID\"  of the model so that we would be able to identify the model within many model hashes.\r\n\r\nLooking forward to your ideas and feedback on the current progress. We are moving really fast with that feature aiming to push it as beta release early next week."}}
{"actor":{"display_login":"SubPointSupport"}}