{"action":"created","issue":{"url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/44","repository_url":"https://api.github.com/repos/jiscdev/lrw-udd-api","labels_url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/44/comments","events_url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/44/events","html_url":"https://github.com/jiscdev/lrw-udd-api/issues/44","id":214995449,"number":44,"title":"1.3.0 testbed","user":{"login":"ht2","id":1352590,"avatar_url":"https://avatars1.githubusercontent.com/u/1352590?v=4","gravatar_id":"","url":"https://api.github.com/users/ht2","html_url":"https://github.com/ht2","followers_url":"https://api.github.com/users/ht2/followers","following_url":"https://api.github.com/users/ht2/following{/other_user}","gists_url":"https://api.github.com/users/ht2/gists{/gist_id}","starred_url":"https://api.github.com/users/ht2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ht2/subscriptions","organizations_url":"https://api.github.com/users/ht2/orgs","repos_url":"https://api.github.com/users/ht2/repos","events_url":"https://api.github.com/users/ht2/events{/privacy}","received_events_url":"https://api.github.com/users/ht2/received_events","type":"User","site_admin":false},"labels":[{"id":563018445,"url":"https://api.github.com/repos/jiscdev/lrw-udd-api/labels/v1.3.0","name":"v1.3.0","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-03-17T12:56:02Z","updated_at":"2017-10-05T10:32:47Z","closed_at":null,"author_association":"COLLABORATOR","body":"Hi all (@robwynj, @michaelwebbjisc, @huwrobertsjisc, @leebaylis, @amason, @pspaude, @tomglanville)\r\n\r\nThis was originally going to be an email, but @andrewhickey cleverly suggested I post it here instead! Apologies in advance for the wall of text. Grab a cup of tea, and I'll try not to send everyone to sleep...\r\n\r\n### UDD v1.3.0 dev/testing server\r\n#### The new application\r\n\r\nSo we've been pretty busy here beaving over a new application to handle the new 1.3 UDD data format in a relational database (RDBS), whilst still tying it to the authentication and scoping system we already have established for the LRW.\r\n\r\nHappy to say that we now have a working dev environment that we'd like you to give a thrashing! Currently this is an entirely isolated system from the LRW, so any credentials I give you are actually linked to a new organisation I have created for you in https://devv2.learninglocker.net, which is running a canary build of V2 that adds in some authentication aspects we require for the more modular system we're moving towards. \r\n\r\nThere shouldn't be a need to share any logins for that with you as we aren't testing any xAPI components here and I will provide you with Basic credentials you can just start testing with below.\r\n\r\n#### Application stack/Deployment\r\n\r\nThe UDD application is now a separate Node application that securely requests scope information from the main LRW. This allows you to create credentials and set scopes in the existing V2 UI exactly as you do now. It also means that any existing credentials (and their scoping) will \"just work\" when we come to push this live.\r\n\r\nCurrently we have the application running on an EC2 instance (eu-west1) and it is utilising an encrypted Aurora replica cluster, again in eu-west1. This is very likely to mirror the eventual production deployment that I would suggest we move with (albeit in a auto-scaling-group), but that is something I can look into further down the line.\r\n\r\n#### Implementation\r\n\r\nMoving to the UDD implementation itself, the API matches the 1.3.0 specification currently in draft, the only omission being the extensions that are still being built (but we don't anticipate that taking too much longer.\r\n\r\n@ryansmith94 has worked incredibly hard to make the format in which you submit, query and remove data is almost exactly the same as the existing API. The selects and populates in the query parameters should broadly work in the same manor. The soft delete system is exactly as it was, requiring you to pass extra query string parameters to retrieve \"trashed\" models and \"force\" delete from the database entirely. For a refresher on how that all works, refer to the current UDD API docs @ https://jiscv2.learninglocker.net/api/jisc/stable\r\n\r\nWhat is the immediately obvious difference is the primary ID used to refer to a model. The database/Mongo specific \"_id\" has been dropped in favour of each models' defined primary key. Read more here: https://github.com/jiscdev/analytics-udd/tree/v1.3.0#primary-keys\r\n\r\nWhat this allows for is direct reference to a model by that key. So for example to GET course with COURSE_ID of \"COURSE_1\"...\r\n\r\nGET /models/course/COURSE_1\r\n\r\nThis key is permitted to be set by the client on POST, and is unique inside that client's organisation. If the primary key is not sent in the POST data of a new model, then the LRW will look to generate a unique key for the model. We are using UUID v4 to do this: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_.28random.29. This new value would be returned in the full model in the POST response.\r\n\r\nOne new area we have added, is the ability to clear down an entity en masse. That isn't quite live yet (its in testing) but I am actually looking to create a new scope on the LRW which explicitly allows a client to perform that action (@robwynj, I know you would like this!!)\r\n\r\n#### Performance and load testing\r\n\r\nSo, what would be absolutely great would be for some heavy testing of the new API using some of your scripts where we saw limitations previously. \r\n\r\nWe've actually done our own load testing (via loader.io) on some simple course<->institution relations with some excellent results. I setup a test where up to 50 clients each simultaneously pushed in 1000 course models with a relation to a Institution for a full minute (with the UDD performing ID generation per model). Over the full minute we saw 735,000 entries successfully enter the database. You can view the results for yourself @ https://loader.io/reports/17c95ee847e34b75205deb0e679f73dd/results/28d4ba24fb777c666398b328f534c183\r\n\r\n I then selected the institution and populated the full set of courses via /models/institution/INST1?populate=courses&select={\"courses.COURSE_ID\":true}. The UDD was able to return the institution with all 735k courses populated in ~13000ms. \r\n\r\nThat is obviously a very extreme example (and hopefully one that would not normally be repeated!). What it does show though, is that client load on POSTing is no longer an issue with the UDD application. \r\n\r\nWhat I did see is that requesting that amount of data directly from the database (via the mysql command line tool) takes a very similar time. At this point any limitations we have are entirely around the database instance we are running, which can be easily provisioned up or down as required.\r\n\r\n#### Test credentials\r\n\r\nSo, I've waffled quite a lot - here are some actual connection details so you can look to try it out for yourselves:\r\n\r\nHost: **34.251.87.128/models/ENTITY**\r\nAuthorization: **Basic MDAzZWMyMmQ3YmVlY2M2NDY0YTMyYzM0OTc1NTgyNTg5ZTliYWY2MDoxZTk0NTE2OGIwZjYwZjI0ZTk2MThiMjIzYzc1NmY5ODUwMzNmOGZh**\r\n\r\nThese credentials have the ALL scope and \"ENTITY \" is using exactly what exists in the current API:\r\n\r\n  assessmentInstance\r\n  course\r\n  courseInstance\r\n  institution\r\n  module\r\n  moduleInstance\r\n  moduleVleMap\r\n  period\r\n  staff\r\n  staffCourseInstance\r\n  staffModuleInstance\r\n  student\r\n  studentAssessmentInstance\r\n  studentCourseInstance\r\n  studentCourseMembership\r\n  studentModuleInstance\r\n\r\nIf you require additional credentials with different scopes then let us know and we can make them for you.\r\n\r\n#### Looking forward\r\n\r\nOne issue we will need to think about moving forward is how we will look to transition between version of the API. No longer will we be able to offer `1.3.0` data out of another version _if there are breaking changes between versions_! For example, if a field is renamed or removed, then we would need to be looking to create an entirely new database and migrate the existing data set. \r\n\r\nThis probably needs more thought and I'll look to create another issue so we can discuss how we need can implement this.\r\n\r\n#### Wrapping up\r\n\r\nSo - lots to digest! We are still working on some areas (entity clear down, timestamps, X-Count header, the extensions entity). I would think we'll have the first 3 wrapped up today, with the extensions swiftly following.\r\n\r\nPlease let us know when you find issues/problems (simply make an issue in here, tag it up with `v1.3.0` label, and assign @ryansmith94, @andrewhickey or myself. \r\n\r\nLooking forward to seeing your results,\r\nJames\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/comments/334426352","html_url":"https://github.com/jiscdev/lrw-udd-api/issues/44#issuecomment-334426352","issue_url":"https://api.github.com/repos/jiscdev/lrw-udd-api/issues/44","id":334426352,"user":{"login":"ryansmith94","id":3045513,"avatar_url":"https://avatars0.githubusercontent.com/u/3045513?v=4","gravatar_id":"","url":"https://api.github.com/users/ryansmith94","html_url":"https://github.com/ryansmith94","followers_url":"https://api.github.com/users/ryansmith94/followers","following_url":"https://api.github.com/users/ryansmith94/following{/other_user}","gists_url":"https://api.github.com/users/ryansmith94/gists{/gist_id}","starred_url":"https://api.github.com/users/ryansmith94/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryansmith94/subscriptions","organizations_url":"https://api.github.com/users/ryansmith94/orgs","repos_url":"https://api.github.com/users/ryansmith94/repos","events_url":"https://api.github.com/users/ryansmith94/events{/privacy}","received_events_url":"https://api.github.com/users/ryansmith94/received_events","type":"User","site_admin":false},"created_at":"2017-10-05T10:32:47Z","updated_at":"2017-10-05T10:32:47Z","author_association":"COLLABORATOR","body":"Hi @leebaylis, sorry that you've ran into some issues, I'll address each of them below.\r\n\r\n### Selecting using numbers\r\nCurrently trying to fix this in the [issue/selectWithNumbers branch](https://github.com/ht2/udd-api/tree/issue/selectWithNumbers).\r\n\r\n### MySQL limit error\r\nThis is caused by our attempts to use a limit inside a sub query for the population. I'm trying to fix this in the [issue/innerLimit branch](https://github.com/ht2/udd-api/tree/issue/innerLimit) (not pushed to Github at the time of this message).\r\n\r\n### Selecting inside populates\r\nI'm not sure if what you're trying to do worked prior to UDD 1.3, but it certainly won't work now. You can instead add your selection to the select parameter like so:\r\n```\r\nhttps://jiscudd.learninglocker.net/models/studentassessmentinstance?populate={\"path\": \"student\"}&query={\"GRADE_DATE\": {\"$gt\": \"2017-09-29\"}}&select={\"GRADE_DATE\": true, \"student.FIRST_NAME\": true}\r\n```"}}
{"actor":{"display_login":"ryansmith94"}}