{"action":"created","issue":{"url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/744","repository_url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge","labels_url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/744/labels{/name}","comments_url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/744/comments","events_url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/744/events","html_url":"https://github.com/Sage-Bionetworks/DigitalMammographyChallenge/issues/744","id":216081856,"number":744,"title":"Maintenance before Validation Phase","user":{"login":"tschaffter","id":3056480,"avatar_url":"https://avatars3.githubusercontent.com/u/3056480?v=3","gravatar_id":"","url":"https://api.github.com/users/tschaffter","html_url":"https://github.com/tschaffter","followers_url":"https://api.github.com/users/tschaffter/followers","following_url":"https://api.github.com/users/tschaffter/following{/other_user}","gists_url":"https://api.github.com/users/tschaffter/gists{/gist_id}","starred_url":"https://api.github.com/users/tschaffter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschaffter/subscriptions","organizations_url":"https://api.github.com/users/tschaffter/orgs","repos_url":"https://api.github.com/users/tschaffter/repos","events_url":"https://api.github.com/users/tschaffter/events{/privacy}","received_events_url":"https://api.github.com/users/tschaffter/received_events","type":"User","site_admin":false},"labels":[{"id":486102154,"url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/labels/Major","name":"Major","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"tschaffter","id":3056480,"avatar_url":"https://avatars3.githubusercontent.com/u/3056480?v=3","gravatar_id":"","url":"https://api.github.com/users/tschaffter","html_url":"https://github.com/tschaffter","followers_url":"https://api.github.com/users/tschaffter/followers","following_url":"https://api.github.com/users/tschaffter/following{/other_user}","gists_url":"https://api.github.com/users/tschaffter/gists{/gist_id}","starred_url":"https://api.github.com/users/tschaffter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschaffter/subscriptions","organizations_url":"https://api.github.com/users/tschaffter/orgs","repos_url":"https://api.github.com/users/tschaffter/repos","events_url":"https://api.github.com/users/tschaffter/events{/privacy}","received_events_url":"https://api.github.com/users/tschaffter/received_events","type":"User","site_admin":false},"assignees":[{"login":"tschaffter","id":3056480,"avatar_url":"https://avatars3.githubusercontent.com/u/3056480?v=3","gravatar_id":"","url":"https://api.github.com/users/tschaffter","html_url":"https://github.com/tschaffter","followers_url":"https://api.github.com/users/tschaffter/followers","following_url":"https://api.github.com/users/tschaffter/following{/other_user}","gists_url":"https://api.github.com/users/tschaffter/gists{/gist_id}","starred_url":"https://api.github.com/users/tschaffter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschaffter/subscriptions","organizations_url":"https://api.github.com/users/tschaffter/orgs","repos_url":"https://api.github.com/users/tschaffter/repos","events_url":"https://api.github.com/users/tschaffter/events{/privacy}","received_events_url":"https://api.github.com/users/tschaffter/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2017-03-22T14:26:31Z","updated_at":"2017-03-27T20:59:12Z","closed_at":null,"body":"## Upgrade Leaderboard training machines for Validation\r\n### Protocol\r\n- Check whether there are Docker submissions running. If yes, postpone the update.\r\n- Update firmwares using SoftLayer web interface (it is recommended to shutdown the machine first).\r\n- `yum update -y`\r\n- Keep only 4 user data spaces\r\n  - `umount /data/dataset4 && lvremove /dev/mapper/data-dataset4`\r\n  - Remove the corresponding line from `/etc/fstab`\r\n- Resize partitions\r\n  - `lvresize --resizefs --size 10T /dev/mapper/data-data`\r\n  - `for i in {0..3}; do lvresize --resizefs --size 12T /dev/mapper/data-dataset$i; done`\r\n  - `for i in {0..1}; do lvresize --resizefs --size 300G /dev/mapper/data-scratch$i; done`\r\n- Transfer the Validation Training Set, see instructions https://github.com/Sage-Bionetworks/DigitalMammographyChallenge/wiki/Install-Challenge-Dataset.\r\n- Reinstall Docker from dmclient (for update)\r\n  - `./DigitalMammographyChallenge/setupNewServer.sh softlayer bmXX`\r\n- If machine has been given access to another machines for the transfer of the dataset, remove the access (ssh).\r\n- Reboot and performs required checks (e.g. `nvidia-smi`)\r\n- On Synapse, delete the reference to the content of the `/data/datasetX`\r\n- Make sure that the agents manage 4 /data/datasetXX and not longer 5\r\n\r\n### Kernel can not be installed because not enough space\r\nRun the following command to keep only the two most recent kernels.\r\n```\r\n# yum install -y yum-utils && package-cleanup -y --oldkernels --count=2\r\n```\r\n\r\nSee https://www.cyberciti.biz/faq/installing-kernel-2-6-32-131-2-1-el6-x86_64-needs-8mb-on-boot-filesystem/\r\n\r\n### Partitions\r\n```\r\n[root@bm04 ~]# df -h\r\nFilesystem                 Size  Used Avail Use% Mounted on\r\n/dev/sda3                   99G  6.9G   92G   7% /\r\ndevtmpfs                   252G     0  252G   0% /dev\r\ntmpfs                      252G     0  252G   0% /dev/shm\r\ntmpfs                      252G  202M  252G   1% /run\r\ntmpfs                      252G     0  252G   0% /sys/fs/cgroup\r\n/dev/sda1                  248M  195M   54M  79% /boot\r\n/dev/mapper/data-dataset1   12T  162G   12T   2% /data/dataset1\r\n/dev/mapper/data-scratch1  296G   64M  283G   1% /data/scratch1\r\n/dev/mapper/data-tempDir1   50G  448M   47G   1% /data/tempDir1\r\n/dev/mapper/data-model0     20G  5.4M   19G   1% /data/model0\r\n/dev/mapper/data-scratch0  296G   64M  283G   1% /data/scratch0\r\n/dev/mapper/data-model1     20G  5.4M   19G   1% /data/model1\r\n/dev/mapper/data-data       10T  9.2T  329G  97% /data/data\r\n/dev/mapper/data-dataset2   12T  299G   12T   3% /data/dataset2\r\n/dev/mapper/data-dataset3   12T   57M   12T   1% /data/dataset3\r\n/dev/mapper/data-dataset0   12T  948G   11T   9% /data/dataset0\r\n/dev/mapper/data-tempDir0   50G  2.0G   45G   5% /data/tempDir0\r\ntmpfs                       51G     0   51G   0% /run/user/0\r\n```\r\n\r\n### Status\r\n- [x] bm01 (kept as Leaderboard Training machine)\r\n- [x] bm02\r\n- [x] bm03\r\n- [x] bm04\r\n- [x] bm05 (admin machine, update last)\r\n- [x] bm06 (Validation Dry Run)\r\n- [x] bm07\r\n- [ ] bm08 (transferring dataset from bm07...)\r\n- [x] bm09\r\n- [x] bm10\r\n- [ ] bm11 (transferring dataset from bm03...)\r\n- [ ] bm12 (transferring dataset from bm04...)\r\n- [ ] bm13 (transferring dataset from bm06...)\r\n- [ ] bm14 (transferring dataset from bm09...)\r\n- [x] bm15\r\n- [x] bm16 (Validation Dry Run)\r\n- [x] bm17\r\n- [x] bm18\r\n- [x] bm19\r\n- [x] bm20\r\n- [x] bm21\r\n- [x] bm22\r\n- [x] bm23\r\n\r\n## Upgrade the AWS Express Lane and Inference machines for the Validation Phase\r\n### Protocol\r\n\r\n## References\r\nProtocol for past maintenance: #492 "},"comment":{"url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/comments/289584293","html_url":"https://github.com/Sage-Bionetworks/DigitalMammographyChallenge/issues/744#issuecomment-289584293","issue_url":"https://api.github.com/repos/Sage-Bionetworks/DigitalMammographyChallenge/issues/744","id":289584293,"user":{"login":"tschaffter","id":3056480,"avatar_url":"https://avatars3.githubusercontent.com/u/3056480?v=3","gravatar_id":"","url":"https://api.github.com/users/tschaffter","html_url":"https://github.com/tschaffter","followers_url":"https://api.github.com/users/tschaffter/followers","following_url":"https://api.github.com/users/tschaffter/following{/other_user}","gists_url":"https://api.github.com/users/tschaffter/gists{/gist_id}","starred_url":"https://api.github.com/users/tschaffter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschaffter/subscriptions","organizations_url":"https://api.github.com/users/tschaffter/orgs","repos_url":"https://api.github.com/users/tschaffter/repos","events_url":"https://api.github.com/users/tschaffter/events{/privacy}","received_events_url":"https://api.github.com/users/tschaffter/received_events","type":"User","site_admin":false},"created_at":"2017-03-27T20:59:12Z","updated_at":"2017-03-27T20:59:12Z","body":"All the SoftLayer machines have the same version of Docker.\r\n\r\n```\r\n$ for i in `seq -w 1 23`; do ssh bm$i 'echo \"$(cat /etc/hostname) $(docker --version)\"'; done\r\nbm01 Docker version 17.03.0-ce, build 60ccb22\r\nbm02 Docker version 17.03.0-ce, build 60ccb22\r\nbm03 Docker version 17.03.0-ce, build 60ccb22\r\nbm04 Docker version 17.03.0-ce, build 60ccb22\r\nbm05 Docker version 17.03.0-ce, build 60ccb22\r\nbm06 Docker version 17.03.0-ce, build 60ccb22\r\nbm07 Docker version 17.03.0-ce, build 60ccb22\r\nbm08 Docker version 17.03.0-ce, build 60ccb22\r\nbm09 Docker version 17.03.0-ce, build 60ccb22\r\nbm10 Docker version 17.03.0-ce, build 60ccb22\r\nbm11 Docker version 17.03.0-ce, build 60ccb22\r\nbm12 Docker version 17.03.0-ce, build 60ccb22\r\nbm13 Docker version 17.03.0-ce, build 60ccb22\r\nbm14 Docker version 17.03.0-ce, build 60ccb22\r\nbm15 Docker version 17.03.0-ce, build 60ccb22\r\nbm16 Docker version 17.03.0-ce, build 60ccb22\r\nbm17 Docker version 17.03.0-ce, build 60ccb22\r\nbm18 Docker version 17.03.0-ce, build 60ccb22\r\nbm19 Docker version 17.03.0-ce, build 60ccb22\r\nbm20 Docker version 17.03.0-ce, build 60ccb22\r\nbm21 Docker version 17.03.0-ce, build 60ccb22\r\nbm22 Docker version 17.03.0-ce, build 60ccb22\r\nbm23 Docker version 17.03.0-ce, build 60ccb22\r\n```"}}
{"actor":{"display_login":"tschaffter"}}