{"action":"created","issue":{"url":"https://api.github.com/repos/conda/conda/issues/4253","repository_url":"https://api.github.com/repos/conda/conda","labels_url":"https://api.github.com/repos/conda/conda/issues/4253/labels{/name}","comments_url":"https://api.github.com/repos/conda/conda/issues/4253/comments","events_url":"https://api.github.com/repos/conda/conda/issues/4253/events","html_url":"https://github.com/conda/conda/issues/4253","id":199808158,"number":4253,"title":"Conda does not fall back to using soft links","user":{"login":"nehaljwani","id":1779189,"avatar_url":"https://avatars3.githubusercontent.com/u/1779189?v=3","gravatar_id":"","url":"https://api.github.com/users/nehaljwani","html_url":"https://github.com/nehaljwani","followers_url":"https://api.github.com/users/nehaljwani/followers","following_url":"https://api.github.com/users/nehaljwani/following{/other_user}","gists_url":"https://api.github.com/users/nehaljwani/gists{/gist_id}","starred_url":"https://api.github.com/users/nehaljwani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nehaljwani/subscriptions","organizations_url":"https://api.github.com/users/nehaljwani/orgs","repos_url":"https://api.github.com/users/nehaljwani/repos","events_url":"https://api.github.com/users/nehaljwani/events{/privacy}","received_events_url":"https://api.github.com/users/nehaljwani/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-01-10T12:26:30Z","updated_at":"2017-05-05T04:59:51Z","closed_at":null,"body":"Let me describe a scenario. \r\n\r\nThere is a company, called `marvelcomics` and inside that, is a team called `avengers`.  Two members of this team are:\r\n\r\n```bash\r\n[shield@earth ~]$ id thor\r\nuid=1004(thor) gid=1002(avengers) groups=1002(avengers)\r\n[shield@earth ~]$ id loki\r\nuid=1003(loki) gid=1002(avengers) groups=1002(avengers)\r\n```\r\n\r\nNow, there is a playground called `asgard` where these two guys can play:\r\n```bash\r\n[shield@earth ~]$ ls -ld /asgard\r\ndrwxrwxrwt. 4 root root 4096 Jan 10 11:24 /asgard/\r\n[shield@earth ~]$ getfacl /asgard\r\ngetfacl: Removing leading '/' from absolute path names\r\n# file: asgard/\r\n# owner: root\r\n# group: root\r\n# flags: --t\r\nuser::rwx\r\ngroup::rwx\r\nother::rwx\r\n[shield@earth ~]$ mount | grep /asgard\r\n/dev/sdb1 on /asgard type ext4 (rw,relatime,seclabel,data=ordered)\r\n```\r\n\r\nThe user `thor` creates a conda environment on `earth`, inside the playground using conda and installs his favourite tool: `curl`.\r\n```bash\r\n[thor@earth ~]$ wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh\r\n[thor@earth ~]$ chmod +x Miniconda2-latest-Linux-x86_64.sh \r\n[thor@earth ~]$ ./Miniconda2-latest-Linux-x86_64.sh -b -p /asgard/kingdom\r\n[thor@earth ~]$ source /asgard/kingdom/bin/activate\r\n(root) [thor@earth ~]$ conda install curl\r\n```\r\n\r\nThe user `loki` tries to steal this kingdom:\r\n```bash\r\n[loki@earth ~]$ source /asgard/kingdom/bin/activate \r\n(root) [loki@earth ~]$ conda config --add envs_dirs /asgard/loki\r\n(root) [loki@earth ~]$ conda create -n stealth curl\r\nFetching package metadata .......\r\nSolving package specifications: ..........\r\n\r\nPackage plan for installation in environment /asgard/loki/stealth:\r\n\r\nThe following NEW packages will be INSTALLED:\r\n\r\n    curl:    7.49.0-1\r\n    openssl: 1.0.2j-0 (soft-link)\r\n    zlib:    1.2.8-3  (soft-link)\r\n\r\nProceed ([y]/n)? y\r\n\r\nLinking packages ...\r\nWARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]\r\nWARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]\r\nWARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]\r\n\r\n\r\nCondaOSError: OS error: failed to link (src=u'/asgard/kingdom/pkgs/curl-7.49.0-1/bin/curl', dst=u'/asgard/loki/stealth/bin/curl', type=1, error=OSError(1, 'Operation not permitted'))\r\n``` \r\nWhy did `loki` think that he could steal that easily? Because, he had read in conda's docs:\r\n\r\n> When allow_softlinks is True, conda uses hard-links when possible, and soft-links (symlinks) when hard-links are not possible, such as when installing on a different filesystem than the one that the package cache is on.\r\n\r\n... and he had double checked:\r\n\r\n```bash\r\n(root) [loki@earth ~]$ conda config --show | grep allow_softlinks\r\nallow_softlinks: True\r\n```\r\n\r\nThat's weird. Let's see if `shield` has added any protection mechanisms:\r\n```bash\r\n[shield@earth ~]$ sudo cat /proc/sys/fs/protected_hardlinks \r\n1\r\n```\r\n\r\nOf course there is one, so, unless `loki` has ownership of a file or at least `read+write` permissions on it, he can't steal it. Let's check:\r\n```bash\r\n[shield@earth ~]$ ls -l /asgard/kingdom/pkgs/curl-7.49.0-1/info/index.json \r\n-rw-rw-r--. 1 thor avengers 274 Jul 21 18:56 /shared/thor/pkgs/curl-7.49.0-1/info/index.json\r\n```\r\n\r\nWell, `loki` is an avenger, so he has `read+write` permissions on this file and he indeed *can* create a link to *this* file. And conda is correct till now in assuming that he can... However, the reason why he fails to steal:\r\n\r\n```bash\r\n[shield@earth ~]$ ls -l /asgard/kingdom/pkgs/curl-7.49.0-1/bin/curl\r\n-rwxr-xr-x. 2 thor avengers 182238 Jul 21 18:56 /asgard/kingdom/pkgs/curl-7.49.0-1/bin/curl\r\n```\r\n\r\nThere is no `write` permission on this `inode` for `loki`. And hence, he cannot steal blindly.\r\n\r\nThis scenario makes me raise some queries:\r\n\r\n1.  Why does conda check only `${CONDA_PREFIX}/info/index.json` for determining whether or not it can use `LINK_HARD` for the entire package? In other words, how/why is this file the source of truth?\r\n2.  Is there an option in conda using which I can specify that I want to prefer soft-links over hard-links, while creating new environments (using `conda create -n` or `conda env create ...`),  regardless of whether creating hard-links is possible? Why/Why not?"},"comment":{"url":"https://api.github.com/repos/conda/conda/issues/comments/299375618","html_url":"https://github.com/conda/conda/issues/4253#issuecomment-299375618","issue_url":"https://api.github.com/repos/conda/conda/issues/4253","id":299375618,"user":{"login":"kalefranz","id":1418419,"avatar_url":"https://avatars3.githubusercontent.com/u/1418419?v=3","gravatar_id":"","url":"https://api.github.com/users/kalefranz","html_url":"https://github.com/kalefranz","followers_url":"https://api.github.com/users/kalefranz/followers","following_url":"https://api.github.com/users/kalefranz/following{/other_user}","gists_url":"https://api.github.com/users/kalefranz/gists{/gist_id}","starred_url":"https://api.github.com/users/kalefranz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kalefranz/subscriptions","organizations_url":"https://api.github.com/users/kalefranz/orgs","repos_url":"https://api.github.com/users/kalefranz/repos","events_url":"https://api.github.com/users/kalefranz/events{/privacy}","received_events_url":"https://api.github.com/users/kalefranz/received_events","type":"User","site_admin":false},"created_at":"2017-05-05T04:59:51Z","updated_at":"2017-05-05T04:59:51Z","body":"> Why does conda check only `${CONDA_PREFIX}/info/index.json` for determining whether or not it can use `LINK_HARD` for the entire package? In other words, how/why is this file the source of truth?\r\n\r\nI've started making fairly extensive use of \"magic files\" (see https://github.com/conda/conda/blob/4.4.x/conda/base/constants.py#L108-L110).  (Hmmm even the name sort of goes a long with the theme here.)  `info/index.json` should be added to the list of constants.\r\n\r\nThe value of this pattern really solidified somewhere around conda 4.3.13, and it's really proven itself since.  We can make assumptions that these files are indicative of the state of the \"unit\" they represent, and we will be right the VERY VAST majority of the time.  It's only under extremely exceptional circumstances that we'll be wrong (like if a user `chmod`s a single random file for some reason somewhere), and we can then balk without too much consequence if/when that ever happens (due in large part to how rock solid the unlink-link transaction has been).\r\n\r\nFor the three magic files I linked to above, the test we're throwing at them is \"are you--and the unit you represent--writable or read-only?\"  Before this magic file change, conda would test permissions by just seeing if a file of a random name could be created (and then removed) in a directory.  Obviously problematic.\r\n\r\nSo is it really ok to use a single file to represent the state of a whole unit of files?  For the three examples I've been talking about so far, I think so.  If you think through how we'd handle the situation if we tested every single file in the unit of files, and found one to be abnormal compared to the others, I don't think the end result would be much different than what we're already doing now.  It's just that the path from point A to point B--the in between--might be a little different.\r\n\r\nNow, to the `info/index.json` file specifically... You use `${CONDA_PREFIX}/info/index.json`, which is actually incorrect.  `info` directories from conda packages are left out of what is linked into a conda prefix.  More accurate is `<EXTRACTED_CONDA_PACKAGE_DIRECTORY>/info/index.json`.  We use `info/index.json` because that is absolutely guaranteed to be there for every conda package.  If a tarball does not have an `info/index.json` file, then it is not a conda package.  Easy test.  Here again for the hard-link test, I think it's safe to use this one path as a proxy for the state of the entire package.  If you think of the general flow, we download tarballs to a cache directory, and then extract them there too.  They're extracted in one go as a single unit.  Under any normal circumstance it doesn't make sense that, for example, that the package could be extracted on more than one mount point.  Similarly, on the other end, we're making the assumption that a conda prefix is all contained within a single mount point (this is probably the more tenuous assumption).\r\n\r\nWhat happens if we're wrong and the file can't be hard-linked?  First, we use a procedure to figure out what the *requested* link type is when linking a package into a prefix.  https://github.com/conda/conda/blob/4.3.17/conda/core/link.py#L48-L58  That function should really be named `determine_requested_link_type()`; the result is called [`requested_link_type`](https://github.com/conda/conda/blob/4.3.17/conda/core/link.py#L401) in many other places in the code, and that's more accurate.  It's a requested link type because it's not honored in all situations.  There a few places of logic [here](https://github.com/conda/conda/blob/4.3.17/conda/core/path_actions.py#L156-L195) and [here](https://github.com/conda/conda/blob/4.3.17/conda/gateways/disk/create.py#L217-L254) where we'll override the request saying no, you can't have that.\r\n\r\nAre there edge cases?  Probably still a few.  Much of this strategy conda has been using well before I started working on the code base.  We haven't yet encountered an edge case that presents a fatal flaw in the design.\r\n\r\n> Is there an option in conda using which I can specify that I want to prefer soft-links over hard-links, while creating new environments (using conda create -n or conda env create ...), regardless of whether creating hard-links is possible? Why/Why not?\r\n\r\nThe short logic in [`determine_link_type()`](https://github.com/conda/conda/blob/4.3.17/conda/core/link.py#L48-L58) shows the available configuration parameters and how they're handled.  To your question directly, the answer is 'yes', and the configuration parameter is `always_softlink: yes` in a condarc file and `CONDA_ALWAYS_SOFTLINK=yes` as an environment variable.  In this case, the value `always_copy` has to be `False` (which is the default) for this to work.  These are boolean switches rather than a single enum parameter because, well, probably for the same reason conda 'features' are right now binary on-off rather than enums.  No further comments on design philosophy limitations there.  The `always_softlink` option was actually added somewhat recently.  The 'why' discussion is at https://github.com/conda/conda/issues/3308."}}
{"actor":{"display_login":"kalefranz"}}