I've got a few files which are used in several of my repos:

  • functions.sh, shell library to for example print a colored warning/error message or the documentation of a script file.
  • Makefile; a standardized one which installs the file $(CURDIR)/[dirname].sh to $(PREFIX)/[dirname] and references a test script.
  • tools.mk; Makefile commands to for example print all the variable definitions in the parent Makefile.

These are more or less stable, and some are used in probably over a dozen repos. I've been thinking how to keep this DRY, but none of the options so far seem satisfactory:

  1. Keep doing it like now, creating a copy for each new repo. This keeps the code together with all its dependencies (avoiding bugs when the general solution is not general enough), but changes which are applicable in multiple places have to be applied in each separately.
  2. Add an executable to each of the repositories to download the files needed. This means that developers and end users will have to run an extra command to get all the relevant files, and it breaks the possibility for developers to modify and commit/push the included files.
  3. Use git submodule or equivalent. This at least keeps the repositories connected, but in the Git case it seems like it's restricted to "a dedicated subdirectory", so no top-level files, and no mixing with parent repository files which belong in the same directory. This could be circumvented with symlinks, but that's an ugly workaround for the obvious ideal situation.

The ideal solution should:

  1. Communicate with the correct repository when doing an operation on a file.
  2. Allow includes in the same directories as the parent directory.
  3. A single, simple command should be enough to update the entire repository and all includes, no matter how many or how deeply nested.
  4. Allow includes in the top-level directory.
  5. Not incur significant developer or user constraints (must be online while installing) or extra work (running a "pre-install" command separately from the "install" command).
  6. Allow "cherry-picking" of files to include. Many projects might need a different Makefile, for example, and including one which is not used is just ugly (and would get uglier as more files are added).

Is this sort of thing possible with current software?


In SVN the concept is called "vendor branches", ie you think of your shared files in much the same way as you think of a 3rd party library that you're using. You make a 'read-only' copy of the external files to your repo and use them. but never change them - that only occurs when you re-import the vendor files.

If you do make changes to a shared file in a project, you can (by branching it inside your repo). When you get a new import, you need to merge the changes again.

|improve this answer|||||
  • From what you say and a short Google excursion, it looks like this doesn't support point 1 (submitting changes back to the child repo). Does it support points 5 (no extra operation for the end user when installing) and 6 (cherry-picking files)? – l0b0 Jan 19 '12 at 12:51
  • +1: our team had several issues similar to this one and after numerous attempts/approaches, this one was the best we found. Also if you are going to have a separate repository shared by numerous projects, you might want to consider creating a smaller, dedicated team to manage those shared files. Otherwise, someone on project C could make a change without understanding all implications for other projects and break projects A, B and D. This would be another argument for skipping point 1 in final solution. – DXM Jan 19 '12 at 19:34
  • #1: you don't want to do this, however, it's trivial to update the external directory of shared files. #5: you have to check the 'use externals' flag when updating to tell it that you want all linked directories. #6: not really, you'll have a directory elsewhere on the directory tree with all shared files in it. You can use file-externals but they're not quite as simple to work with (you have to have them already checked out somewhere for them to work). – gbjbaanb Jan 20 '12 at 15:32

If files themselves put-together are independent (rather self dependent) - it does make sense to have them in separate repo and treat it as a dependency project for a give repo. This will minimize the copying files.

However, from the example you gave - these are things like Makefile, license etc. are not full fledged module by themselves. If you keep them separate - this will be random amalgamation and wont help as much as the pain it will put you in.

Also, given the same examples, i also don't see that to be a great resource saving exercise either. So overall, i don't quite see much value in this exercises.

I am not saying anything against DRY as a principle; but DRY or NOT DRY - will it have any significance here?

|improve this answer|||||
  • I tend to agree, but only because I haven't found any collection of developer tools which allow you to seamlessly and effortlessly mix and match repos. Maybe next-generation VCSes will take up the challenge. – l0b0 Jan 19 '12 at 14:22

One option is to have a separate repository for the shard files and then all repositories that use those files either fork from or merge in the shared file repository.

Here's an example with mercuiral. Say I've got a repo shared with the shared files and a repo project which has a project that uses the shared files. To get the shared files into the project repo I run the following commands from the project repo:

hg pull -f /path/to/shared/repo
hg merge
hg ci -m "merged shared repo"

This pulls and merges everything from the shared repo into the project repo. I can also make any changes to the files merged from the shared repo (say a specific makefile rule) and those changes will remain local to the project repo.

Now say I've updated the shared repo with some new features or bug fix and want to include that in my project repo. Then from the project repo I simply run

hg pull /merge/to/shared/repo
hg merge
hg ci -m "merged updated shared files"

And I have the updated versions. No -f flag was needed on the pull this time since the repos are now related.

For git the commands are almost identical. To merge the shared repo with the project repo I simply run

git fetch /path/to/shared
git merge FETCH_HEAD

git also supports remote repos making it so I don't have to enter /path/to/shared whenever you want to merge. I can add shared as a remote repo with

git remote add shared /path/to/shared

Then whenever I want to merge with it I enter

git fetch shared
git merge shared/master

Where master is the name of the branch in the shared repo.

While this isn't a single command, I don't think two commands is so bad (the commit form mercural doesn't count :P). Also it allows you to easily "cherry-pick" files since the files merged in from shared are like any other tracked files in the project repository. The downside is it doesn't support requirement 1. If I change the shared repo's files in the project repo, these changes will not be reflected in the shared repo. Instead I have to change them in the shared repo and merge those changes in.

|improve this answer|||||
  • This looks equivalent to the option to "download the files needed". Repositories won't do this for you unless there's a built-in mechanism like submodules, so AFAICT it doesn't fulfill points 1, 3 and 5. – l0b0 Jan 19 '12 at 12:31
  • @l0b0 I'm not sure what you mean by "repositories won't do this for you." The shared files will be in the specific project's repository, the only difference is they got there by being merged in or forked from a shared files repository. – David Brown Jan 19 '12 at 12:35
  • You mean merging and then committing/pushing, losing the connection with the include repo completely? That's equivalent to "creating a copy for each new repo." – l0b0 Jan 19 '12 at 12:42
  • You don't lose the connection between the repos since it's in the repo's history. Any new changes you make to the shared repo can be included with a simple pull and merge. – David Brown Jan 19 '12 at 12:48
  • I don't think I understand. Which VCS is this? git pull does merge without the need for a separate command, and git submodule uses update, not pull. Do you have an example session? – l0b0 Jan 19 '12 at 12:57

Your Answer

