Assets and version control

Discussion in 'Game Development (Technical)' started by ggambett, Mar 15, 2010.

  1. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    Since the beginning of Mystery Studio I've been keeping everything related to a game in source control (CVS at first, SVN for a few years now).

    The thing is, a full backup of the repository is now over 9 GB. Anything over the size of a DVD is problematic.

    Most of the space is taken by assets which don't really need version control - cutscenes, for example. I really want the full history of source code and scripts, but not of assets.

    I guess I'm looking for a hybrid version control system where I can keep only the latest version of some files and the full history of others. SVN doesn't do this and I'm not aware of any VCS that does. Does anyone have anything to recommend?
     
  2. Pogacha

    Original Member

    Joined:
    Jan 21, 2005
    Messages:
    605
    Likes Received:
    0
    There is one tool I don't rememer the name for doing that ... I searched for it but coudn't find it, if I remember it later I'll post again.

    What I do is to have a repository for Code, other for assets sources y other for binaries. The binaries one holds the assets objective files and executables.

    The assets one is the biggest, after finishing a project I make a backup on DVD of the assets and delete the repository and start again with it.
     
  3. Indiepath

    Indiepath New Member

    Joined:
    Aug 22, 2004
    Messages:
    999
    Likes Received:
    0
  4. frozax

    Indie Author

    Joined:
    Sep 16, 2005
    Messages:
    95
    Likes Received:
    0
    You can do this with Perforce (perforce.com). Perforce is free for two users or less.
    You can specify for each file the number of revisions you want to keep. You can set it to one revision for binary files and unlimited revisions for source code.

    I personally use Perforce for source code only. My assets are saved using a standard backup system from time to time (no source control, but I sometimes keep two versions of my backups just in case).
     
  5. chanon

    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    468
    Likes Received:
    0
    Do you have 1 game per repository? Or is it lots of games in a single repository? If it's 1 repository per game then that must be a BIG game to take up 9 GB. Otherwise you should have separate repositories per each game and backing up would be easier / you can easily archive away old projects.
     
  6. dewitters

    Original Member

    Joined:
    Dec 19, 2005
    Messages:
    128
    Likes Received:
    0
    Losing history on assets sounds horrible to me. I always want to view history or go back to any previous version.

    Shrinking a repository is best done by svndump (as Indiepath suggested). This way your latest repository gets smaller (with any filter you want to apply on svndump), but you still keep *all* of your history backed up in your old repository in case you still need it. So in my opinion it offers the best of both: small latest repository and complete history (although in another repository).

    The bad part is that now and then, you need to run svndump, but normally this shouldn't happen often.
     
  7. Michael Flad

    Indie Author

    Joined:
    Aug 4, 2004
    Messages:
    190
    Likes Received:
    0
    It's possible to remove old revisions in CVS, haven't done it though as I don't mind the required space so far.

    What I did wrt to most easy and fast backups ... our CVS and SVN server runs in a virtual machine - so doing a full backup to an external HD is a question of a few minutes. External 2.5" HDs (no need for external power supply) are dead cheap these day's so I use 3 of them ... one to do current daily backups exchanged with another one every few days. I have 3 because I always have one in another house, just in case the office burns down etc.

    One additional advantage of the VM is, if the machine running the VM dies for whatever reason I can move our CVS/SVN to another one easily and with almost no installation/configuration work.
     
  8. Bad Sector

    Original Member

    Joined:
    May 28, 2005
    Messages:
    2,742
    Likes Received:
    5
    With git you can write a script that after a commit it removes the history of some files. Your script can check a special file in your repository for the list of files to keep only the latest commit of. How to do that exactly depends on your repository configuration. Git itself doesn't provide this functionality, but it provides a lot of hooks to add it yourself. If you look around you might find premade receipes for this since its something that some people need (usually those who want to keep binaries).

    But start with a "git svn clone <your svn repos>" first. Git repositories are much smaller than SVN repositories and you might be able to fit it in a DVD without the need of losing binary history (which might not be a good idea since old code usually depends on old binaries/formats).

    Another option would be to buy a BluRay recorder :p.
     
  9. PoV

    PoV
    Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    2,132
    Likes Received:
    0
    I use multiple repositories, and retire ones that I'm not using currently from my backups. I think my largest repository is only 1 GB, but yeah, SVN Dump sounds like the way to go.

    Video and Audio I don't use a repository, but burn a backup ("master") copy once I'm satisfied. Songs and project files I save multiple numbered files for revisions. Not perfect, but it's better than nothing.
     
  10. Julio Gorge

    Original Member

    Joined:
    Nov 26, 2005
    Messages:
    152
    Likes Received:
    0
    Another possibility would be using a pro account of Dropbox for asset storage. Dropbox saves a history of all deleted and previous versions of files for 30 days by default. An all-in-one simple VCS and backup solution :)

    If you wanted to keep the full history of files, you could always enable the pack-rat add on. It costs an extra $3.99/month I think, but the file history does not count against your storage limit so you can forget about all file versions taking too much space.
     
  11. chanon

    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    468
    Likes Received:
    0
    One thing to add, when working with multiple repositories, svn:externals is very useful. With it you can create a "virtual" folder in one repository that points to a folder in another repository. You can fix the revision of the source folder or set it to just get the latest on update.

    You could for example create a separate assets repository for a project and create a svn:externals folder in your main repository that points to the assets repository. This way you can backup them seperately, or if later you decide to clear old versions of assets you can do that in the assets repository without messing with the main repository.
     
  12. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    Thanks for your suggestions!

    I thought of svnadmin dump but some things have moved around in the repo, so getting the paths right may be very tricky. From what I read, svnadmin dump is quite strict regarding that.

    I didn't think about svn:external, it may be very useful - thanks!

    @dewitters : this is for projects where we don't do the art, only programming and scripting, so it's not that terrible.
     
  13. S2P

    S2P New Member

    Joined:
    Dec 1, 2006
    Messages:
    105
    Likes Received:
    0
    We made our own software (Asset Management) to keep track of code, screenshots of artwork assets, location on the network, licence agreements and so on as our pool of assets started to get out of control. We're going to release this as a tool for game studios in the next few weeks, if you want me to keep you updated so you can have a look at the demo, drop me a PM with your email and I'll keep you updated. It's totally online through a web browser with full search functionality (requires php/mysql database) so if you have remote teams updating it then you can give them access with priviledge levels. It can also be used as a portfolio, or showcase work to clients with their own login.

    Edit: forgot to mention - free for Indie developers.
     
    #13 S2P, Mar 16, 2010
    Last edited: Mar 16, 2010
  14. Mark Fassett

    Moderator Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    541
    Likes Received:
    0
    How do you do this in Perforce? I can't find the option...
     
  15. Gary Preston

    Original Member

    Joined:
    Aug 5, 2005
    Messages:
    239
    Likes Received:
    0
    Is your SVN server local? If so, you could do the daily/nightly backups to an external HDD. Added benefit is you can rotate disks every day or week (allowing offsite storage of the unused backup disks).

    Every so often you can then take a new full dump and burn that to DVD. If it's over the size of a dual layer dvd then use tar with the multi-volume option to span disks, you could also pipe the results through BZip which should help reduce the size.

    If the repo is remote, then a full dump once every month or two + incremental dumps is probably a better option.

    As already mentioned svn:external is worth looking into. We've started to use this quite a bit with our latest projects. Your final option is to buy a blu-ray burner :p
     
    #15 Gary Preston, Mar 16, 2010
    Last edited: Mar 16, 2010
  16. frozax

    Indie Author

    Joined:
    Sep 16, 2005
    Messages:
    95
    Likes Received:
    0
    With P4V, right click a file in a changelist (marked for add and change), select "Change Filetype...", Checkbox +S server limits the number of revisions.
     
  17. James C. Smith

    Moderator Original Member

    Joined:
    Aug 21, 2004
    Messages:
    1,768
    Likes Received:
    0
    I use one repository for all the code for every game I ever worked on. Then I have a separate repository for each game's assets. (actually, squeals share the same repository as the first game) This keeps the size of the repositories manageable, allows sharing of code between projects (which is VERY common) and doesn't allow sharing of assets between projects which is almost never a problem. This means programmers need to work with two repository at once which is a slight inconvenience but not a real problem. All the contents creators (artists, level designers, sound & music guys) deal with just one repository at time (unless they are working on multiple games at once).
     

Share This Page

  • About Indie Gamer

    When the original Dexterity Forums closed in 2004, Indie Gamer was born and a diverse community has grown out of a passion for creating great games. Here you will find over 10 years of in-depth discussion on game design, the business of game development, and marketing/sales. Indie Gamer also provides a friendly place to meet up with other Developers, Artists, Composers and Writers.
  • Buy us a beer!

    Indie Gamer is delicately held together by a single poor bastard who thankfully gets help from various community volunteers. If you frequent this site or have found value in something you've learned here, help keep the site running by donating a few dollars (for beer of course)!

    Sure, I'll Buy You a Beer