Apparently, our Virtual Studio project is interesting to more than just us. I got a number of remarks about it. Since then, partly due to input I received after the post, I can make some minor changes to my plans.
CGRU/Afanasy
I knew of course that other open-source render farm software already existed, but I had not yet discovered that RenderChan already has some integration work done to support one of them. That’s Afanasy, which is part of a larger package called CGRU. This is all available as Debian packages already, so not hard to install.
This makes the “RenderQueue” concept probably unnecessary, or if we do some work on it to provide missing features, it’ll probably just be a patch for RenderChan, rather than a Flask/Python project.
Thanks to Konstantin Dmitriev from Morevna Project, for pointing this out.
Amazon S3 / Fuse / S3fs
I had not really considered how much of a financial strain it will be to continue supporting our repository on our web server. The Subversion repo is currently 15GB, and the TACTIC repo will of course grow larger than that quickly. As it is, I’m hitting our 60 GB file storage limit on the server in the process of testing Subversion repository backups, so it’s clear that storage is going to become an issue. Also, we’re paying about $20 extra a month for the 60GB limit, versus the regular 30GB on our VPS. Going higher gets pricey fast, and we may need several times this much space before we’re done even with the first episode. I remember when 60GB seemed like a lot of space!
So, I’ve been looking into alternatives, and the best option appears to be cloud storage using Amazon’s “Simple Storage Service” (S3) protocol (which includes, but isn’t limited to Amazon). A few estimates show that for the same $20/month extra, we could be using up to about 2 TB of repository storage, versus the 30 GB we have now (i.e. a factor of about 70X!) using Amazon’s storage hosting, and that’s entirely sufficient for foreseeable needs.
Just as a side note, this doesn’t build in a closed-source dependency, as there are open-source S3 API library for providing compatible hosting. It’s possible we should look into hosting from a vendor using one, but it doesn’t really affect our software plans.
To integrate this with TACTIC, we’ll need to make this storage available as a filesystem, but that can be done with open-source software: specifically “FUSE” and “s3fs“, both already packaged in the standard repository for Debian 9 “Stretch”. (Of course, the s3fs storage is not as fast as true file-based storage and some POSIX filesystem features aren’t supported, but that’s not a serious problem for us).
So I’ve updated my server deployment plans to include these arrangements.
