I’ve been looking for a good solution to manage source (and render) files from the project, for people who simply want to have the entire open movie project, and for contributors who need a “local copy” to work with. For a long time, we used Subversion, then I attempted to migrate to Git with a self-hosted Gitea (and LFS, etc). and then I got a recommendation from Konstantin Dmitriev at Morevna Project to simply keep the files in Nextcloud.
Nextcloud
Using Nextcloud seemed like a nice option, since I was already experimenting with it to use for our business documents. It boasts several advantages over previous systems I have tested:
- Simple to use folder structure and office tools.
- Familiar to many users coming from a business setting (in addition to being popular itself, it resembles SaaS offerings like Google Office).
- No elaborate user interface for synchronization (unlike Git).
- Relatively free from entanglements and conflicts from commits.
- Lacks complex tools for text merging that just get in the way with multimedia files.
On the other hand, it does, of course, have some disadvantages:
- No actual version control. Just synchronization of current state.
- Does not support symbolic links.
- Can be slow for synchronizing large folders.
I have begun testing Nextcloud for use with the Lunatics Repo, on the Dell PowerEdge server that I’m (still) preparing for co-location.
The lack of version control is something we can address with a snapshot backup system, like rsnapshot. But we’ll have to be careful about the granularity we want.
The lack of symbolic link support is the current thorn in my side: I don’t really have a proper solution for this. I may have to write a script to revise the links on the client side after synchronizing. I’m really not sure how to streamline this for contributors. Yet.
Nextcloud Desktop
Compared to running rsync, which has a lot of optimizations for handling large transfers, using compression and minimizing the amount of data to copy, Nextcloud’s synchronization, via the Nextcloud Desktop application is pretty slow. Even though I’m only syncing between the workstation on my desk (“Siintel”) and the test server (“Narya”), the sync time runs into hours for the 26 GB source tree and days for the render folders with EXR-stream output (about 330 GB).
This is not actually that terrible, considering the large volume of data to be synchronized, but I know it will only get worse when it’s communicating between my server in a data center, across various links, and out to a consumer ISP to a contributor, who will likely get frustrated with the time it takes. The bandwidth requirements might be prohibitive.
The first attempt was not too promising:
The application report 720 GB partly because it is counting the “Renders” folder twice: once by direct reference and once via a symlink in the S1E01 episode, which get dereferenced. That is, Nextcloud doesn’t know that S1E01-Prolog/Renders is the same folder as Renders/S1E01, so it copies it twice.
The entire repository is actually on the filesystem on the server, which is loaded into Nextcloud as an “External Folder”. Since Nextcloud doesn’t support symbolic links, it simply dereferences the symbolic link I used to separate the renders from the source. Thus the two folders are not recognizable as two links to the same folder in the Nextcloud interface. This isn’t really a problem for using the files via the web interface. But when I try to synchronize the data back onto my desktop machine, using the Nextcloud Desktop application, it now tries to copy the data twice.
And really, it probably shouldn’t be trying to copy that data even once, because it’s so huge.
Grsync
It might be desirable to use something simpler, but more efficient, for handling the large “Renders” folders. About the most efficient tool I know for this is rsync, but it is a old Unix utility with a somewhat arcane interface that might be off-putting to contributors.
I looked into using an rsync-based utility, Grsync, which simply provides a GUI front-end for rsync. But the program is fairly complicated to use, and I can see how it’d be easy to make destructive mistakes with it, just as it is when using rsync on the command line:
I could produce a little tutorial for contributors on how to do this transfer with Grsync. Or I could explain how to do it on the command line with rsync. Or, better yet, I could just write rsync scripts with the required options set up.
On the other hand, there may be complications, especially for contributors using consumer internet connections at home (as am I). These services may block ports rsync needs to work properly, so rsync might not do the trick. Then it might be necessary to do some kind of “tunneling” through the available ports. Nextcloud, presumably works around these problems, since it is specifically designed for such users.
Mailing Hard Drives?
The latency is terrible, but the bandwidth is enormous!
We could arrange to ship the render data on a portable hard drive or solid state drive. We’d need about 500 GB – 1 TB devices for this, and obviously that has a cost. But we could offer a refund if the drives are returned afterwards. This might be a way to handle the initial transfer of the large “Renders” folders. We could offer the current episode in progress as well as past episode masters as an item in the Gumroad store, priced to cover the expenses, including the drive. Then have a refund program for receiving our disk drives or SSDs back.
Or for the cash-strapped, we could offer to copy the data onto a disk sent to us with a SASE (“self-addressed stamped envelope”, for younger readers who may not remember this ancient postal ritual).
Documentation
As with a number of other software packages, like Github, Nextcloud interprets a “README.md” file in any folder as documentation for that folder, and formats it, alongside the file and folder display. This has been very useful for me to document the structure of the project folders, and I’ve been adding lots of these read-me files. Since the folders are actually directly on the filesystem, and I’m simply working through Nextcloud, these are written directly to the source tree. So this does not build-in any direct dependency on Nextcloud, which I like. Markdown is a widely-supported format, and Nextcloud is simply acting as a wysiwyg editor for them, which certainly speeds up the process.
