2023 Year’s Progress


Assessing 2023

The video gives only a very quick overview, and without much explanation, so here’s a more in-depth breakdown of what got done on Lunatics Project in 2023. As usual, the year involved a combination of management, technology, development, and production work. I’ll start with production achievements, even though I actually did most of those later in the year.

Ragdoll Rig, PLSS Control Panel

As the Spring of 2023 wore on, I felt pressure to get back to doing production work on Lunatics. I had run into problems with the “SU-SuitingUp” sequence, and decided I needed to break up some shots, as well as create some updated assets.

The first of these was to try using a simulation-based “ragdoll rig” for Georgiana’s “space bunny” ragdoll (surely a “ragdoll rig” would be a good choice for an actual ragdoll, right?).

Ragdoll-rigged bunny
Ragdoll rigged ragdoll model, just starting to fall, under simulation.

Close up of new control panel for PLSS prop.

Upgraded PLSS Prop

Acceleration Couch with PLSS Controls Added
Acceleration Couch with PLSS Controls Added

This was a very interesting project to work on. I got it to more-or-less work, but it does have some strange collision behavior, and I’m not sure I’ve fully worked out how to cope with that. I have not yet incorporated this into any shots, but I can try to see if it helps (versus just animating with a conventional key-framed rig).

I also needed to create a control panel prop to detail the Portable Life Support System (PLSS) pack that is part of Georgiana’s acceleration couch (and the ground-test model shown in the SU sequence).

Pod-Staging Shot

I’m always a little daunted by re-working an already “completed” shot, but as the assets are updated, it sometimes becomes necessary. There were some serious limitations on the rigging for the Soyuz launch vehicle that limited how far I could follow the pods as they separated from the vehicle, which resulted in a not-quite-satisfying shot that I used earlier (this is from 2014):

In 2023, I finally got around to re-doing this animation, using the updated and rerigged Soyuz launch vehicle and flame-plume assets. I found a major problem with the flame-rigging, and solved that, first, and then worked to match the staging. It is still a little tricky to work with, as the pods have to transition from one rigging system to another, but ultimately pretty convincing:

Soyuz Interior Character Animation

The earliest animation for this project was the “teaser trailer” we developed from 2012 to 2013, which used early versions of our character and set assets. A lot of these have been completely replaced or greatly reworked since then!

So the new shots are almost entirely new creations. I had been collecting the assets and assembling the shot files over a long time, but had shied away from the character animation, even though I knew it would be fairly simple for this sequence, where the characters are meant to be strapped down in their seats, and so not moving much.

Soyuz Interior
Updated interior animation still of characters in Soyuz.

Also, in 2023, I finally solved the screen display rigging for the Soyuz set. This is intended to be controlled by manipulating custom bone properties on a bone object in the control panel armature. But it was not working yet. I also decided that it would make more sense to combine the four different videos I was using into a single large video, using Kdenlive. So I created this new combined video, from the sources I had:

Combined Video Texture
Combined video textures for the Soyuz consoles, being edited in Kdenlive.

Revised Pacing Shots

It can be difficult to correctly judge pacing from animatics, because the art is simpler, and the viewer gets bored with it more easily. The tendency is to run it too fast. Once you start adding detail, the shot becomes harder to follow and frustrating to perception. So, I have slowed down some of the shots in the “LA-Launch” sequence to compensate for this. Of course, this changes the sync with the music, and requires further revisions.

Soyuz Rollout Start
Image from the closeup of the Soyuz rolling by on the transporter-erector car.
Soyuz Rollout Two
Still from the middle of the closeup of Soyuz rolling by.
Soyuz Rollout Three
Near the end of the shot of the Soyuz rolling by and to the launch pad.

Adding Billboard Extras

A lot of the shots associated with the launch sequence were always supposed to have a modest crowd of onlookers, which I planned to create using “billboard extras” — that is, characters represented by models like cardboard cutouts, but which always face the camera. This is a very useful trick for keeping renders manageable, and the effect is actually quite convincing.

Billboard Extras as seen in the Blender viewport while editing a shot file.
Billboard extras seen from behind in the Blender viewport, which makes their true nature apparent.

Recovering the TR Train Sequence

The worst casualty of bitrot since the early phase of the project has been the files for the “TR-Train” sequence. The file was original created at an incorrect scale, and rather than fixing this, we rescaled the characters in the process of animation, with mixed results.

The set file was also not one our best, being very early, and thus not incorporating how much we’ve learned since then. So I spent some time in Summer 2023 fixing these assets at the corrected scale. The biggest challenge will come when I scale the character animation up to match. I wrote code in ABX to help with this, because scaling animation does not work quite like you might expect in Blender.

Reviewed All Sequences in S1E01

In Fall 2023, I re-assembled all of the sequences in the pilot episode, to see what the status is of all the shots (storyboards, animatics, blocking, animating, rendered, or composited). This has also allowed me to reassess the pacing and other factors, and I made the decision to break the existing timing, allowing me to lengthen the episode by a bit. Since the episode is a bit short as it is, I’m not worried about this (ideally, this is comparable to a “half-hour” animated show, which is usually about 20-22 minutes long, and the previous edit of Lunatics S1E01 was just 16.5 minutes, including titles).









Rendering on Dell Server

At some point during the server installation process, it occurred to me that this machine has very similar specifications to the server Andrew Pam had donated some time on for rendering — so I could install Blender and start doing some rendering on it, while it’s sitting here in my office. So I did that!

I installed multiple versions of Blender (specifically 2.71, 2.79b, 2.83LTS, 2.93LTS, 3.0.0, 3.6.5LTS, and 4.0.1) on my workstation, testing to make sure they can all run (I also attempted 2.69 for interest sake, but it didn’t work, and I don’t actually need it):

Blender versions
Several versions of Blender open at once on my workstation, to demonstrate that they all work.

I then transferred the same installations to the server machine, having to add a few extra library dependencies to get them all to run. In fact, I only really need 2.71 and 2.79 for rendering, but I wanted to be sure the others were available for future use.

As a result, I have a lot more shots fully rendered now, particularly in the “LA-Launch” sequence.

Shot from LA-Launch sequence, now with extras characters, and fully rendered.

Leaving Git

Coming up with a good, free-software solution for digital asset management for multimedia studio production has been one of my main project goals since I conceived of Lunatics Project in 2010. We’ve gone through several different solutions and partial attempts:

  • 2010-2012: MediaWiki
  • 2012-2018: Subversion + Trac
  • 2018-2022: Subversion + ResourceSpace
  • 2022-2023: Git + GitLFS + Gitea
  • 2023-2024: Nextcloud (still working on this migration)

In addition, I experimented extensively with Southpaw’s “TACTIC” as a multimedia project DAMS in 2020.

I’m still not really satisfied with this situation, and since 2020, I’ve been working more-or-less on my own, so effective collaboration has been an academic question. But if “Lunatics!” is ever to get back into full production, I’m going to need a practical solution.

The Git/GitLFS/Gitea solution, closely related to my decision to move to YunoHost as a basis for our self-hosted “virtual studio” website platform, was, unfortunately a disaster. Not only was it extremely complex to use, but I had multiple occasions of somehow creating conflicts — despite the fact that no one else was involved!

The last straw was when I somehow managed to replace every SVG file in the repo with a conflict file, apparently representing the LFS pointer (taken as a literal file), conflicting with the SVG original. Which made the SVG files unusable, of course. I eventually wrote a Python script to recover all the SVG files.

No doubt, I, a person with some programming and version-control experience, would eventually figure this problem out, but the prospect of trying to train prospective multimedia contributors to use is simply inconceivable. I know what would happen: everyone would wind up working around the problem, leaving me to take up the slack.

Better to just manage files without versioning, as was recommended to me by Morevna Project, which uses Nextcloud. This is fortunately, supported by an app for YunoHost, so it’s very easy to host.

Early in 2023, my task was to migrate all of the source files out of Git and back onto a simple file-tree on disk, and then start moving them into Nextcloud.

I should note that, although Git/Gitea was a disaster for production sources, I see no reason not to use them for managing source code for our software packages. Contributing programmers will already be familiar with Git’s quirks, and I am certain I can learn my way around them. Furthermore, the features which are basically useless for multimedia work are very useful for the software files they were designed for. So, I still plan to use Gitea — just not for animation files.

VS Codium

I used to simply use an editor and a terminal for Python development. Kate, which I used for logs and general writing, is also a pretty good programming editor.

But a few years ago, I decided that using an IDE would be essential for me, if I wanted to maintain my long-running programming projects. And in particular, Blender extensions involve complex testing rigs, which it’s better to automate. If I just manage this with the filesystem and an editor, it becomes very easy to lose my place, forget how things work, or where they are stored during the long gaps between when I have time for development sessions.

For this, I adopted Eclipse, as a general-purpose, free-software IDE. The favorite options for Python programmers are proprietary and/or too specialized to Python (whereas I also need to work with Javascript, C/C++, HTML, CSS, and other programming and data languages).

VS Codium
LunaGen in VSCodium

But Eclipse is a large, cumbersome package, to which I needed to add PyDev for Python development support, and a problem that arose is that Eclipse would automatically update itself, break PyDev (which seems to need some manual configuration in order to keep working), and then I would have an unusable IDE.

On tips from, among other people, my offspring, I decided to do the unthinkable and try a Microsoft product. Visual Studio Code, is, of course, a proprietary IDE. But the original codebase is under a free-license, and there is a free build of the software, called “Codium” or “VS Codium”, which I was able to install on my Ubuntu Studio machine in 2023.

And I have to say that I liked it a lot better than Eclipse. I have been using it for all of 2023, and I will probably stick with it.

My only concern is that the ecosystem does try to push you towards installing proprietary extensions (which Eclipse also did, of course). My intention is to stick meticulously to the free-software versions of everything. But so far, it has not broken itself by trying to upgrade without my authorization.

Installing and learning to use VS Codium was an important project for me in 2023.

Codium and Testing Blender Add-Ons

After recovering my projects into Codium, a major goal was to get my automated tests back to working. This included the tests for the LunaGen site generator and the Anansi Blender Extensions (ABX).

Without PyDev and Eclipse, I needed to move more of this into Python scripts. This makes me a little happier, as it’s more explicit and less dependent on the IDE.

I have been hoping to continue refactoring LunaGen to make it more useful outside of Lunatics Project (I do have other sites I want to manage, and also this might make it useful to other people). With ABX, a major limitation is that I have only got it to work in Blender 2.79, which is getting increasingly obsolete by now, and I want to use it for compositing, which I hope to get working in a later version of Blender (a moving target, but currently this would probably mean Blender 3.6.5 LTS). This means that I need a practical solution for testing ABX across multiple versions of Blender automatically. Testing for Blender Add-Ons is already complicated by the fact that they need to run in Blender, and this multi-version requirement means they need to:

  • Launch Blender (a list of version, one at a time).
  • Inject the test suite code
  • Run the test suite
  • Collect the test results
  • Present the results in a useful way to the developer

To me, this says “make a table of results”, so after much fiddling with the Python unittest API and Blender, that’s what I got it to do in September 2023:

ABX Test Table Output
Tabular output from unit-tests running on ABX. The rows identify the individual tests, while the columns show the versions of Blender they are running in. It correctly shows that ABX is only working in 2.79. This is mostly because the package doesn’t automatically install the PyYAML library that ABX needs.

Consolidating YunoHost-Based Virtual Studio

In 2022, I experimented with YunoHost as a solution for building our “Virtual Studio”, but in 2023, I started paring things back down to what we would really need, and trying to get it to work. However, it became clear that I was hitting the limits of what I could do with cloud-based infrastructure. This diagram shows how the system was set up during 2023, including the server and my own workstation:

Diagram of 2023 Virtual Studio & Workstation
A plan of the “Virtual Studio” website, as currently implemented on Digital Ocean, and of my animation workstation, Sintel, as currently setup in late 2023.

However, this arrangement has been pushed to its limits. I used to pay about $5/month for a minimal Digital Ocean droplet. Now, I’m having to pay about $70-$100 to pay for the various cloud resources I’m using. At this point, the cost is similar to very low-end datacenter colocation rent.

WordPress-Based Production Log

Nextcloud File View

Software Spreadsheet Open in Collabora on Nextcloud

Film Freedom Project Gitea
Film Freedom Project Gitea

Playlists of videos on the Lunatics Channel on our PeerTube server
Lunatics Channel (PeerTube)

Terry Hancock (Producer/Director)
Terry Hancock (Producer/Director) Misskey Profile

Pixelfed for Lunatics Project
Pixelfed for Lunatics Project

And while switching to colocation won’t actually save me money, it will greatly extend the capacity of the server, particularly in local data storage, which is a strong limitation of the cloud solution. On the cloud system, I’m obligated to put most of that data into “object storage”, which is a very useful service, but performs very sluggishly.

That might be at the root of some of my performance issues with PeerTube, although I suspect that may be a more complex problem to solve.

But most importantly, it made accessing files in my Lunatics production repository painfully slow.

So, I made the decision to migrate to a physical server that I control myself, running in a datacenter, because we lack the bandwidth to run such a thing on our premises.

Migrating Cloud to Metal

The only real problem with my repository is that the bulk of the files is such that I could not manage them with any affordable cloud-based virtual server without putting all that data into “object storage” (like Amazon S3 or Backblaze B2, which I have been using). Manipulating files can literally take hours to process. At such extremes, there are also failures due to timeouts and the resulting connection losses.

After some extensive cost analysis, I decided that the only real solution would be to move from our existing cloud-based system to a dedicated physical server in a local datacenter. I spent much of 2023 working on this solution:

  • Found a local “retail colocation” provider in Dallas and got a quote.
  • Purchased an off-lease Dell PowerEdge R720 2U server, with plenty of drive bays.
  • Configured an SSD and a larger hard drive for repositories
  • Installed Debian 11 + Proxmox VE + YunoHost 11 in a virtual machine on the server
  • Purchased a Netgate 1100 pfSense router to set up for virtual private network (VPN) maintenance access
  • Migrated YunoHost applications and data from the cloud server to the new server

And that’s where I’m at at the beginning of 2024.

Dell PowerEdge R720
Dell PowerEdge R720 set up for testing.

Not one of these steps was easy for me: the provider was hard to find; the server is an older model and not quite standard; the SSD was initially rejected, until I replaced it with different hardware, requiring a bizarre boot sequence; the operating system had to be installed in three parts; I was completely unprepared for the virtualization and computer networking requirements; I’ve never set up a VPN before (still haven’t); and then I ran into migration problems with three of the most-critical YunoHost applications: PeerTube, Nextcloud, and Gitea.

That last bit is my current problem: the backups seemed to work for these three packages, but the restore scripts don’t work. I will need to unpack the backup archives and debug the script. Somehow, a large part of the contents are not getting unpacked, even though I can see the files are present in the archive file.

I also am a little concerned that the provider may get frustrated with me jerking their chain over move-in dates, since I keep having to delay, and retail colocation is not as widely available as it once was.

My goal, still not yet achieved, is to get the new Virtual Studio set up something like this:

Planned 2024 Virtual Studio
The updated 2024 Virtual Studio, as I plan to set it up for 2024 on the co-located server.

The server is running Debian 11, with the Proxmox Virtual Environment running on that, and YunoHost 11 running in a virtual machine. This provides some flexibility, in case I need to have additional software that can’t be managed by YunoHost. I can then run a second virtual machine, with a different management scheme (“docker compose” perhaps, or manual installation, as I used to run all of it). And it is possible to run some software on the host computer.

The challenges of configuring the networking and maintenance for all this have demonstrated to me that this is actually pretty far outside of my comfort zone with computing. I spent a good part of November 2023 just watching online video lectures to try to catch myself up. I have yet to tackle setting up the Virtual Private Network router, or the OpenVPN configuration I’ll need to set up on my workstation to access it.

LunaGen Refactoring

The “LunaGen” script I use to generate our static release site (lunatics.tv) was kind of an ad hoc solution, since I couldn’t find a good off-the-shelf match for what I wanted. It did the initial job, but it’s hard to maintain. There’s not good separation between the Lunatics site specific design and the static generator code.

LunaGen Test Page
One of the functional tests pages for LunaGen in my FireFox browser.

My plan is to “refactor” it into more of a “plugin” architecture, and to separate the site-specific stuff from the general-purpose parts. I did some work on this in 2023, though it’s still not completed. Other priorities pulled me away from it.

Realtime Screencasts

For the Lunatics development work, the Ragdoll Rig, and the PLSS Control Panel projects, I decided to try doing “realtime screencasts”.

Realtime Screencast Cover Poster
Realtime Screencast for work on LunaGen and Lunatics website.

Livestreams are a popular item on YouTube, Twitch, and so on. I can’t do those for connectivity reasons — the upload bandwidth from our very rural site is only about 0.5 mbps. It’s a consumer-based service, unfortunately, so they didn’t allow much for upload capacity. We looked into upgrading, but any solution is pretty expensive, and the best option remains to put up a 50-foot tower for a line-of-sight wireless connection. Fiber optic service is, sadly, not yet available out here.

This was pretty fun, but ultimately, I realized that it was too labor intensive and it was distracting me from work on finishing the pilot for Lunatics, which I decided was more important. Still, I might do more as time allows.

Lib-Ray / Bunsen

I’ve been working very slowly on the “Bunsen” project to create a fully free-software control system for my Primera Bravo II Disk Duplicator, which I’ll need for producing M-Disc based Lib-Ray discs.

In 2022, I modified the duplicator to be able to handle M-Disc media. And in 2023, I collected the documented control codes for the robotics (a starting point for experimental testing) and tested a simple method of running tests so I can record results against what is being sent to the machine, by using my webcam.

Screen capture of Bunsen debugging configuration.
Screen capture of Bunsen debugging configuration.

Heat Pump

This may seem more personal than project related, but Summer 2023 was very hot, and it’s pretty difficult to concentrate when the temperature in my office is hitting 90 F (32 C), not to mention bad for the equipment. The portable air conditioner I had been using was just not up to the task. So, in July, I installed a DIY “mini-split” heat pump system. This was fairly expensive (about $2000), but it has been worth every penny. As I write this in January, I’m appreciating how much better it works for heating, and it was fantastic during the Summer for cooling.

Mini-Split Heat Pump (Blower)
Interior part of the new mini-split air-conditioner/heat pump.

Inkscape Font Folders

As you probably know, I use Inkscape extensively for creating texture and “decal” artwork for Lunatics Project.

One feature I have wanted to see for a very long time is a better way to organize fonts, so I can group the common fonts I use in Lunatics, or group fonts by broader categories (like handwriting, comic lettering, stencil, display, high-readability, etc).

And in 2023, this finally arrived in Inkscape 1.3!

So I naturally proceeded to categorize my fonts, including a set I’ve selected for use in Lunatics:

Font Collections in Inkscape 1.3
New “Font Collections” dialog in Inkscape 1.3, with common fonts for Lunatics Project opened up.


I’m a little disappointed that I didn’t manage to finish any of the personal and professional goals I set for 2023. It would’ve been a nice symbolic gesture to be able to check them off at the New Year. But in fact, I still have some work to do on them all. However, the point was really to put the work in on them, and I feel like I have done that.

2023 Goals Wallpaper

I decided to move the goalposts a little, and made myself a set of modified goals for “2024 Q1” (that is, before April):

I dropped the storefront goal, deciding that our existing Gumroad account is adequate. And I decided not to write a commercial Kdenlive book, but instead simply put that energy into “Film Making for Lunatics”, which is on my list for the end of 2024 (not shown).

I’ve carried over the bathroom remodeling and Lunatics S1E01 goals, as well as a more-concrete Virtual Studio goal (just getting it running on the new server). The fourth goal is one I can’t shift from the 2024 goals: getting ready for the total solar eclipse on April 8th!  I’m very optimistic that I can adequately meet these goals by April, so I feel pretty good about this. I guess we’ll see if that’s justified — in about three months!

Happy New Year!


Avatar photo
Terry Hancock is the director and producer of "Lunatics!" and the founder for "Lunatics Project" and the associated "Film Freedom" Project. Misskey (Professional/Director Account) Mastodon (Personal Account)