2023 Year’s Progress

Ragdoll Rig, PLSS Control Panel

As the Spring of 2023 wore on, I felt pressure to get back to doing production work on Lunatics. I had run into problems with the “SU-SuitingUp” sequence, and decided I needed to break up some shots, as well as create some updated assets.

The first of these was to try using a simulation-based “ragdoll rig” for Georgiana’s “space bunny” ragdoll (surely a “ragdoll rig” would be a good choice for an actual ragdoll, right?).

This was a very interesting project to work on. I got it to more-or-less work, but it does have some strange collision behavior, and I’m not sure I’ve fully worked out how to cope with that. I have not yet incorporated this into any shots, but I can try to see if it helps (versus just animating with a conventional key-framed rig).

I also needed to create a control panel prop to detail the Portable Life Support System (PLSS) pack that is part of Georgiana’s acceleration couch (and the ground-test model shown in the SU sequence).

Pod-Staging Shot

I’m always a little daunted by re-working an already “completed” shot, but as the assets are updated, it sometimes becomes necessary. There were some serious limitations on the rigging for the Soyuz launch vehicle that limited how far I could follow the pods as they separated from the vehicle, which resulted in a not-quite-satisfying shot that I used earlier (this is from 2014):

In 2023, I finally got around to re-doing this animation, using the updated and rerigged Soyuz launch vehicle and flame-plume assets. I found a major problem with the flame-rigging, and solved that, first, and then worked to match the staging. It is still a little tricky to work with, as the pods have to transition from one rigging system to another, but ultimately pretty convincing:

Soyuz Interior Character Animation

The earliest animation for this project was the “teaser trailer” we developed from 2012 to 2013, which used early versions of our character and set assets. A lot of these have been completely replaced or greatly reworked since then!

So the new shots are almost entirely new creations. I had been collecting the assets and assembling the shot files over a long time, but had shied away from the character animation, even though I knew it would be fairly simple for this sequence, where the characters are meant to be strapped down in their seats, and so not moving much.

Also, in 2023, I finally solved the screen display rigging for the Soyuz set. This is intended to be controlled by manipulating custom bone properties on a bone object in the control panel armature. But it was not working yet. I also decided that it would make more sense to combine the four different videos I was using into a single large video, using Kdenlive. So I created this new combined video, from the sources I had:

Revised Pacing Shots

It can be difficult to correctly judge pacing from animatics, because the art is simpler, and the viewer gets bored with it more easily. The tendency is to run it too fast. Once you start adding detail, the shot becomes harder to follow and frustrating to perception. So, I have slowed down some of the shots in the “LA-Launch” sequence to compensate for this. Of course, this changes the sync with the music, and requires further revisions.

Adding Billboard Extras

A lot of the shots associated with the launch sequence were always supposed to have a modest crowd of onlookers, which I planned to create using “billboard extras” — that is, characters represented by models like cardboard cutouts, but which always face the camera. This is a very useful trick for keeping renders manageable, and the effect is actually quite convincing.

Recovering the TR Train Sequence

The worst casualty of bitrot since the early phase of the project has been the files for the “TR-Train” sequence. The file was original created at an incorrect scale, and rather than fixing this, we rescaled the characters in the process of animation, with mixed results.

The set file was also not one our best, being very early, and thus not incorporating how much we’ve learned since then. So I spent some time in Summer 2023 fixing these assets at the corrected scale. The biggest challenge will come when I scale the character animation up to match. I wrote code in ABX to help with this, because scaling animation does not work quite like you might expect in Blender.

Reviewed All Sequences in S1E01

In Fall 2023, I re-assembled all of the sequences in the pilot episode, to see what the status is of all the shots (storyboards, animatics, blocking, animating, rendered, or composited). This has also allowed me to reassess the pacing and other factors, and I made the decision to break the existing timing, allowing me to lengthen the episode by a bit. Since the episode is a bit short as it is, I’m not worried about this (ideally, this is comparable to a “half-hour” animated show, which is usually about 20-22 minutes long, and the previous edit of Lunatics S1E01 was just 16.5 minutes, including titles).

Rendering on Dell Server

At some point during the server installation process, it occurred to me that this machine has very similar specifications to the server Andrew Pam had donated some time on for rendering — so I could install Blender and start doing some rendering on it, while it’s sitting here in my office. So I did that!

I installed multiple versions of Blender (specifically 2.71, 2.79b, 2.83LTS, 2.93LTS, 3.0.0, 3.6.5LTS, and 4.0.1) on my workstation, testing to make sure they can all run (I also attempted 2.69 for interest sake, but it didn’t work, and I don’t actually need it):

I then transferred the same installations to the server machine, having to add a few extra library dependencies to get them all to run. In fact, I only really need 2.71 and 2.79 for rendering, but I wanted to be sure the others were available for future use.

As a result, I have a lot more shots fully rendered now, particularly in the “LA-Launch” sequence.

Early in 2023, my task was to migrate all of the source files out of Git and back onto a simple file-tree on disk, and then start moving them into Nextcloud.

I should note that, although Git/Gitea was a disaster for production sources, I see no reason not to use them for managing source code for our software packages. Contributing programmers will already be familiar with Git’s quirks, and I am certain I can learn my way around them. Furthermore, the features which are basically useless for multimedia work are very useful for the software files they were designed for. So, I still plan to use Gitea — just not for animation files.

VS Codium

I used to simply use an editor and a terminal for Python development. Kate, which I used for logs and general writing, is also a pretty good programming editor.

But a few years ago, I decided that using an IDE would be essential for me, if I wanted to maintain my long-running programming projects. And in particular, Blender extensions involve complex testing rigs, which it’s better to automate. If I just manage this with the filesystem and an editor, it becomes very easy to lose my place, forget how things work, or where they are stored during the long gaps between when I have time for development sessions.

For this, I adopted Eclipse, as a general-purpose, free-software IDE. The favorite options for Python programmers are proprietary and/or too specialized to Python (whereas I also need to work with Javascript, C/C++, HTML, CSS, and other programming and data languages).

But Eclipse is a large, cumbersome package, to which I needed to add PyDev for Python development support, and a problem that arose is that Eclipse would automatically update itself, break PyDev (which seems to need some manual configuration in order to keep working), and then I would have an unusable IDE.

On tips from, among other people, my offspring, I decided to do the unthinkable and try a Microsoft product. Visual Studio Code, is, of course, a proprietary IDE. But the original codebase is under a free-license, and there is a free build of the software, called “Codium” or “VS Codium”, which I was able to install on my Ubuntu Studio machine in 2023.

And I have to say that I liked it a lot better than Eclipse. I have been using it for all of 2023, and I will probably stick with it.

My only concern is that the ecosystem does try to push you towards installing proprietary extensions (which Eclipse also did, of course). My intention is to stick meticulously to the free-software versions of everything. But so far, it has not broken itself by trying to upgrade without my authorization.

Installing and learning to use VS Codium was an important project for me in 2023.

Codium and Testing Blender Add-Ons

After recovering my projects into Codium, a major goal was to get my automated tests back to working. This included the tests for the LunaGen site generator and the Anansi Blender Extensions (ABX).

Without PyDev and Eclipse, I needed to move more of this into Python scripts. This makes me a little happier, as it’s more explicit and less dependent on the IDE.

I have been hoping to continue refactoring LunaGen to make it more useful outside of Lunatics Project (I do have other sites I want to manage, and also this might make it useful to other people). With ABX, a major limitation is that I have only got it to work in Blender 2.79, which is getting increasingly obsolete by now, and I want to use it for compositing, which I hope to get working in a later version of Blender (a moving target, but currently this would probably mean Blender 3.6.5 LTS). This means that I need a practical solution for testing ABX across multiple versions of Blender automatically. Testing for Blender Add-Ons is already complicated by the fact that they need to run in Blender, and this multi-version requirement means they need to:

• Launch Blender (a list of version, one at a time).
• Inject the test suite code
• Run the test suite
• Collect the test results
• Present the results in a useful way to the developer

To me, this says “make a table of results”, so after much fiddling with the Python unittest API and Blender, that’s what I got it to do in September 2023:

Consolidating YunoHost-Based Virtual Studio

In 2022, I experimented with YunoHost as a solution for building our “Virtual Studio”, but in 2023, I started paring things back down to what we would really need, and trying to get it to work. However, it became clear that I was hitting the limits of what I could do with cloud-based infrastructure. This diagram shows how the system was set up during 2023, including the server and my own workstation:

However, this arrangement has been pushed to its limits. I used to pay about $5/month for a minimal Digital Ocean droplet. Now, I’m having to pay about $70-$100 to pay for the various cloud resources I’m using. At this point, the cost is similar to very low-end datacenter colocation rent.

And while switching to colocation won’t actually save me money, it will greatly extend the capacity of the server, particularly in local data storage, which is a strong limitation of the cloud solution. On the cloud system, I’m obligated to put most of that data into “object storage”, which is a very useful service, but performs very sluggishly.

That might be at the root of some of my performance issues with PeerTube, although I suspect that may be a more complex problem to solve.

But most importantly, it made accessing files in my Lunatics production repository painfully slow.

So, I made the decision to migrate to a physical server that I control myself, running in a datacenter, because we lack the bandwidth to run such a thing on our premises.

Migrating Cloud to Metal

The only real problem with my repository is that the bulk of the files is such that I could not manage them with any affordable cloud-based virtual server without putting all that data into “object storage” (like Amazon S3 or Backblaze B2, which I have been using). Manipulating files can literally take hours to process. At such extremes, there are also failures due to timeouts and the resulting connection losses.

After some extensive cost analysis, I decided that the only real solution would be to move from our existing cloud-based system to a dedicated physical server in a local datacenter. I spent much of 2023 working on this solution:

• Found a local “retail colocation” provider in Dallas and got a quote.
• Purchased an off-lease Dell PowerEdge R720 2U server, with plenty of drive bays.
• Configured an SSD and a larger hard drive for repositories
• Installed Debian 11 + Proxmox VE + YunoHost 11 in a virtual machine on the server
• Purchased a Netgate 1100 pfSense router to set up for virtual private network (VPN) maintenance access
• Migrated YunoHost applications and data from the cloud server to the new server

And that’s where I’m at at the beginning of 2024.

Not one of these steps was easy for me: the provider was hard to find; the server is an older model and not quite standard; the SSD was initially rejected, until I replaced it with different hardware, requiring a bizarre boot sequence; the operating system had to be installed in three parts; I was completely unprepared for the virtualization and computer networking requirements; I’ve never set up a VPN before (still haven’t); and then I ran into migration problems with three of the most-critical YunoHost applications: PeerTube, Nextcloud, and Gitea.

That last bit is my current problem: the backups seemed to work for these three packages, but the restore scripts don’t work. I will need to unpack the backup archives and debug the script. Somehow, a large part of the contents are not getting unpacked, even though I can see the files are present in the archive file.

I also am a little concerned that the provider may get frustrated with me jerking their chain over move-in dates, since I keep having to delay, and retail colocation is not as widely available as it once was.

My goal, still not yet achieved, is to get the new Virtual Studio set up something like this:

The server is running Debian 11, with the Proxmox Virtual Environment running on that, and YunoHost 11 running in a virtual machine. This provides some flexibility, in case I need to have additional software that can’t be managed by YunoHost. I can then run a second virtual machine, with a different management scheme (“docker compose” perhaps, or manual installation, as I used to run all of it). And it is possible to run some software on the host computer.

The challenges of configuring the networking and maintenance for all this have demonstrated to me that this is actually pretty far outside of my comfort zone with computing. I spent a good part of November 2023 just watching online video lectures to try to catch myself up. I have yet to tackle setting up the Virtual Private Network router, or the OpenVPN configuration I’ll need to set up on my workstation to access it.

LunaGen Refactoring

The “LunaGen” script I use to generate our static release site (lunatics.tv) was kind of an ad hoc solution, since I couldn’t find a good off-the-shelf match for what I wanted. It did the initial job, but it’s hard to maintain. There’s not good separation between the Lunatics site specific design and the static generator code.

My plan is to “refactor” it into more of a “plugin” architecture, and to separate the site-specific stuff from the general-purpose parts. I did some work on this in 2023, though it’s still not completed. Other priorities pulled me away from it.

Realtime Screencasts

For the Lunatics development work, the Ragdoll Rig, and the PLSS Control Panel projects, I decided to try doing “realtime screencasts”.

Livestreams are a popular item on YouTube, Twitch, and so on. I can’t do those for connectivity reasons — the upload bandwidth from our very rural site is only about 0.5 mbps. It’s a consumer-based service, unfortunately, so they didn’t allow much for upload capacity. We looked into upgrading, but any solution is pretty expensive, and the best option remains to put up a 50-foot tower for a line-of-sight wireless connection. Fiber optic service is, sadly, not yet available out here.

This was pretty fun, but ultimately, I realized that it was too labor intensive and it was distracting me from work on finishing the pilot for Lunatics, which I decided was more important. Still, I might do more as time allows.

Lib-Ray / Bunsen

I’ve been working very slowly on the “Bunsen” project to create a fully free-software control system for my Primera Bravo II Disk Duplicator, which I’ll need for producing M-Disc based Lib-Ray discs.

In 2022, I modified the duplicator to be able to handle M-Disc media. And in 2023, I collected the documented control codes for the robotics (a starting point for experimental testing) and tested a simple method of running tests so I can record results against what is being sent to the machine, by using my webcam.

Heat Pump

This may seem more personal than project related, but Summer 2023 was very hot, and it’s pretty difficult to concentrate when the temperature in my office is hitting 90 F (32 C), not to mention bad for the equipment. The portable air conditioner I had been using was just not up to the task. So, in July, I installed a DIY “mini-split” heat pump system. This was fairly expensive (about $2000), but it has been worth every penny. As I write this in January, I’m appreciating how much better it works for heating, and it was fantastic during the Summer for cooling.

Inkscape Font Folders

As you probably know, I use Inkscape extensively for creating texture and “decal” artwork for Lunatics Project.

One feature I have wanted to see for a very long time is a better way to organize fonts, so I can group the common fonts I use in Lunatics, or group fonts by broader categories (like handwriting, comic lettering, stencil, display, high-readability, etc).

And in 2023, this finally arrived in Inkscape 1.3!

So I naturally proceeded to categorize my fonts, including a set I’ve selected for use in Lunatics:

Summation

I’m a little disappointed that I didn’t manage to finish any of the personal and professional goals I set for 2023. It would’ve been a nice symbolic gesture to be able to check them off at the New Year. But in fact, I still have some work to do on them all. However, the point was really to put the work in on them, and I feel like I have done that.

I decided to move the goalposts a little, and made myself a set of modified goals for “2024 Q1” (that is, before April):

I dropped the storefront goal, deciding that our existing Gumroad account is adequate. And I decided not to write a commercial Kdenlive book, but instead simply put that energy into “Film Making for Lunatics”, which is on my list for the end of 2024 (not shown).

I’ve carried over the bathroom remodeling and Lunatics S1E01 goals, as well as a more-concrete Virtual Studio goal (just getting it running on the new server). The fourth goal is one I can’t shift from the 2024 goals: getting ready for the total solar eclipse on April 8th!  I’m very optimistic that I can adequately meet these goals by April, so I feel pretty good about this. I guess we’ll see if that’s justified — in about three months!

Happy New Year!