One of the line items on the budget for this Kickstarter is to build a “render cluster”. You might be wondering what that is, and why we might need one.
We’ve thought about several options for handling rendering from Blender. One is to simply rely on our desktop workstations to do the rendering — but that would be very, very slow, even if my system weren’t several years out of date. We could also run Blender network rendering on several different desktop computers, spreading out the load. We could even use the publicly-available renderfarm.fi cluster. However, as exciting as I think renderfarm.fi is, I feel that our using it for such a large (and ultimately commercial) project might border on abusing a public resource.
It would be far better, I think, for us to build our own rendering hardware. And then, when we’re not currently using it for production, we might consider making it available to other free-culture projects as well, to give back to the community rather than always taking. So, that’s why I’ve budgeted to build a cluster. Of course, I also wanted to do it for the least amount of money I could, and so I did a very careful cost-benefit analysis. This is what I came up with.
An Inexpensive 48-Core Rendering Cluster
Our plan is to build what is affectionately known as an “Ikea cluster”. It’s an extraordinarily cheap way to build a high-powered computing cluster — in our case, to handle the enormous, but highly parallel task of rendering animation frames from Blender.
A home computer builder some time back noticed that a standard “Micro ATX” motherboard would fit rather nicely into the drawer slots for an Ikea “Helmer” cabinet. His cluster build looked kind of sloppy, but it was a really enormous savings over buying server case hardware — because the Helmer case cost him only about $30, which is about the price of a typical single motherboard budget-priced tower case built for the purpose.
A later builder did something similar, but he turned the motherboards 90-degrees and installed them directly into the drawers of the same cabinet, which made for a much neater case and better airflow. In both cases, they were able to build a 6-motherboard computing cluster. At the time, the high-end desktop CPU had 4-cores, so they were 24-core clusters.
Technology has moved on a bit, but the Micro-ATX is still a very standard size for desktop computer motherboards, and very affordable ones can be had. After a careful optimization study, I priced a specific set of hardware to make my computing power and cost estimates for this budget and find the best performance/price ratio possible at this time, although of course, the market is constantly fluctuating — but even if these disappear from the market, it will likely be possible to find replacements:
- Motherboard: BIOSTAR A960D+ AM3+ AMD 760G Micro ATX
- CPU: AMD FX-8350 Vishera 4.0GHz AM3+ 125W (8-core)
- RAM:16GB (2 x 8GB) 240-Pin DDR3 1600
There are two options for power supply. One is to put a standard switching PC power supply in each drawer, so each system is independent. This is what the previous builders did. Another alternative, though, is to simply use a DC-DC adapter on each board and provide a single high-powered supply for the whole cluster. This makes
very little difference in cost, but I think it may be more energy efficient and run cooler (because there isn’t a power supply heating up
each drawer). For pricing research, I chose these product examples:
- Per-board adapter: PicoPSU-150-XT 12V DC-DC ATX
- Main power supply: Tripp Lite PR60 DC 60 Amp 120VAC-13.8DC
Likewise, it’s possible to get little PC airflow fans for each drawer, but I’m considering the alternative of simply running a household duct-fan to draw straight from an air-conditioner into the box, pass it over all of the boards and then out the top.When you figure all of this out for a 6-motherboard system with the common power-supply and cooling system, the total cost is around $3000. Which is not that much more than an off-the-shelf commercial laptop computer.
If the project has enough additional funds, of course, we could opt to double or triple the available cluster to get double or triple speed. We’ll have to be careful about whether we’re spending too much on the hardware, though, versus how much we’re paying to artist commissions. But it is true that the hardware will make us more productive by wasting less time waiting on rendering.
About 18X Faster Performance
According to benchmark data, this cluster should be able to do Blender rendering about eighteen times faster than my current desktop. Put another way, it will turn 1-minute of render time on my current desktop system into about 3-seconds of render time. Or turn a render that takes an entire day into about an hour-and-a-half.
That’s a huge improvement if you’re having to consider whether to revise a scene which has a small error in it (and it often happens that you don’t catch an error during modeling, animation, or pre-visualization, but only after you’ve fully rendered it).
It also makes it feasible to consider final renders with more compute-intensive options in Blender. The line-rendering tool Freestyle, for example, is mainly used for rendering still images rather than animation, because it really eats CPU cycles. But as you’ve seen in my prior update, Freestyle really has the potential to improve the look of our characters.
We’ll also likely want to use more compositing, which adds multiple render passes for each render-layer. Again, this is really going to consume rendering time.
Fortunately, rendering an animated film is an extremely parallel task, so that adding extra cores really does speed things up — pretty much linearly. That’s because each frame is really a separate rendering job (there are not many things that can be shared between frames in 3D rendering), and you’re having to render thousands and thousands of frames. So, you’re never likely to have more CPUs than frames, and thus the speed-up will be close to linear with the number of CPU cores you can put on the task.
Putting it Online
Of course, this only really solves the problem for me. In order for other project members to make any use of the cluster, it will be necessary for them to upload files onto it and run rendering processes there. That means we have to provide some kind of access to the internet.
Now we could try to put our Ikea cluster into a data center — but the rental rates on a large enough cabinet would wipe out all of the savings of building this cheap system. It would actually be cheaper to spend on much more expensive rackmount hardware and just rent a 1U or 2U space at a data center in Dallas. Cheaper — but not cheap by a long shot (and yes, I also costed such a 1U or 2U data center solution, but it was not as cost-effective, so I’m not going to explain it here).
There is an alternative, though, which is to simply upgrade the internet service to our project site (i.e. my office), and hook the cluster up to that. Serving directly to the Internet wouldn’t really work for us, but we don’t really have to do that. We currently run a virtual private server (VPS) host for our website, so we have services available to the Internet and can install our own packages as needed. We currently run several free software server packages on this site, including Plone, Mediawiki, Trac, and Subversion. We can install additional software as needed, because while the site is technically shared with other hosting, it looks and acts just like a dedicated machine from our point of view (except for performance limits, obviously).
We can, therefore, use that site as a go-between to share information with our on-site cluster. To do that, though, would require an upgrade. Our site service is asymmetric — we can download data very quickly, but uploading is much slower (most client-side internet installations are like this). It’s not really designed to support servers.
However, for less than what 1U data center hosting would cost us, we can upgrade to a service that will give us at least 2 Mbps upload speeds. That’s still a potential bottleneck, but as long as we are just using it to synchronize data with our public server, it’s workable.
And that’s the currently-proposed solution. Obviously, there are several factors here which are subject to change, and we’ll re-evaluate whether we’ve really got the most cost-effective solution we can find. But the general plan should be the same.
And the Helmer case? Well, the local Ikea does indeed have them in stock ($40 at last check). I took a picture!
