Over the weekend, we released a beta of the next major agent version. This agent, version 3.x, includes new job upload technology we’ve dubbed Corral. Corral represents a significant improvement in how data is sent to the farm, and we wanted to talk about it in detail so you can understand what benefits it holds.
The most important function of Corral is farm-side caching of uploaded data. This is done on a per-file basis, which affords everyone some notable advantages. The first advantage is that you no longer have to keep your asset files in a fixed location on your drive or network in order for the agent to compare them to what exists in the cache. This means that your files can be moved around on your drive or network, yet still only have to be uploaded once as long as their contents do not change. That’s quite an improvement over our existing Project Cache mechanisms, which are inextricably tied to file location in order to maintain the farm-side cache. The second advantage is that the same file used by multiple users gains the benefits of caching. If one user uploads an asset file with a job that’s, say, an HDRI background from a plugin developer, another user won’t have to upload the same file when they submit their job. Have you ever needed to re-render a scene file before? If so, you’ll quickly realize the third advantage, which is that all of your external asset files already reside on the farm, so only the scene file itself needs to be uploaded with the job. Fourth, files cached in Corral are maintained independently of the storage array used by the render nodes in our farm. This means that you may submit multiple jobs that use common file sets, and not have to have them rendered one at a time due to small changes in one or more files between job runs. Our Project Cache feature had this limitation, as the cached files were both stored on the primary storage array and directly accessed by render nodes. Open and locked files couldn’t be updated, so multiple jobs that used the same Project Cache (but changed a few files between job runs), resulted in jobs having to render one at a time. Lastly, because cache comparison is done per-file, you don’t need to do anything in advance of simply submitting a normal render job to take advantage of it. You no longer need to pre-upload a Project Cache for the data to be stored on the farm side before you submit a job. Just submit the job(s), and let the agent do the work.
Also different in the v3 agents is how they upload job data to the farm. Now, rather than compressing the entire job directory, uploading it, and extracting it, compression is done on-the-fly. This cuts a previously 3 step procedure down to only 1 step. Since the agent is no longer creating a local compressed archive of data, you don’t need to have a lot of available drive space where you have the agent installed. Files are uploaded directly from the job directory, being compressed as they leave your machine. This saves you temporary drive space on your local machine, as well as time during the job upload phase.
Due to the lack of user interaction necessary to take advantage of Corral, it is entirely replacing the current Project Cache feature. V3 agents will not even display a Project Cache tab, as they are simply no longer necessary. You may want to remove any Project Caches you have on your account prior to upgrading your agent.
Corral is a fundamental part of the Pixel Plow agent. It is on by default, so that everyone can benefit from it. Because it saves both you and us bandwidth, we have decided to not charge any fee for the feature. Corral is free to use.
Lastly, it should be noted that the V3 agent is a beta release. If you are working on a project that has tight timelines, then we advise you to continue using the stable agent release for now. If you’d like to take Corral for a spin, beta download links are in the News section of the Home tab or in the About window of current agents, both on the Windows and MacOS platforms.
Let us know how V3 works for you.
