Archives for posts with tag: storage management

JupiterFileCache (JFC) is a general purpose, multi location, multi process, thread safe file read & write cache library.

This is a C++ library that allows caching files in remote (network) locations onto the local client to speed up future accesses to these files. This reduces network traffic, but also, more importantly disk access, on the server. Modern network infrastructures are rarely the problem, but a server’s disks often don’t cope well when hundreds of clients read large files, often repeatedly, by querying small pieces of data, all at the same time.

It comes with a SWIG interface to be easily accessible through scripting languages (e.g Python) without the need to use a C++ compiler.

RSL shadeop implementation, modeled after the automatic network cache in DNA research’s 3Delight. It allows caching files like textures, point clouds or other heavy data, on a render farm client, automatically, by simply specifying the cache location with an environment variable, or in the RIB (with an option). In your shaders you can use the cacheFile() shadeop. It should work with any renderer that has an RSL shadeop API but was tested & used in production with Pixar’s PhotoRealistic RenderMan.

Project Page:
Language: C++ Python
Platform: Linux, OS X, Windows
License: Creative Commons 3.0 BY-SA
Sponsor: Jupiter Jazz

Earth allows you to find files across a large network of machines and track disk usage in real time. It consists of a daemon that indexes filesystems in real time and reports all the changes back to a central database. This can then be queried through a simple, yet powerful, web interface. Think of it like Spotlight or Beagle but operating system independent with a central database for multiple machines with a web application that allows novel ways of exploring your data.

Earth is initially focused on managing the explosion of data that occurs in digital visual effects work.

Project Page:
Language: Ruby
Platform: Linux, Windows, OS X, etc
License: The GNU General Public License (GPL) version 2
Sponsor: Rising Sun Pictures