OpenNI + OpenCV + PCL + Mountain Lion

This is a setup guide for getting a Kinect Sensor to work partially with OpenCV and PCL at the same time under Mountain Lion. THIS GUIDE USES THE NEWLY OUTDATED VERSIONS OF THESE FRAMEWORKS. IT WILL BE UPDATED TO KEEP UP. Keep in mind that visualization with VTK(PCLs default visualization kit) and its own OpenNI module will not function properly or at all. As a workaround I am using the following pipeline:

  1. Capture with OpenNI module of OpenCV
    • Capture PointCloud maps and BGR color maps, and others if needed
  2. Apply OpenCV functions if needed
    • Calibration
    • Feature extraction
    • etc.
  3. Pipe OpenCV matrices to pcl::<PointCloud> types
  4. Apply PCL functions if needed
    • Filters
    • Normal estimation
    • Registration
    • etc.
  5. Visualize with OpenGL

Disclaimer: I have a fine arts degree, not computer science. Kindly point my mistakes, redundancies, my overall stupidity, but most importantly your comments and improvements to muratg[at]gmail if you would like to. Thank you.

Before anything is installed, it is best to clean the system from any remnants of previous installed packages. Third party installations of MacPorts, OpenNi, QHull, Flann, Boost, Eigen, VTK, GLEW, GLUT, OpenCV, PCL, etc has to be removed for a smooth setup. Please take care not to accidentally uninstall any system packages.

Homebrew

This guide uses the Homebrew package manager. Please install its dependencies before anything else and install itself right after.

Homebrew will throw warnings and errors during use. Please take these into account before continuing to avoid bigger problems later on.

XCode dependency

XCode or at least XCode Command Line Tools is required. XCode is now available as a free download at the Mac App Store. XCode itself is a heavy piece of software, but fortunately Apple made the Command Line Tools available independent of Xcode. It can be downloaded from https://developer.apple.com/downloads/index.action with a free Apple Developer Connection account.

Install Homebrew

Installing Homebrew is very simple, just copy and paste the following command in a terminal window running bash:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

You can also check their homepage http://mxcl.github.io/homebrew/ for any changes to install command in the future or for more information.

If you have MacPorts it is best to get rid of everything it installed and itself before installing Homebrew. Following listing will first uninstall everything installed by MacPorts and uninstall MacPorts itself:

sudo port -fp uninstall installed
sudo rm -rf \
/opt/local \
/Applications/DarwinPorts \
/Applications/MacPorts \
/Library/LaunchDaemons/org.macports.* \
/Library/Receipts/DarwinPorts*.pkg \
/Library/Receipts/MacPorts*.pkg \
/Library/StartupItems/DarwinPortsStartup \
/Library/Tcl/darwinports1.0 \
/Library/Tcl/macports1.0 \
~/.macports

You can also check MacPorts official uninstall documentation for changes at http://guide.macports.org/chunked/installing.macports.uninstalling.html.

If you have installed anything in /usr/local/* it is best to take a prior look in there and make sure that what Homebrew is about to install won't conflict your software development. In this case if there are any remnants of OpenNI, OpenCV, PCL, and their dependencies, they have to be cleared for a clean install. For example, following command will display all files in the system that have pcl in their name:

sudo find / -name "*pcl*"

Slow network mounts and USB drives will cause find command to perform slow. Each line of the output has to be checked and deleted if it belongs to PCL.

After Homebrew succesfully installs brew doctor command checks the system and points out any current and potential problems. It helps to run brew doctor once in a while to keep an eye on the development environment.

Homebrew install formulas are kept up-to-date rigorously to keep up with daily updated open source software and to iron out bugs. Running brew update will synchronize the Homebrew package installer ruby scripts with their GitHub repository. Keep in mind that update command will only update the installer scripts and not the already installed packages. brew upgrade will upgrade all packages installed and their dependencies. upgrade command might break the developer environment due to changing library files or other internal calls.

Homebrew is very capable, please check its GitHub page at https://github.com/mxcl/homebrew/wiki for a lot more.

Install OpenNI

Homebrew itself does not have OpenNI installer scripts. However, it is possible to tap to repositories that do.

Following command will tap the local Homebrew installation to download an OpenNI installer repository:

brew tap totakke/openni

Documentation specific to this repository can be found at https://github.com/totakke/homebrew-openni.

After the installer scripts are downloaded, install OpenNI with the following commands:

brew install openni
brew install sensor-kinect
brew install nite

Please run these one by one and keep in mind that openni package will fail to install the first time if you don't have Java Developer Kit installed. If a dialog box asking if you wish to install JDK appears, please say yes and try brew install openni again before continuing with the other lines.

If another ONI compatible depth camera will be used other than Kinect, and this includes the PrimeSense Carmine 1.08 too which is identical to Kinect, please run brew install sensor instead of sensor-kinect. Unfortunately, both Kinect and another ONI compatible sensor cannot be used with OpenNI simultaneously.

To test the installation and your device, copy and paste the following listing into terminal:

cd /usr/local/Cellar/openni/unstable-1.5.4.0/sample/Bin/x64-Release
./Sample-NiSimpleViewer

You may have to change the unstable-1.5.4.0 to the version you have installed.

If you have OpenKinect it is best to get rid of it prior to installing OpenNI since they will both attempt to use the same libusb library and each have different libusb libraries. Please remove libfreenect with its libusb properly.

Install OpenCV

OpenCV installer script is available in the official Homebrew repository, however it is located in the science section. So first tap in with:

brew tap homebrew/science

Homebrew does have a solid formula to install OpenCV within its science tap. However, the formula won't find the OpenNI directories installed by the previous openni tap by default. Luckily, a commit request at GitHub solves this issue.

Until this commit is merged to the master branch, you have to open /usr/local/Library/Taps/homebrew-science/opencv.rb and replaces its contents with the contents of the commit request at https://github.com/Homebrew/homebrew-science/pull/224/commits. It is also best to make sure that ffmpeg is not installed in the system prior to installing OpenCV. Alternatively you can add the following to the args within def install block to to both disable ffmpeg support and to build the OpenCV world library:

-DWITH_FFMPEG=OFF

Following listing will install OpenCV and its dependencies:

brew install openexr
brew install opencv --with-eigen --with-libtiff --with-qt --with-tbb --with-openni 

This will attempt to compile OpenCV with OpenCL if OpenCL is available. Unfortunately, the script doesn't compile the examples so there is no way to test if OpenCV functions before actual development.

If you already have OpenCV already installed, remove each and every file that belongs to it and its dependencies. If you used an installer package, its "bill of materials" or bom file is located in /var/db/receipts/. You can see all the files installed by the installer package with the following command:

lsbom -l /var/db/receipts/*opencv*.bom

Please note that all the paths are relative. It is best to check the entire output of lsbom before proceeding with:

lsbom -fls /var/db/receipts/*opencv*.bom | (cd /; sudo xargs rm)
lsbom -dls /var/db/receipts/*opencv*.bom | (cd /; sudo xargs rmdir -p)
sudo rm *opencv*.bom *opencv*.plist

Listing above will first delete the files, then the empty directories, and finally the bom and plist files in the receipts folder. Replace *opencv* with the actual package name.

If the previous OpenCV was compiled from source, go to its build folder and run make uninstall. If the cmake generated files are still in place make uninstall will run smoothly. If not, or if the build folder doesn't exist, the previous version of OpenCV will have to be built again for a clean uninstall. Following listing will compile OpenCV and uninstall it from the system:

cmake ..
make
make uninstall

Install PCL

Official Homebrew repository does not have a formula to install PCL, and regrettably the formula PCL website provides is out of date and not updated as often as it should. As a result, the best way to install PCL is to compile it from source.

Following listing will clone the PCL git repository for the latest unstable version:

mkdir ~/source
cd ~/source
git clone https://github.com/PointCloudLibrary/pcl pcl-trunk

Following listing will prepare the build folder and compile PCL:

mkdir build
cd build
cmake -DOPENNI_INCLUDE_DIR=/usr/local/include/ni -DGLEW_INCLUDE_DIR=/usr/X11R6/lib -DCMAKE_BUILD_TYPE=Release ..
make

Although the arguments passed in to cmake should fix most errors and warnings, make might throw other errors and these have to be fixed before proceeeding. Once PCL is compiled, install it with

make install

It is a good practice to keep the build folder intact for future proofing. If you are running out of precious disk space, run make clean instead of removing it altogether. This way you will still be able run make uninstall smoothly and re-compile if needed.

If you installed PCL and its dependencies before from packages, you will have to remove all of it before compiling the new PCL. Packages found at http://pointclouds.org/downloads/macosx.html are prepared with MacPorts, and the system should be clean after uninstalling MacPorts. However, some files might have been left over. System can be checked for any left overs with the first line of the folloing listing:

sudo find / -name "*pcl*"
sudo find / -name "*pcl*" -exec rm -i {} \;

Second line of the above listing will ask the user each time to delete a found file. Please consider that not all files in your system that contain the string pcl actually belong to PCL. Read twice, delete once. Keep in mind that the command above is case sensitive and it will only look for "pcl" and not "PCL". Previous installer might have installed PCL under a different name such as "PointClouds" as well. A through search is required to make sure every remnant of PCL and its dependencies are uninstalled.

The above steps and instructions should create the development environment for the aforementioned pipeline. However, the versions of the software change daily and independent from each other. It is very likely that one package may break the installation or functionality of another over time. Fortunately, developers of these packages are very dedicated and attentive that a broken installer might be fixed in a very small amount of time.

Reading in an OBJ sequence in Maya.. w/o importing

This will be a quick little update. Everyone knows each 3D package has their strengths and weaknesses. Like Maya is super flexible but horrible with simulations, and Houdini is amazing with simulations but you can pretty much forget about UV's and keyframe character animation of anything more than 2 bones. So artists end up transferring assets from one package to another one. For instance, sending an obj sequence from Houdini to Maya can become a horrendous task. Houdini reads and writes files without saving them in its own memory or scene files which boosts its performance. In the mean while Maya needs each obj file to be absorbed into its memory and scene file which can easily bump the size of a scene to GBs. Here is a little 2 liner for you to make Maya read those obj files per frame and not absorb them:

string $frame = ( "/path/to/the/obj/sequence/" + `currentTime -query` + ".obj" ); file -loadReference "sphereRN" -type "OBJ" -options "mo=0" $frame;

This assumes that the name of the Reference is sphereRN and the obj sequence is not padded with zeroes. Below is a sample, you will have to open the expression editor, find the expression and edit the path to your extracted folder.

QUICK UPDATE

This is long over due.

Turns out this method will fail when used with batch render as pointed out by Jacob Niedzwiecki.

Simply put, root of the problem is that the currentTime function cannot query the frame number during a batch or command line render. When set by the expression editor to be evaluated always, currentTime looks at the GUI time slider for input.

Thanks to Jacob for pointing this error out :) and also for finding a very elegant and simple solution for it as well. What I had in mind for a fix was overly complicated.

What follows is his solution to the problem. But first a little explanation on why this solution actually works.

Maya allows for scripts to run before render jobs, before frames, after frames, and after render jobs. These can be set at the Render Globals window. Parameter names are Pre render MEL, Pre render frame MEL, and so on. These parameters are MEL scripts that are set to run at specified times during a render or a batch render or a command line render.

When the same script is set to be evaluated always by the expression editor and also as a Pre render frame MEL script, it will update both the viewport and the renderer with the correct file name for the frame.

So to iterate again, the solution is to copy and paste the exact same script to Pre render frame MEL parameter in the Render Globals window.

Ping Graph in Maya

Here is a script I wrote that runs under Maya Python terminal. It checks the time the workstation spends to connect to about 20 webpages, and updates a bar graph that it creates real-time.

I implemented multi-threading to the script which Maya Python supports to a point. That point is using maya.cmds library which includes all the maya commands with transformations and such in threads.

So I had the Python script to do its trick with multi-threading, then stored the results to be executed thru MEL in strings. And piped those strings back into Maya's MEL commandPort via the Python telnet module.

Script is not fine tuned and it is not bug free, but it works. It also may only work on a Mac station since I used the ping command built into Mac OS X and there are slight differences between systems.

download ping.py

MEL2py conversion

mel2py-random

I have been converting all the MEL procedures from the matrix assignment into Python. It needs a lot of debugging in the parts that create the shaders and groups, but the part that creates the geometry works.

Notice the different random patterns generated with the Python.random module compared to MEL random function.

Attaching archives during render time with -addsphereRI

This project was about creating and editing RenderMan attributes in Maya. preshapemel attribute was imported to the objects in the scene to switch the points of the actual object with a rib archive or a simple sphere before prman takes the scene over. The interface is created with the help of an rman script which does the heavy lifting of MEL UI programming.

First video on the right is the model without alterations, as seen in the Maya viewport. Second video is with the injected spheres, and the third one is with the custom rib archives.

xxxUI.rman script generates the xxxUI.mel script which is the graphical user interface for the newly created keyable parameters. xxxRI.mel is the actual script file that contains the backend of the replacement process. It calls the procedure in xxxUI.mel file to query the interface and pick up the values. Below are the complete scripts

UPDATE

I added a few more controls to the interface as well as centralizing them on a null (yes they are called nulls, not locators). New controls include the display of the original surface in the render, number of points to be used for injection, and some more experimental (read non-functioning) channels. The reason I wanted everything to be controlled from a null is because there may be more than one object in the scene where an artist might want to slap these injections on.

I have been experimenting with blobby surfaces as well as converting the vertices to particles so that they can generate curves, like a motion trail. This part of the code needs a lot of work, and I also have to find an elegant way to make the converted particles follow the vertices.

NOTE: If you need to delete the rman parameters from your scene and start over and don't have a clean scene file in hand, save your scene file as Maya ascii (it will hang maya for a while if you are on unix), search for all the lines that include "rman" and delete them. RenderMan UI is very buggy in terms of refreshing. Also if you used the script, there should be expressions attached to each shape node linking them to the control null, you will have to go thru each shape node and break the connections manually before removing the RenderMan attributes from the attribute editor. Otherwise you will encounter an epic bug that will force you use the ascii method.