Before I get to masking in MODO here's a couple of tests I did with Octane 2.0, MODO 801 and the MODO Octane plug-in. You can freeze fur in MODO using the Render->Bake Geometry Cache command. All of the volumentric fur is frozen as polyline geometry. You can then render this in Octane like you would render any other mesh. There isn't a fur specific material in Octane yet (I used a blended glossy and diffuse materials) - but maybe that will come. The results are pretty good - but you're missing a fair number of MODO specific shading abilities (texturing along the length of the hair for instance). It's great that you can just click 'render curves' on the mesh item and pick a radius though. You can render any curves (convert them to polylines though, otherwise there's a linear interpolation of the curve) - but Octane doesn't yet recognize the curve radius or twist gradient....or the Curve Start or End channels. It's a good start though. The fur on the torus just uses some image maps from the intertubes for color..same with the dog which is from MODO's stock content.

Another tutorial is up for you (here and on the Sabertooth MODO Videos YouTube channel). I've seen some posts (which I do from time to time) on the public forum which were expressing some confusion on how to achieve some effects in the Shader Tree. I know there's a number of Shader Tree tutorials floating around out there (I recommend 'Shader Tree Fight' if you're interested in the bird's eye view of how the Shader Tree works) but I think we are in need of one that focuses on nothing but masking. So here it is, a video on all the myriad ways to isolate bit and pieces of your scene in the Shader Tree so you can add materials, textures, outputs and shaders to your heart's desire.

Next up I think will be a tutorial on Render Passes....which is one part of MODO that is (shockingly) probably even less understood than the Shader Tree. Happy MODO'ing. : )