ACES: Overrated or misunderstood?

It's quite often I see a forum thread bashing ACES, almost always for the same reason: aggressive highlight and shadow compression. I've worked with real ACES, fake ACES and no ACES at all, so I figured I'd give my opinion on the matter, even though I'm just a 3D artist.

Introduction: What is ACES?

ACES is a set of colorspaces and a display transforms, but among the 3D visualization community it's most well known for its display transform (tonemapping).

Tonemapping is the process of taking scene linear light data, which goes from 0 into the thousands, and converting it to a range that is pleasing to the human eye. Our eyes are much more sensitive to changes in dark areas than in bright areas. E.g., you notice a slight brightness shift in the shadows and mid tones, but the same linear shift in a bright area is much less noticeable to our eyes. This is what's called a logarithmic response.

So our eyes see a linear world in a non-linear way. Like the world, cameras record light values linearly, and they simulate the human response by applying a tone curve. Rendering works much the same way: it generates linear light values, then applies a tone curve. The ACES display transform is exactly this step.

Here's the thing that finally made tonemapping click for me. Imagine you had a magical display that could output any brightness it wanted, as dim or as blindingly bright as the real scene. On a display like that you wouldn't need a tone curve at all. You'd just look at it and your eyes would compress the range themselves, exactly like they do when you look at the real world. The tonemapping would happen in your eyes, for free.

The only reason we need an artificial curve is that real displays don't get anywhere near bright enough. A screen only puts out a few hundred units of brightness while the real scene runs into the thousands. So we can't just hand your eyes the real range and let them sort it out. Instead we fake it: we squeeze that huge range down into the sliver a display can actually show, and we shape that squeeze around the darker tones our eyes care about most. That shaped squeeze is the tone curve.

Fun fact: Because HDR displays get so much brighter than regular displays, they need way less highlight compression. This is in large part what makes HDR displays look so good. More brightness is simply better at fooling our eyes.

So why does ACES often look bad?

The problem isn't with ACES, it's with our expectations. RAW files from cameras also look horrific most of the time, even with the tone curve applied (which is done automatically when you open it in any RAW editor like Lightroom).

In almost every way, a RAW file opened in Lightroom is the same as the render that rolls out of your render engine. And professional editing is not done with the crappy controls of your camera, it is done with a dedicated piece of software like Lightroom. The same should go for your render.

Earlier I talked about how a theoretical display bright enough to match the real scene wouldn't need tonemapping, because our eyes' logarithmic response would handle the compression automatically. But that log response is only half of what our eyes do. They also adapt locally: lifting the shadows and taming the highlights depending on the region you're looking at, partly in the eye and partly in the brain. It's why you can stand in a dim room with a bright window and still see detail in both at once.

A tone curve only fakes the first half, the global squeeze. And by cramming everything into the narrow range a display can show, it flattens out the very brightness differences that would have triggered that local adaptation. So the local part is exactly what's left missing.

That's where programs like Lightroom come in. The key difference from the post processing controls in your render engine is the local adjustment tools. They put back the last piece, the one our eyes were doing for us all along.

The solution

Photographers and film makers have long figured this out. Filming a dark room prompts you to adjust the exposure to expose for the room. Since linear light values are so enormous, the sky isn't "twice as bright as the room", it's dozens of times brighter than the room. The result might not be clipping, but it's definitely pushed into that compressed highlight range where everything seems hazy and devoid of detail

The trick is to use local adjustments. Something not possible in most simple 3D VFB interfaces. Most highlight and shadow controls in other software simply lift the dark pixels and lower the bright ones, across the whole frame. That flattens everything. Lightroom instead looks at the surrounding pixels, finds the shadowy areas, and lifts those as a region. That's the difference, and that's how photographers turn crappy RAW files into Instagram-worthy masterpieces.

It's a bit funny that being good at rendering means being good at photography, architecture, light transport physics, interior design, and a pile of other fields that aren't technically 3D at all. In my experience it's easier to teach a photographer 3D than to teach a 3D artist photography. (I'm not just here to roast you, I promise.) What really helped me is watching photo editing tutorials for whatever subject I was trying to represent. I also started doing some photography myself. Make sure to shoot in RAW though, because your phone already does a bunch of automatic HDR and local adjustments for the exact reasons I'm talking about in this blog.

Treat the 3D as replicating reality as accurately as you can, and treat the lighting and post-processing as the artistic steps that come afterwards. Different job, different knobs to turn.

Video and real time

These tricks sound great, but how does it relate to real time and video: It doesn't. The main cause of this is that most automatic local adjustment detection algorithms (like the ones in Lightroom) look great in a single frame, but aren't temporally stable.

Video (film and animation) fixes this by using programs like DaVinci Resolve, which provide top tier manual animated masking tools and a host of other tricks, but these controls aren't present in any 3D software I know of (maybe Blender?).

The real time case is even trickier, since you never know where the player is going to look next, it is impossible to know which parts to brighten and which to darken. A blanket approach has been to visually make the sky darker, artificially light the inside of buildings with hidden lights to compensate for the lack of light. But all of these solutions are trying to solve a post processing problem with tricks instead of real solutions.