Blog – Many Worlds Productions

July 6, 2026July 6, 2026

Solving failures in AI image generation: we’ve come a long way

The technology of AI image generation has fascinated me from its very beginnings. Join me in exploring some of the breakthroughs that have improved image generation.

The early days of generative AI were pretty exciting. I discovered it myself in November 2022, when I was working on a short film for a weekend competition. For some reason I had written a fantasy script set in medieval-ish times featuring a seamstress and a reluctant Duke. To depict the seamstress’s fantasy life, I had written in visuals of several tapestries depicting battle scenes and the Duke himself.

I don’t know where I thought I would find these tapestries. In fact, I don’t know what I was thinking at all.

It was one of those 72-hour competitions, where we had three days to write, shoot, edit, and deliver a short film, so I didn’t have time to fly out to the Smithsonian or anything. After desperately searching for tapestry images online (of which there are very few), I wandered into the Midjourney website. There I was thrilled to find that I could generate all my tapestries within minutes.

Tapestry depicting the battle at the castle

Tapestry depicting the Duke and his brothers

I couldn’t get the tapestries to look exactly the way I wanted, though. One problem I ran into when generating the images is that the 2022 Midjourney model didn’t handle things like attribute binding or negation very well. No matter how many times I asked it to replace the Duke’s hood with a peasant farmer’s hat, or turn his robe green, the model just wouldn’t do it.

As a side note, in November 2022 generative AI was new enough that other filmmakers asked me where I was able to find such rich tapestries on short notice. (It was also early enough that the competition hadn’t yet banned AI-generated content, but that’s another story altogether.)

The shape of model failures

Midjourney wasn’t alone in this problem. DALL-E and every other image generation model I tried shortly afterward had the same failures.

The failures weren’t that surprising, when we look at the way generative AI is written to work. In their original forms, models couldn’t interpret composition-related instructions like “next to” as in “a cat next to a box,” or negation like “a house with no roof.”

They also struggled with numeration — a prompt asking for “three apples next to two bananas” might produce six apples and one banana, or any combination thereof. And if you ask for red apples next to green bananas, you added attribute binding to the mix, which seemed to confuse the model even more.

Solving the failures

AI model developers definitely noticed these failures, and the years since 2022 have seen a flurry of technical papers around solving these problems specifically. (If you want to explore the technology without heavy reading, this paper and this paper have a lot of images that show the evolution.)

The problem was that image-generation models had not been trained on compositional relationships, a catch-all term for numeration, attribute binding, the relationships inherent in prepositions such as “on,” “under,” “next to,” “behind,” and “in front of”. The same solution could also apply to verbs that imply positioning, as in “a dog chasing a cat” or “a woman pulling a cart.”

Solving these problems wasn’t just a matter of rewiring the model. The training data needed to be looked at again to see how it could help the model figure out these prompts. More images and better data annotation became key to the solution.

Box annotation was a huge boon to the solution. A box around each object makes it possible to count them, which goes a long way toward solving numeration.

Photo of people and cars with box annotation

More robust image labels also helped solve positioning problems. With box annotation in place, captions and labels help the model learn what various prepositions and verbs mean with regard to spatial composition.

“A basket on top of a box”
“A box under a basket”
“A basket and box next to a window”

In the case of negation, the model had trouble when one object was always expected to go with another, for example a house and a roof, or cheese with pizza. So the model would need images of houses with no roofs and pizza with no cheese as part of its training dataset, and also special negation labels.

The “no roof” label gives the model something to work with, even if the prompt is worded differently, as in “a house with its roof removed.”

It’s a lot better now

All the foundation models — DALL-E, Midjourney, and even specialty models like Leonardo — all incorporate compositional relationships into their thinking. The models themselves now work so well that if you run into these failures, it’s likely an issue with the training data.

The challenge of getting high-quality, well-annotated data for training is a topic all by itself, for another time.

June 27, 2026June 27, 2026

What is an AI agent?

There’s been a lot of talk about “AI agents” lately. But what does this really mean?

The official definition of AI agent, according to Google, is a software system that uses AI to pursue goals and complete tasks on behalf of users. That’s still pretty general.

To get a better understanding of what an agent is, let’s take a look at a familiar example: ChatGPT.

A very familiar agent

In the olden days, and by that I mean 2022, if you typed a prompt into ChatGPT, it sent that prompt pretty much directly to the GPT AI model and then served you back the response. At the time, ChatGPT was really just a prompt/response interface for GPT.

Then DALL-E was released as an image generation model, and you could type in a prompt to get DALL-E to make an image for you. ChatGPT, at that time, was still little more than a traffic cop; you’d type in a text or image request, and it would route the request to either GPT or DALL-E depending on whether the prompt started with “image of…”

Keep in mind that at the time, each AI model was confined to what it learned during its training cycle, which was already long over by the time you typed in your prompt. If it didn’t have the answer in its training, the model might make something up based on what it did know, a practice we call “hallucinating”. And the hallucinations could be pretty wild.

But now, in 2026, ChatGPT works differently. When you type in a prompt, ChatGPT takes a look at what your intent is, and might do any of a number of things depending on what it thinks you want. For example, if you type in “images of Mount Rushmore,” ChatGPT might just go find some existing photos and serve those up as the response, rather than creating something new. This saves tokens, and gives recognition to creators who are already in the business of taking great photos of landmarks.

These photos weren’t part of a model’s training. They are simply what the agent went out and found, on the fly, after reading your prompt.

Another example is a prompt like, “What is the architecture like in the Akihabara neighborhood of Tokyo?” The AI agent might use Google Maps to find photos from that neighborhood, then feed those photos to an AI model and tell it to analyze the architecture in the photos. The model already knows how to analyze photos, based on its training. The difference is that an AI agent found the photos and served them to the model itself, rather than you having to go find photos and the model yourself.

In this regard, ChatGPT now functions more like an agent than an AI model.

We also see similar behavior in Google Search. When you enter AI Mode, Google doesn’t just perform as a search engine anymore; it acts as an agent to go find you the answer, then summarizes it for you.

Agent-ness is a spectrum

Is ChatGPT actually an “agent” now? That’s a great question. The thing is, the agentic quality of a piece of software is a spectrum, ranging from agents that do everything for you to mildly assistive agent-like interfaces that require you to do 99% of the work.

For example, a lot of websites now have AI chat agents that will answer your questions without needing to involve a human. The least agentic ones require you to do all the heavy lifting by phrasing things a certain way, or by choosing from a menu. The most agentic ones can take a human-worded question like, “I’m looking for a toaster. I never make bagels but I do toast thick bread, sometimes with cheese on it,” and figure out that what you really need is a toaster oven.

Hopefully these examples answer the question “What is an AI agent?” to your satisfaction, so you can recognize an agent when you see one. And always keep in mind that agents are software, not people, and are prone to making mistakes, sometimes even more than humans are.

No AI was used in writing this post.

June 1, 2026June 1, 2026

Why does it work that way? Why software makes no sense

A confused user looks at a dev writing software

Back in the 1980s, when I was a developer at a software company, I was walking a user through a feature that did not work intuitively at all. After a particularly frustrating session, with me struggling to explain and the user struggling to understand, the user finally asked:

“Why does it work that way?”

The answer to that question is the same today as it was then: it works that way because some developer somewhere decided that’s the way it should work.

Users are often surprised to hear that there is no higher authority, guild, union, or standards organization that determines how software should be designed. There is no Software Council that hands down rulings about what user interface elements must be present or how a program should behave. There are guidelines, best practices, and conventions, and sometimes developers even follow them. But for all software, it comes down to somebody making a decision.

And that person, being human, may be tired from months or years of development, frustrated by changing technology that forces rewrites, or even scared by some tech article that predicts no one will want their product anymore (which usually turns out to be not true, but they won’t find that out until three months later). They might have been distracted by some pretty, shiny thing that the industry says everyone wants, but in practice, users hate it.

That’s why the software works that way.

Nowadays we have UX (User Experience) designers, but that role didn’t become a thing until the 1990s. Before that, any user interface was pretty much built by developers. And even today, for some companies, the ‘UX team’ is a dev with a Figma license who draws pastels of fruit baskets on the weekend.

But if that’s the case, why don’t developers make more of an effort to understand users, and make the software more useful and easier to use?

Reason #1: Developers and users are two different species

A developer comes at a software problem from a completely different perspective than a user. The developer looks at the logic of the data being passed around, where and how it’s going to get stored and retrieved. The user looks at it from the viewpoint of how to put in real-life data, such as a customer invoice or sales transactions, and later extract some kind of useful compilation or report.

In other words, the user is interested in the beginning and end — putting in the data, and getting it out in another way — while the developer is interested in the middle part: the storage, retrieval, and manipulation of the data.

Reason #2: Users are ‘stupid’

My first job as a developer was at a small startup, where every member of the development team was required to spend time doing phone support for the software we had written.

Some devs, like myself, came away with a deeper understanding of how software design needs to meet the user where they are. Menu options needed to make sense, and things on the screen needed to resemble, as much as possible, where they sit on real-world items like pieces of paper. This was especially important in the 1980s, when them new-fangled computers were still a bit scary, but it still holds now.

For me, the customer support experience even sparked an interest in documentation, which is eventually where I took my career. I found it far more rewarding to explain how software works and help people get the results they wanted than I ever found writing the software itself.

But others forced to endure customer support came away with a different attitude — they were convinced their knowledge made them superior, and that users complain and have problems with software because they’re stupid.

Yes, there are developers, some of them writing software right this moment that you will end up using someday, who genuinely think you are stupid.

Reason #3: Users should ‘learn programming’

When I first started writing documentation, I needed to understand how a particular piece of accounting software worked. Usually I could figure things out on my own — if the interface didn’t make it obvious, I could poke around the screen, even read the source code in a pinch.

But this one entry screen completely baffled me, so I went to talk to the developer. (I should add that at some software companies, the documentation team has next to zero access to developers, which can be super annoying. At this place, however, I was able to get a few minutes with them when needed.)

It turned out that the entry screen had been designed to make data storage and access easy and quick, with no regard for how hard it would be for a normal user to actually use it. The structure made perfect sense from the developer’s perspective, but was pretty useless from the user’s perspective.

When I asked the developer how a user was supposed to understand all this, he replied that if users understood how it worked under the hood, they wouldn’t need documentation. How dare they not understand data structures, arrays, and do/while loops!

I had to pinch the bridge of my nose for a second to keep from spitting out a rather rude reply.

As a simple parallel, it would be like setting up an SQL database with employee information, and then telling the user, an accountant by trade, they could go ahead and figure out tax withholding themselves with some clever JOINs. Then having disdain for them when they can’t figure it out.

When I asked this dev how I was supposed to explain database structure and query language in the documentation, he shrugged, conveniently forgetting that these subjects are ordinarily taught at university level, as part of the second or third year of a programming degree. Yeah, sure, I’ll just slip in these advanced topics on page 34 of the User Manual, right after we explain what the up and down arrows do.

So no, Mr. Disdainful Dev, the disdain should be aimed at you, for not making the ‘user experience’ something that a user could actually experience, or even want to experience.

It’s a rough world out there

There isn’t much you, as a user, can do about disdainful devs except demand better software and refuse to buy products that provide a poor user experience. Market pressure works, eventually.

But perhaps it helps to know that when software behaves in a baffling, frustrating, or seemingly irrational way, there usually isn’t some deep reason behind it.

A developer somewhere, sitting in a dark room, listening to a heavy metal playlist, sipping a ginger ale, and munching on a bag of Cheetos, decided that’s how the software should work.

So if you’re frustrated, your frustration is valid. Just know it’s not you — it’s the software.

And if you need some decent documentation, I’ve written dozens of excellent user manuals, without ever so much as hinting that the user is the problem. Because they never are.

October 29, 2023

Commercials for local businesses

Many Worlds Productions is fortunate enough to be part of a thriving community in the historic Bywater neighborhood of New Orleans. We love being in the midst of all the locally owned restaurants, coffee houses, bars, and theaters, and within walking distance of some of the most amazing art installations the city has to offer.

One of these local gems is Frady’s, an old-fashioned grill and convenience store run by brother-and-sister team Kirk and Kerry Frady. Poboys and breakfast are cooked fresh and assembled in their tiny kitchen while you wait. The hot bar offers a different New Orleans classic every day of the week (red beans and rice on Mondays, y’all). All food is to-go, but many patrons elect to sit at one of the outside tables and chat with friends and neighbors while they enjoy their meal.

We’re big fans of their food, so we were more than happy to make this commercial for Frady’s social media. The commercial package includes several photos and videos for posting over a period of weeks, to keep Frady’s fans engaged and coming back for more.

Watching the commercial makes our mouths water, and we hope it does the same for you. If you’re ever in the area, stop in for a delicious roast beef poboy or the classic Grumpy Old Man breakfast, and be sure to say Hi to Kirk and Kerry!

June 10, 2023

It’s a wrap! Storyboards with Unreal Engine and MetaHumans

We just did a shoot for a short horror film using our virtual production setup, shooting about 15 minutes of script over 15 hours of production. We’re making this film for the Louisiana Film Prize, a short film competition taking place in October 2023.

One of the things that made the shoot go so smoothly was the fact that we made a storyboard ahead of time using MetaHumans. Most of the cast and crew was new to virtual production, and the storyboard gave them insights into what we’d be shooting and how it would work. We were also able to use the same camera setups from Unreal Engine to get the backgrounds up in seconds, and to pose the actors according to the storyboard.

Check out our video about the process:

One of the best aspects of doing storyboards and production with Unreal Engine and MetaHumans is the way the cast and crew responded. Everyone was excited to be part of this new technology, and many came up with creative ideas on the fly. Because it’s so easy to adjust the background, lights, and camera angle, we were able to try different things and get a lot of footage for the editor to play with.

We’ll have more as we get into post and complete the film.

May 17, 2023

Table read!

One of the best ways to get cast and crew excited about a film is to host a table read, where we read through the script together. It’s great for learning what works in the script (and what doesn’t), and in getting everyone on board with what you’re looking to accomplish.

Left to right: Kay (lighting), Yvonne (PA), Jarrod (actor), Lizzy (Script Supervisor)

Last week we held a table read at our home for our upcoming short film “The Last Influencer Standing,” a comedic horror story of influencers battling to the death for one million followers. About half the cast and crew were able to attend, and we assigned some crew to read for the cast that wasn’t there.

Terese (lead actor), Michele (director), Jen (costumes)

It also gave us an opportunity to do a quick costume fitting for the lead actor, and for the director to model the T-shirts the “influencers” would wear during the shoot.

It was great to see everyone have such a great time, and also to hear the laughs come at all the expected spots.

Principal photography is scheduled for next month. We can’t wait!

January 28, 2023January 28, 2023

Conquering banding and artifacts with rear projection

When we began our quest to set up a virtual production studio using rear projection, we thought it would be as simple as setting up a projector and screen, piping out some Unreal Engine graphics to the projector, putting actors in front of the screen, lighting the actors to match the projection, and shooting them with a decent video camera. We were right on some counts, but not all.

One of the issues we battled right away was artifacts when we shot the screen. When shooting, we got rolling bands on the recording. The bands appeared only on the projected background, not on the actors. And the bands weren’t visible to the naked eye, just the camera.

The artifacts resembled the type of banding one can get with live shoots when the frequency of the lights used is out of sync with the camera’s shutter speed. This led us to suspect that there was some kind of synchronization issue between the camera and the projector. This ended up being true, but not for the reason we originally thought.

Key takeaway: When shooting a rear projection screen, you need a camera that shoots at true 24 fps.

Banding in live shoots

To get a better handle on why we got these artifacts, let’s look at a parallel problem: banding in recorded video of live actors and sets.

Physical lights don’t emit light continuously–they are actually “flickering” at a very fast rate. In the USA and other countries that use alternating current (AC), lights flicker at a rate of 60Hz, meaning they turn on and off 60 times per second. In other countries that use direct current, the rate is 50Hz.

When you shoot a scene lit by such lights, you need to be aware of your camera’s shutter speed, the rate at which its shutter opens and closes to gather light and color information for recording. If your camera’s shutter speed is at odds with the light’s flicker rate, this will cause banding in your recording.

In the USA, for example, you’ll want to shoot with a shutter speed that’s a multiple of 1/60 (opens and closes 60 times per second) to jive with the flickering lights. This means that with a shutter speed of 1/60, 1/120, or 1/180, you won’t get any banding. The video below shows and explains this phenomenon very well.

If you’re shooting at 24fps, the ideal shutter speed for a cinematic look is double that number, or 1/48. But since 1/48 isn’t a multiple of 1/60, you might get banding at that shutter speed. Bumping up the shutter speed to 1/60 will stay close to the “cinematic look” shutter speed of 1/48 while also eliminating banding in your shots.

This advice works great for live shoots, but shooting a projector isn’t exactly the same thing. Still, we were able to apply that lesson to our problem.

Banding in projector shoots

In the same way a light flickers multiple times per second, a projector projects light and color at a particular rate. A common rate for projectors is 120Hz, meaning the screen refreshes 120 times per second.

Theoretically speaking, if the projector has a rate of 120Hz, one should be able to set the shutter speed to 1/60 or 1/120 and get a solid image. However, we didn’t find that to be the case.

For the past couple of years we’ve been using a Sony a6500 camera for our live film shoots, and we’ve been able to get some great quality with it. For example, we shot this award-winning comedy short with the Sony a6500.

But when we set up our virtual production studio and attempted to shoot the projection screen with the Sony a6500, we got rolling bands at every shutter speed. At 1/40 the bands were the least present and visible. We were able to reduce the rolling lines by displaying an Unreal Engine scene that was very bright or dark, but they were still always there.

Scene from “A Common Thread”, with actor against a projected background, using Sony a6500 to shoot at a shutter speed of 1/40. The rolling bands are minimized, but still very visible at high resolutions. The gray bars point up a couple of the more visible rolling bands in this scene.

To remove the lines, we tried everything–switching to 30 fps and 60 fps, setting the ISO to every possible number, using lights with different frequencies to light the actors (in case light spillage was the issue), and so on. We tried several different projectors, switching between LCD, DLP, and Laser, thinking that maybe the method of generating light and color was responsible for frequency issues. It didn’t matter what type of projector we used, the lines were still there.

(DLP, LCD, and Laser are all different projector bulb technologies. This link explains some of them. In our case, it didn’t matter–they all produced banding with the Sony a6500.)

On the Unreal Engine side, we tried pumping out imagery at different frame rates. We even piped live footage out to the screen, and recorded that. No matter what we tried, the rolling bands remained.

It’s the camera, not the projector!

Then we had a brainstorm: we invited a couple of cinematographer friends over to try out the virtual production setup with their cameras. One friend has a Lumix S5, and the other a Blackmagic Design Pocket Cinema Camera 6K. And…

Both their cameras worked! No banding when shooting the projection screen at a specific shutter speed. Just a crystal-clear image.

But why? When I say the Lumix and Blackmagic cameras worked at a particular shutter speed, I’m not being precise in my terminology. Both the Lumix S5 and Blackmagic DPCC 6K have settings for the shutter angle, not speed.

The term shutter angle is closely tied to the shutter speed for a particular frame rate–when you change any one of the three, at least one of the others changes. Some cameras have a setting for shutter angle, while others have a setting for shutter speed, and they’re generally thought of as being the same thing, just expressed a different way.

When each of the friends’ cameras were set to a shutter angle of 144 at a frame rate of 24 fps, which calculates to a shutter speed of 1/60, the rolling bands completely disappeared.

So why didn’t this work with the Sony a6500? After much digging, I discovered that the setting of 24 fps on the Sony actually means 23.967 fps. The technical reasons for forcing cameras to 23.967 fps rather than 24 are beyond the scope of this blog post, but suffice it to say that the reason is historical and has to do with the invention of video, and that Sony apparently likes to substitute 23.967 for 24 fps in all their cameras. For live shoots, it’s usually not a big deal–23.967 and 24 fps look very nearly the same. But when you’re shooting a projected image, you’re going to get artifacts no matter what you do.

With the Sony recording at 23.967, something is thrown off when the camera tries to shoot the 120Hz screen. Either the supposed shutter speed of 1/60 is not a true 1/60, or it needs a shutter angle that isn’t an integer like 144 to get true 1/60 and can’t get there. All I know is that with all other things being equal, the true 24 fps cameras did the job perfectly, and the Sony failed.

Our new camera

This means we’re going to need a new camera. We’re seriously considering the Blackmagic DPCC 6K, but we’re currently weighing it against Blackmagic’s 4K version for portability, battery life, and price. We’ll let you know when we make a decision.

In the meantime, my camera-rich friends are coming over to help us shoot some videos for our portfolio, so we can start doing gigs in the coming months. Stay tuned!

December 3, 2022December 11, 2022

Virtual production on a budget

Earlier this year, Many Worlds Productions embarked on new adventure called virtual production. Virtual production is any filmmaking technique that seamlessly combines virtual and physical worlds in real time. You can think of it as any technique that cuts out post-production steps by mixing in the virtual world with the physical world during production, right there at the shoot. The production techniques used on the Disney+ series The Mandalorian are a classic example of virtual production.

There have been lots of filmmaking techniques over the years that combine the virtual and physical, but virtual production takes things a step further by eliminating that post-production step. And while budgets for The Mandalorian run into six figures per season, there are much less expensive ways to do virtual production.

We decided to start with a relatively inexpensive option: in-camera VFX (ICVFX) with rear projection. This technique is essentially a replacement for green screen. Instead of shooting an actor against a green screen and compositing the background in during post-production, the actor is shot against a screen showing the background. The background comes out of Unreal Engine live, meaning we can reposition it at any time, and compose our shots on the fly.

Doing rear projection live in-camera also eliminates many of the challenges of compositing green screen footage–color spill, rough or mushy-looking edges around the live talent, green reflections on shiny objects, not to mention the basic challenge of lighting the green screen itself for an evenly distributed tone.

In the image above, for example, you can see the fine hairs on the actors’ heads against the bright window background. With green screen, these hairs wouldn’t show up, or might even disappear and reappear from frame to frame. ICVFX eliminates these issues.

Improving on historical rear projection

The grandfather of ICVFX is rear projection with pre-filmed footage, where a video or film is projected behind the actor during the action. One of the most famous uses of this classic technique was in the classic film The Wizard of Oz (1939), where the crew made a fake film of a cyclone ahead of time, and projected it behind the actors during the shoot.

This method has strict limitations, the most severe being that any movement of the camera upsets the perspective and breaks the realism. For example, if you look closely at the video above at around 1:40, you can see the ground immediately behind the fence sliding along in a strange manner as the camera pans to the right. It looks like the crew moved the projector to the right to move along with the camera and keep the distant tornado at the correct perspective, at the expense of realism for items in the foreground.

In 1939 these visuals blew everyone’s minds. Even today, the effect holds up pretty well–I watched this film at least a dozen times before noticing this problem–but the film was originally shown in theaters and then TV, where viewers couldn’t pause or rewind. But today’s viewers are a lot more savvy, and a situation like that would never fly on today’s streaming services.

Another limitation with traditional rear projection is that the projected film is what it is, and can’t be altered during the shoot. If the perspective of the projected image is just a bit off, you just have to deal with it.

When you use a background coming live out of Unreal Engine rather than a pre-filmed or pre-rendered background, there are two distinct advantages:

The background can be moved around or updated on the fly before shooting, so you can get the background you want. You can also blur it, lighten or darken it, change the sun’s position for different times of day, add fog, or do any manner of adjustments in real time, just before hitting the Record button on the physical camera.
You can connect up the physical camera’s position with the position of the Unreal Engine camera so that any movement of the physical camera during the shot will update the background to reflect the correct perspective. This technique, called camera tracking, opens up a whole world of possibilities. We haven’t explored this option yet, but we plan to in the near future.

Below is an account of our journey with rear projection out of Unreal Engine, when we used it to produce the short film A Common Thread for a three-day film competition.

For future projects, we’re doing a lot of test shots with backgrounds we found in the Unreal Marketplace, giving us the ability to put our actors in an office, a hospital, a snowy landscape, a spaceship, the Moon, or anywhere we like, all without leaving the comfort of our studio.

Stay tuned for more virtual production adventures!