Tile Genetics

This is a continuation of an earlier post which was some kind of introduction to Wang tiles. These tiles allow you to cover an infinite plane using a small set, and you can do it in a way that makes repeating patterns a lot less noticeable.

I was recommending everyone to learn about them because there are many problems in procedural generation that lend very well to aperiodic tiling. But the approach in vanilla Wang tiles was only the beginning  This same principle can be applied in many different ways. For instance, here is a brilliant approach to doing rooms using what the author has called Herringbone Wang Tiles. Here you can see how they were used to generate a system or corridors and rooms.

You can find a description of the method here.

There is also a quite popular improvement over classic Wang tiles which uses colored corners instead of colored edges. It results in smaller sets, more diversity and of course their main claim, which is to solve the "corner problem" in Wang tiles. This problem arises from the fact Wang tiles constrains which tiles can be placed along the edges, but says nothing about the corners, that is, the diagonal directions: NW, NE, SE and SW. This often creates noticeable artifacts in the tiling. Corner tiles still suffers from this. The discontinuities are now pushed to the middle of the edges but the problem is somewhat smaller since only two tiles meet there, compared to four tiles in the corner for vanilla Wang.

And there is this very popular adaptation of the Wang algorithm to the GPU in GPU Gems 2. I found this nice tutorial about how to create textures for it. This is from the game Path Of Exile, which appears to use Wang tiles extensively with great results.

As I was saying in my earlier post, the real challenge with all these methods is to come up with the content for the tiles. One approach is to synthesize the content of the tiles entirely. This way you are always in control of what happens at the connection points (corners, edges, etc. depending on which tile flavor you have picked).

This is how I built the cave system for the realtime engine. The crypt profiles, along with narrow tunnels, are drawn by a program. This program makes sure everything connects properly. Here is an example of the output;

The dots inside the large space are massive columns. These are in the tiles as well.

This is not difficult and provides good results as long as you are targeting simple shapes, something you can synthesize. Problem is, in most cases like with textures, you want to use samples you have taken from the real world and use them for the tiles.

If you read the Path Of Exile tutorial I linked before, you will see that this is not a simple task. A lot of work and skill was needed to produce good looking tiles. And this method is only for two colors, that is, the simplest set possible. Imagine if you wanted to have 10 different colors. It would be madness.

There are ways to automate this. If you have an original sample that is large enough, you can take different portions of it and assemble the texture set. Here is a fairly good description of the method (PDF). There is a lot of Frankenstein-esque ingenuity involved, as you must stitch different parts together, hoping the texturing discontinuities will remain small. This produces fairly good results if the original texture is homogeneous, but in most cases "hoping" is not good enough.

I tried this method for terrain synthesis. As a source I used a heightmap with some dramatic features in it. Here you can see a sample:

As the original sample was far from homogeneous, which is after all what you want in interesting terrain, I quickly saw that just one iteration of the method was not enough. This was an optimization problem, somehow I had to minimize the discontinuities produced by the tiles. It is the kind of problem you could solve using genetic algorithms.

You may feel you are entering the mad scientist realm, growing human ears out of mice, playing God with bits, and in fact you are.

The fist step was to design a DNA sequence. Each possible solution (an individual) can be defined as the border and tile coordinates within the sample bitmap. I was doing a set of 200 tiles, with 10 borders in each direction. That resulted in a few hundred bits of DNA.

Next I would do the first population of individuals, which I kept at a thousand. Each individual was randomly generated, just by flipping bits in the DNA.

Next a "fitness" coefficient is computed for each individual. This is a number that indicates how good the individual is as a solution. In my case it was simple, as I could compute the amount of discontinuities the solution had. I also threw in a histogram comparison with the original source, just to make sure all the tones in the original were kept. Adding these two errors gave me a direct measure of the quality of the individual.

Using this measure I could rank the entire population and throw some kind of sexy party. The hot individuals got to reproduce the most, but even the not-so-good ones got their chance to immortalize their seed. This makes sense because a very undesirable trait could become the base of a very nice feature in a future generation. Even the Great Mutato could be the beginning of something beautiful.

I used a very specific way of mixing the DNA, which is called "one-point crossover". It gets a random portion of the father's DNA and the remainder space is copied from the mother's DNA.

This way a new generation is formed. But only 80% of the new members are descendants of existing individuals. The remaining 20% is generated randomly generated, just to make sure there will be enough genetic variety. And of course there is mutation. Each children would have some of its DNA bits randomly flipped.

Watching these things evolve and get better over time is mesmerizing. But it was kind of slow. I realized a needed a massive approach to it. I split the logic into two sections. The part keeping track of the population, doing crossovers and mutations ran in a single process. Then the part that evaluated the fitness of an individual, which was the time-consuming part, ran in a little army of processes over a LAN. (I used good old TCP/IP for distributing the work, ZeroMQ evangelists please go away)

Now this thing ran in eight machines, averaging 3 cores. That is roughly 24 instances of the fitness evaluator. After a week running, I started seeing solutions that I could use. For instance, this image is the result of tiling one of these individuals:

This may seem like a success story, but to me it does not feel like that. The amount of effort and energy that goes into producing tilesets may not justify the advantages you get out of them. Even after weeks of running, most of the individuals still produced noticeable discontinuities.

I know in my case there is a lot of room for improvement. The cutting strategy for mixing tiles with borders can be improved a lot. One thing I want to try it to use patterns in the texture to govern the cutting path. This should mask the transitions even more natural.

The bottom line for me is, Wang tiles are amazing things until you try to use them seriously. They work great for stuff you can synthesize from the ground up. If you are trying to mix samples from real life, get ready for some trouble.

Introduction to Wang Tiles

This is guaranteed to be a very colorful post.

Since the early days of computer graphics, texturing has been the favorite method to bring detail into 3D surfaces.

The problem with textures is they take a lot of memory. While there are some attempts to have 3D worlds uniquely textured, like Id's Rage, the traditional approach remains to tile a smaller rectangle of texture across a much larger 3D surface.

This certainly does the job, but it also brings out these repeating patterns we dislike so much. There are a number of ways to mask the repetition, sometimes by blending more than one texture, or by adding splats of unique detail in strategic places.

Another approach is to use Wang tiles. If they are new to you, once you are done reading this you should Google them up and get soaked on what they are and how they work. You will love learning about this. It will expand the way you look at problems in the future. Wang tiles are much more than just a method to avoid repeating textures. You could use them to generate terrain, geometry, architecture. If you get carried away, you'd see they are actually full fledged computing devices on their own.

You could say we have used them already without knowing, as the traditional way we tile textures is one specific case of Wang tiles.

But forget Wang tiles for a moment. The way you would normally texture a brick wall is to use a 2D image and let the GPU do the tiling. If you just take a picture of a brick wall in real life and use it, you would see there are very noticeable discontinuities where the edges meet. The colors on the North edge of the image do not connect properly to the colors in the South edge, and the same happens with East and West.

This is very noticeable in the following polka dots texture:

This is easily solved by making the texture "seamless". What it involves is to make sure the West and North edges match the East and South edges respectively. In most cases this is done manually, using a tool like Photoshop.

Here you see the fixed texture:

Problem solved. But this texture still has very noticeable repeating patterns. If you tile it long enough they are hard to miss.

Here comes the twist. We made this texture work by making sure edges connected properly. We have one edge that is shared horizontally, and another one that is shared vertically:

So what is inside the tile does not matter as long as the edges connect properly. Instead of just one texture we could use several, all sharing the same edges. Here is an example:

We could pick which tile to use randomly. This will eliminate the repeating patterns coming from the center area of the tiles. But we would still have some repetition coming from the edges.

Here enters Mr Hao Wang. He predicted what we are about to do was not possible, and for being wrong this method bears his name. In reality he did all the heavy work so it is more than deserved. Here is a picture of Hao Wang winning the 20 Km men's race in Chihuahua:

Actually that is not Wang, but it is the spirit what counts.

Wang tiles allow us to have more than just one vertical and one horizontal border. You can choose to have as many as you want. As soon as you know how many different borders there are, the method tells you how many tiles you need so you can tile them all you want without leaving any holes.

Let's say you choose to have two horizontal borders, blue and yellow, and two vertical borders, red and green. Here is a resulting set of 16 tiles:

This is what the tiling authorities call a "complete set", meaning it has all the possible tiles. It contains every combination of every combination of colors. This is how you would compute the number of tiles in a complete set:

Where N is the number of horizontal borders and M is the number of vertical ones. Hence 2 horizontal and 2 vertical make a set of 16 tiles. Now these numbers explode quickly. For 3 colors in each direction you now have 81 tiles. For 4 colors it would be 256 tiles. Four colors do not provide too much variation along the edges, still the number of tiles is huge. If you had to store this info in video memory it would be a problem.

Thankfully the math guys proved you do not need a complete set to achieve full tiling. A much more smaller set exists which can be created in the following way: For each combination of North and West colors, you need at least two tiles. Now the tile count becomes:

This is a lot more manageable. Two borders on each direction requires eight tiles. Four borders, 32.

Once you have your tile set, you can use it in different ways. When time comes to place them, there are a few simple rules to follow. You start from the top-left corner, North-West. At this point you can place any tile from the set, picking one at random will do.

The next tile you place must match the existing one. You can still pick a random tile, but its West side must match the East side of the previous tile. You can continue to do so for the first row:

When starting the second row, you will have to match the South border of the tile on top, in this case you can pick any tile in the set that has a green North.

And then for the rest of the row, you will find tiles that match both the North and West color requirements:

At this point you are ready to go on tiling forever:

While all this seems great and guaranteed to make you interesting at parties, the difficulties with Wang tiling are not in picking edge colors and using the sets. The real challenge is to come up with content for the tiles. In my case I had to resort to genetic algorithms running many instances over a LAN.

I will cover this in my next post.

The Uncanny Valley of Procedural Generation

As a developer of procedural worlds, what worries me most is not failing spectacularly at my goals. My biggest fear is to produce something that looks believable, but still is somehow off. Even a seemingly perfect world could be rejected by your subconsciousness. You may not be able to put it into words, or point your finger at it, but you feel there is something wrong. 

To make matters even worse, it seems we can be collectively hypnotized into liking something just because it is a new way of doing things. Soon the novelty wears off and we realize the Emperor was naked all the time. 

We want to believe things look better that they do. For instance we know 3D graphics are a developing field so we are ready to forgive a lot, until something better appears and sets a new standard.

Remember this beauty?

This is Peter Gabriel's "Kiss That Frog". You can see the video here. It won a MTV award for special effects in 1994. I remember loving this video. It is hard to watch now.

This is not specific of procedural techniques. Humans are equally able to generate uncanny, ugly things. The problem is proceduralism makes it a lot easier. The world is not crafted by hand, any aberration in the algorithms will be mindlessly multiplied. It also depends on the degree of realism you want to achieve. If you are trying to fake nature, odds are your creation is some form of monstrosity that will not stand the pass of time.

I often wonder whether any sort of synthetic reality is doomed in the long term. At this point I don't know for sure, but I have two simple ideas to guide me across this maze:

1. Global rather than Local

Procedural methods can be divided into two large families, global methods and local methods. In local methods content generated for a given point in space does not depend on the content of neighbor points.

The Perlin noise function, for instance, can be evaluated locally. This means the output of the function depends only on the coordinates of the point, plus some constants. It also means many points can be evaluated in parallel. Since they are isolated from their neighbors, you do not need to compute any neighbors before evaluating a single point.

They are blindingly fast, but it comes at a cost: They do not have a soul. They do not produce any information. All what you see comes from a small seed of values and the specific ways these values are churned and shaken by these clever algorithms.

Because of their speed and fairly good results, they can be used in many subtle places, but they should not be the backbone of your world. Our minds are very good a discovering redundancy. All these methods are like a kaleidoscope effect, they can trick you for a moment but soon you start seeing the mirrors. And once you have seen them, the magic is gone for good.

Here is your typical multifractal Perlin terrain:

Just don't do it. We all know it is not real.

Another popular local method is L-Systems. This is when they are in their vanilla form, as context free grammars where symbols are replaced with no awareness of their surroundings. If used to produce trees, you soon have branches that intersect each other, or that go into illogical directions.

Here is one gem that illustrates this point:

Global methods, on the other hand, are closer to simulations. The value of a single point may depend on the values of very distant points. Imagine a fluvial erosion filter. A point at the base of a mountain may be largely influenced by a large streak of points uphill.

Global methods ere effective because they have cause-effect relationships built into them. This makes all the difference. It brings entropy into the world, it gives it time and history.

Here is an example showing some fluvial erosion. The patterns you see here are far more believable:

In the same fashion, most successful tree and plant generators use some sort of simulation or at least global constraints. For my trees I chose a global method that grows branches in full awareness of each other. It is a very simple algorithm and still beats the results you get from vanilla L-Systems.

The problem with simulations is they are costly to evaluate. There is information moving around so they are harder to compute in parallel. 

Consider this example. The location, shape and strength of a river will be determined by a water source many kilometers away. A visitor to the virtual world may encounter the river long before than the source, but still the source and everything between must be accounted for. For worlds that are generated on demand as the viewer moves this may be too much to handle.

Even then, you should always consider using a global method. It may make your solution complex and slower, but you will have something to show.

2. Steal from Mother (Nature)

Nature has already spent a lot of time and energy producing the patterns we accept as real. We already use them to texture 3D models, there is no reason why we couldn't go beyond that.

Here is a very interesting approach to terrain synthesis. It combines elevation samples from real sites from Earth and stitches them into new ways. This results in fairly believable scenes that can cover huge spaces without any apparent repetition.

In this case they used some samples taken from the Grand Canyon.

A similar technique can be used for smaller terrain features like rocks, cliffs and small boulders. It is possible to have a set of volumetric natural textures and map them over larger terrain features. 

The biggest issue is how to mask the repetition, but this can be done using Wang tiles. I have reworked many of my core functions to be like this and I like them better than my previous functions. I will be posting some results soon.

To sum it up, I think entropy matters.