Desktops, Tablets and Phones

One of my goals starting this project was to have relatively simple client applications exposing rich and complex worlds. While we later worked on generating as much as possible in the client-side, there will always be a case where you want access from power-challenged devices. Phones, tablets and even desktop web browsers do not necessarily have the power to generate everything you would like to have in your virtual world, but are still ideal mediums for people to experience it.

The good news is that generation can be offloaded to the cloud. Check out the following video. This is me at my home running several of these simple clients. The content they display is generated by servers running in Amazon's cloud:

Landmark Voxel Creations

Not sure if you knew about this, but Everquest Next Landmark entered Alpha about one month ago. During that month players were introduced for the first time to the voxel world and tools we have jointly developed with Sony Online Entertainment.

We still have a lot of work to do. The game just now entered Beta. Still I am marveled by the incredible creations made by the players in such a short time and with such early versions of the tools.

Have a look:


I do not know about you, but it seems to me player-generated-content does come close to what game studios can do. Hopefully very soon we will be able to completely blur that line.

Probably the biggest surprise was to see all the emergent techniques devised by the players. We knew our voxels were able to encode all sort of funny things, however the specifics of how they could be achieved was a purely player-driven development. Players even had to name these things, so they gave us "microvoxels", "antivoxels", "zero-volume voxels" and other similar things that actually make a big difference on how you can create in the game.

Someone once told me the best software you can write is one that won't have any users. You can relax and have a life. Users (or players in this case) are that reality check developers secretly fear so much. Now I realize this software cannot exist in isolation from the builder community. Thanks to our players we continue to learn and understand about all the emerging properties of the platform we have created.

Keep up the amazing work guys!

Voxel Physics

Voxels are typically associated with creating content. As it turns out, they are quite useful when the time comes for destruction.

Voxels make it easier to break things. Imagine you fire a rocket into a column. You can blast a hole where the rocket hit. Using the column's voxels you could create several fragments of debris. If the column's ability to stand or support other things is compromised, voxels can tell you that. At this point you get even more fragments, which could impact other voxels generating more fragments and so on.

Since you are looking at volumetric data, computing the mass and other dynamic properties of these fragments is much easier. Imagine you have a very irregular shape, made of many different materials. For a proper physics simulation you need to figure out how much the thing weights. This is a trivial process if you are using voxels. Each voxel has a material assigned to it, the material's density tells you how much the voxel weights. The weight of the fragment is the sum of the weight of its voxels. And it is more than that, you can figure out where the mass center is. Imagine a fragment that is half rock, half styrofoam. The object's center of gravity is where the rock is concentrated. The styrofoam adds very little weight.

Here is a video showing a little bit of destruction. This is still in an early stage but hopefully you will get an idea of the potential.


This video was captured on my old PC, with an Intel i5 and an ATI 4770.

Voxel Farm is hiring!

In case you do not know already, in 2012 I started a company called Voxel Farm Inc. All the pajama work I had done since 2006 went into this company. We hired a few developers to work on the contracts we were getting at the time, like EverQuest Next.

At this point we are looking for one or two very talented artists for our mystery project. This is the ideal profile:

- Digital sculpting (z-brush, mudbox or 3D-coat)
- Creation of seamless photo-realistic textures

If you live in Montreal it is a bonus. We are setting a nice sunny office in the south shore. Your involvement can be part-time or full-time. It does not matter as long as you are an ace in your work.

If interested please email your portfolio to: mceperog (at) gmail.

Spread the word!

Video Update for August 2013

August is almost over, time for a video update:

EverQuest Next

EQNext has been revealed. What we got to see was so good, they have set the whole industry on fire. They have Voxels and Procedural generation as one of the main pillars of this game. If you saw the reveal, and have been following this blog for a while, you will find many similarities between EQNext and what Voxel Farm does. This is no coincidence: EQNext is using the Voxel Farm engine.

The engine is just a tool. The EQNext team deserves all the credit in realizing this vision. Their art direction and engineering skills are unlike anything I have seen. I am blown away by what they have achieved with the engine, especially in so little time. I am very proud of being involved in this project.

You can see the entire keynote here. It has plenty of real gameplay footage:

http://www.twitch.tv/soe/c/2680835

And here are a series of videos that have appeared in different gaming sites:

Video Update for July 2013

Here is the latest video update. If you are keeping count you will notice I skipped the one for June. I would have done it in time, but one of my twins snapped the microphone I use for such recordings. I did not have time to get a new one until last week. For that reason this update is a bit longer.

Storage Matters

Imagine you were creating a massive persistent world where everyone would be able to change anything at will. It is a simple, powerful idea that eventually has occurred to everyone ever exposed to a game. Why there aren't many of these worlds out there? Well, this very simple idea is quite difficult and expensive to execute. Not only you need to store the information, you have to be able to write it and read it in a timely fashion.

Then how about your own personal world, something you can run in your PC and invite some friends to play over. How much of your PC's performance are you willing to sacrifice, how many people could you actually invite before you would see the quality of your gameplay begin to suffer?

I began wondering whether all the above could be manifestations of the same problem. What if you could have a storage solution that is lightweight so enthusiasts could run at home, and if you pieced enough of them together you could scale it so it would run massive worlds the size of planet Earth?

As it turns out it was possible. I have now a shinny new database system that does exactly that. The main trick is it aligns with the same other concepts of the voxel world. So this is mainly a voxel database. It won't do any SQL queries, XPath evaluation or any other form of traditional DB interaction. It just stores and retrieves voxel data very fast.

How fast? Over a 10 minute period, a machine with six-year-old Intel processor (T2500 at 2GHz) and an equally crappy HD was able to serve 10 Gigabytes worth of individual queries while another 10 Gigabytes worth of queries were being written. Each query ranged from 500 bytes to 100KBytes worth of data.

That would translate into a lot of friends sharing your server. To give you a better idea, a volume of 40x40x40 worth of player voxels compresses to 2K as an average. Here is how you would compute how much space 10 GB of voxel data would be:

1 chunk = 40x40x40 voxels = 12x12x12 meters
1 chunk = 2K
10 GB = 5,242,880 chunks = 2048x2048x2048 meters

How many people can create this amount of voxel content in 10 minutes? I have no idea, but I bet it will be an entire army. At this point the DB is the least of your concerns. The bottleneck is in the network.

The twist comes now: While this rate was sustained for 10 minutes, it was not meant to push the system to the limit. The DB process CPU usage never went up 1% and the memory usage for the process remained at 3 MB. The system was responsive and usable (well as usable as a six year old PC can be), showing no big difference in behavior.

Here is some evidence:

For most of you who are more artistically or design inclined this is certainly the most boring screenshot I have ever posted. But if you are into programming this kind of thing, this is process porn.

Of course the system is doing real work. The main clue is in a different column not displayed by Task Manager: Virtual memory, which was hovering all the time below 20 Megs. Even then the virtual memory was lower than what Google Chrome was using, which was a whooping 99 Megs.

The voxel database is so fast because it uses the same virtual memory management of the OS. So, instead of writing to files in the HD directly, all the information is mapped through the OS paging system. Only the pages that need to be altered go into memory. Also the system does a lazy write to the HD. Even after the process is gone, the OS continues to save the changes to disk.

I feel this is the stepping stone for great things. It will be fairly easy and inexpensive for people to set up their own servers. They could be hosting a lot of players and barely take a hit for it. This of course depends on how the networking is implemented, which leads into another favorite topic of mine: how to make a server that will not bring your PC to its knees. I will be covering that in the near future.

20 Light Bounces

I rewrote the radiosity solution over the weekend, this time for the Voxel Farm. Here you can see some results for the same forest scene of the earlier video:

It is a bit tricky to understand the light in these pictures. Primary light sources like the sun and the sky are not shown here. You can only see the light that has bounced off the surfaces or went through the tree crowns, which are translucent.

There are no materials or any texturing. The green hue comes from the crowns and parts of the ground that are covered by grass. But don't pay too much attention to the colors. This was to see if the algorithm was working properly. Also keep in mind the crown polygons are there only to collect, scatter and reflect illumination. They are replaced by the instanced leaves.

I found I needed to get up to 20 bounces to get enough illumination down to the forest ground. Otherwise it was too dark.

One quick session in Voxel Studio

As promised, here is a screen capture covering some other features in VoxelStudio. Here you will see terrain rendering happening on the Voxel Farm, also a glimpse of the tree and forest edition process.



I had only two nodes in the farm dedicated to rendering the terrain meshes shown here. A farm with four machines would have rendered twice as fast.

Introducing Voxel Studio

This introduction was also long overdue. For more than a year I had this program that allows me to work on a definition for the procedural world, then schedule it as a render job in the Voxel Farm and finally show the results on screen. Many of the screenshots you have seen in the past were taken from the viewer in this program, but the UI for it remained hidden. Until today.

I give you Voxel Studio:

This program has a series of editors for the different things that make the procedural world. In the previous screenshot you can see the Terrain Layers editor, which allows to define the multiple layers of the terrain. For instance, there is one layer for each type of rock. The placement of these layers is determined by masks, which are the output of an earlier process. It also covers Vegetation and Architecture, but I plan to show that later this year.

Something I like about this program is that it uses the Voxel Farm to create previews. Right now I have only three good machines in the farm, a single scene like the one in the screenshot still takes close to a minute to render. If I had six machines I would wait only half that time.

The Renders view will show you a list of the jobs currently in execution inside the farm:

One only render job at this moment, but there could be several at the same time. The green cube shows a world chunk that has just been received from the farm.

The material editor allows to define the multiple layers that make each material:

These layers follow the same model I had introduced in an earlier post. They are rendered in real-time, although there are no shadows nor global illumination unless you perform a deeper render.

If you are only working on the materials there is no need to recompute the geometry of the scene, this ends up being a pretty neat and fast workflow.

If you want to make sure you are working on the right layer of terrain or material, you can always ask the program to highlight it. The following screenshot shows the same scene as before, but the "Volcanic Talus" layer appears highlighted in red:

In the future I will post more about this program. If I get enough time I will capture a video so you can see a quick session.

In the poll at the top of this site there is an option that reads "Content creation application (like WorldMachine)". This option has received one third of the votes compared to the most popular option. This is a lot of votes once you consider it is a tool and not a game like other entries in the poll. Voxel Studio is that application.

Like anything else in this project, I'm not sure if it will ever be released, but I'm closer to having this finished than a game or anything else. It could be a nice procedural creation suit, including both the studio and the farm. If you see yourself using an application like this, I would like to hear about what features you'd like on it. Please leave your suggestions as comments and we will sort out what is feasible and what is not.

A trip to the Voxel Farm

This post is about something I have not discussed before, but somehow it has been present on every screenshot or video I have posted for a while now.

For more than a year I have been building a little farm of machines. They run a series of programs I wrote for the procedural generation. They work in parallel, sometimes doing the same task over different locations of the virtual world, sometimes running very different tasks one from another. Their efforts are highly coordinated. Thanks to this I can get large portions of terrain, forests and buildings generated in very little time. I can see the results of the changes I do to the world definition without having to wait too long.

What it is best, this setup allows me to throw more nodes in the network at any time. I only have three decent machines in the farm right now. They are old gaming rigs I found on Kijiji around Montreal. The minimum spec is 8 Gigs RAM, 3 or 4 cores and an ATI video card better o equal to a 4770. I need them to have GPUs because some algorithms use OpenCL. I cannot afford to get too many of them right now, but having software that scales over multiple systems is already saving me time.

What I like the most is that it really feels like a single very powerful machine. The existence of the farm is completely transparent to the application I use to design virtual worlds. I will cover this application in a future post, you have already seen many screenshots taken out of it without knowing.

I would like to introduce you to some different animals I keep in this farm and explain a little about what they do.

Dispatcher

The dispatcher controls everything that happens in the farm. There is a single instance of this process for the entire farm. At any time the dispatcher keeps tracks of the different jobs currently active. Each job may be at a different stage. The dispatcher knows to which farm worker should direct the next request. All the coordination happens over TCP/IP. The dispatcher listens on two different ports. One is for the farm workers to report their progress and get new work assignments. The other one is so clients of the farm can request new jobs and also query the status of ongoing jobs.


Contour
Several layers make the virtual world. Some are terrain layers, some vegetation, some are buildings and roads. All these layers have something in common, they represent a volume with an inside and an outside. Contouring is the process that allows to find the surface that divides the inside from the outside. The world is broken into many Octree cells. Each contour worker can process a cell individually. It knows which layers intersect the cell so it runs an algorithm known as Dual Contouring on the contents of the cell. The result is a very detail polygonal mesh.


Decimate
The meshes produced by the contour phase are very dense. If they were fed to the next processing stages it would slow them down. For this reason they go through a phase of decimation. This is a fast Multi-Choice mesh optimization that preserves topology, and only removes those triangles that bring very little difference to the mesh. The resulting mesh is very close to the original, but the number of triangles is drastically reduced..

Reduce
I use a LOD system to replace several distant small cells by a larger cell. Since they are Octree cells, this means combining eight children cells into one large parent cell. Even if it covers eight times the space, the parent cell must be similar in byte-size than the child cells. This means the eight children must be brought together and compressed. The compression at this phase does change the mesh topology, otherwise it would be impossible to achieve the target sizes. Then the resulting parent cells are again combined into a larger parent cell and so on, until the highest LOD cells are obtained.


Project
This process takes a high resolution mesh from the decimate or reduce phases and creates a very simplified mesh out of it. Then it projects the excess geometry on a normal map. The results are compressed as I described before and stored in a cell definition file. These are the files that are sent to the client for rendering. At this point the processing for a single cell is pretty much done.


I have not covered here the generation of cities, architecture, forests and other elements. They blend into this sequence and also live in the farm, but I think they deserve a dedicated post.

Probably the most interesting aspect of writing a collective of programs like this was how to make it reliable. Since I was targeting unreliable hardware to begin with, I realized failure had to be an integral part of the design. I devised a system where none of these processes expects you to do proper shutdown on them. They could just vaporize at any point. Actually I did not implement a way for them to exit gracefully. When one needs to close, the process is simply killed. The collective has to be resilient enough so no data corruption arises from such a failure.