Monday, January 30, 2012

Introducing Voxel Studio

This introduction was also long overdue. For more than a year I had this program that allows me to work on a definition for the procedural world, then schedule it as a render job in the Voxel Farm and finally show the results on screen. Many of the screenshots you have seen in the past were taken from the viewer in this program, but the UI for it remained hidden. Until today.

I give you Voxel Studio:


This program has a series of editors for the different things that make the procedural world. In the previous screenshot you can see the Terrain Layers editor, which allows to define the multiple layers of the terrain. For instance, there is one layer for each type of rock. The placement of these layers is determined by masks, which are the output of an earlier process. It also covers Vegetation and Architecture, but I plan to show that later this year.

Something I like about this program is that it uses the Voxel Farm to create previews. Right now I have only three good machines in the farm, a single scene like the one in the screenshot still takes close to a minute to render. If I had six machines I would wait only half that time.

The Renders view will show you a list of the jobs currently in execution inside the farm:


One only render job at this moment, but there could be several at the same time. The green cube shows a world chunk that has just been received from the farm.

The material editor allows to define the multiple layers that make each material:


These layers follow the same model I had introduced in an earlier post. They are rendered in real-time, although there are no shadows nor global illumination unless you perform a deeper render.

If you are only working on the materials there is no need to recompute the geometry of the scene, this ends up being a pretty neat and fast workflow.

If you want to make sure you are working on the right layer of terrain or material, you can always ask the program to highlight it. The following screenshot shows the same scene as before, but the "Volcanic Talus" layer appears highlighted in red:


In the future I will post more about this program. If I get enough time I will capture a video so you can see a quick session.

In the poll at the top of this site there is an option that reads "Content creation application (like WorldMachine)". This option has received one third of the votes compared to the most popular option. This is a lot of votes once you consider it is a tool and not a game like other entries in the poll. Voxel Studio is that application.

Like anything else in this project, I'm not sure if it will ever be released, but I'm closer to having this finished than a game or anything else. It could be a nice procedural creation suit, including both the studio and the farm. If you see yourself using an application like this, I would like to hear about what features you'd like on it. Please leave your suggestions as comments and we will sort out what is feasible and what is not.



36 comments:

  1. Dude. This is amazing!

    I totally agree with you on the viability of this as a content creation application. One of the most difficult things as an indie is to get affordable, usable assets. Artist time is absurdly expensive for the output compared to programmer time. With your render farm and procedural variables you could flesh out the possible scenes - perhaps with artist assets in the windows and prerendered assets you use - and have a nearly unlimited asset production pipeline. You would be able to undercut the shoddy asset packs and offer unlimited variety. A house, by itself is SO useful. And for someone like me a VOXEL house asset is like a snipe on the internet.

    Still, imo, you will get the most travel out of this if you can offer the interface online. Have you tried Javascript? A login, fee-based HTML5 app that rendered on your farm - even at it's current capacity with a schediuled delivery for the final asset and maybe a faster pre-render - could give you a ton of inflow. I guess the wager is how much, but I'd be willing to bet it would be a lot. Javascript and HTML5 are really easy, imo. The app looks, maybe, qt based but I bet port things are easy to port to JS.

    You know I think I voted for content creation tool when I first ran into this site and this is totally why ;) I think you could get a lot of travel out of doing a proper delivery system for infinite assets.

    ReplyDelete
  2. @kayahan: Thanks.

    A fully online solution with a JS front would normally be my first option. I would have no problem programming that, I have been designing rich AJAX applications for a while now. WebGL would be great for the viewer. Most of the stuff you see in the preview is done in shaders, which would translate very well to WebGL.

    The problem with this approach is there is an big overhead to it. I would need to host servers for the application itself and then enough farm machines for everyone to have an enjoyable experience. I would also need to store all the project's assets on the server.

    The real killer are the bandwidth requirements. A single project will can take easily 100 Megabytes of assets, which you would need to upload. Textures are big, and for each texture you use several maps are required: diffuse, normal and displacement. If you want to have a detailed preview of the resulting meshes, just the meshes for a single scene could be above 100 Megabytes. You would need to download that, and also whatever portion of the textures are being used.

    If I had a lot of cash to invest in infrastructure, maybe I could pull this off. But never on a hobbyist budget.

    It will be more scalable to require users to run a client application, even if it means you will need to install something in your machine. If you don't have access to a large farm, the application could still take your project and render it on the cloud, you would pay only for the computing resources you use.

    Another thing worth exploring is P2P. Each user machine running the software could act as a farm node. A karma system would get your project rendered by other people's machines. For a popular tool this approach could provide computing power and affordability like we have never seen.

    ReplyDelete
  3. Would it be worth considering using this suite purely as an environmental creation service? A client would design their desired terrain themselves using your tool, have the output meshes sent from the farm to their build machine and then use those assets in whatever project they wanted it for (for a price, of course). That way you wouldn't need to worry about handling distribution of the assets to every end user. The client would be responsible for that.

    ReplyDelete
  4. When I first read about the farm, and this post on the client, my first thought was P2P. Depending on the goals there are a couple of options there.

    Like you said, for a content creation work/play-thing the power of a P2P distributed, or cloud based, computing would be formidable. I personally think this would be an interesting direction, but am more excited to see a game result.

    Even if you end up not making this a creation tool for all of to use creating our own worlds, but solely a game of your own design... I still think a P2P model could give you something your hobbyist budget can't, computing power.

    A number of indi developers are capitalizing on the "early adopters fund the development" model, but this requires something that is hard to come by, lots of early adopters paying a little, or a few paying a lot -- and money is still in short supply for many.

    You could give yourself a big leg up by creating a distributed P2P client that a number (say a couple hundred) of "committed fans" can install and run to give you a very powerful farm, in return for something less tangible, like beta-access or a free license to the final product. It would be a win-win and the early adopters will have pride in their participation which could increase their evangelism in the early days of the product.

    Just an idea :)

    I continue to admire your work, BTW.

    ReplyDelete
  5. @MCeperoG: You're totally right: that is a lot of storage!

    I really like your P2P idea! It would undoubtedly lead to the most efficient final product. If that's your goal, have you thought of using a hybrid approach of p2p and hosted servers? I.e. treat your server farm as another, preferred, client and as the tool takes off your server farm will automatically fall to the bottom of the list as it's resources get taxed and more concurrent users open for parallelization. You would probably still need to beef up the server farm and your connection, in the beginning. But that might lead to a pretty beefy solution - if no one's on they have solo server time and if it's taxed they have peers.

    Would it be possible to refactor the rendering code for a p2p javascript application? That would still give you the wildfire potential of a webapp with the low server costs. Dunno if JS is too slow for all the steps though.

    ReplyDelete
  6. @LachlanL: My estimates are considering it is only for content creation. It would never never distribute media to end-users. Even then the bandwidth requirements are huge. A world designer using this tool will download near 100MB per preview. These are huge worlds, you need to render previews anywhere to make sure all your layers are right. I do from 10 to 30 in one single hour. That would be gigabytes of traffic for a single user.

    The option would be to limit the resolution, but if you want these meshes for a movie set you would want to see them in full detail every time.

    The cloud advocates keep saying the "network is the computer". The reality for many real world tasks is far from that. Bandwidth is too small and expensive compared with the processing power in the desktop.

    ReplyDelete
  7. @Jonathan, kayalan: The problem with P2P is how to protect your content from being "stolen" by other users. Let's say you have spent some time developing your world and you send it to the P2P farm. Other nodes in the farm will need your world definition to be able to render it. Even if there is some sort of protection in the node software, it can be easily broken since it is a self-contained system. So no protection in reality, only obfuscation at best.

    Anyone with a node would get access to your work and be able to reproduce it, change it and improve on it.

    Some enthusiasts will be fine with this scenario, but I'm afraid the majority of users would like to be in control of how they share their world projects, if the decide to share it at all.

    So P2P is not so fun when everyone pirates everyone... the irony! :)

    ReplyDelete
    Replies
    1. Seems to me that if someone was really worried about that, perhaps they could switch to a whitelist: Only share data with specific machines. That way they could keep taking advantage of all the machines they trust (every machine in the studio + any home computers and laptops they can dredge up) but not worry about some stranger getting hold of it.

      Meanwhile, everyone else is reaping the rewards of the full network, confident in the knowledge that they can always pull out when they want. Anything they've developed up to that point is irrevocably public, but I honestly don't think most users would mind.

      Delete
    2. Yes, whitelisting would work. I will consider this approach.

      Delete
  8. I have to say the same as everyone else, this looks amazing. Along with your last post about the farm, I am totally in awe.

    I have a suggestion about the P2P idea. As a free/open source software user, I think the idea of a giant, world-wide network to potentially create this content would be great. I wouldn't be one of the people who would mind about people potentially getting my world definition, but I can definitely see where the issue may lie. I think it was mentioned previously that you can have a priority system where premium clients get access to secured nodes, usually under your control. Services like AWS could provide those nodes for you pretty easily, and from what I see, it should be fairly easy for you to throw up a couple of nodes if/when needed, meaning you wouldn't have to necessarily pay for their upkeep if you haven't got the cash coming in to pay for them.

    ReplyDelete
    Replies
    1. I should have mentioned, I don't necessarily expect you to release this for free necessarily, I was just comparing it to the community feeling of open source software. Plus hinting that I would like to play around with it, of course :P

      Delete
  9. @Nistur: No problem, I get the idea.

    ReplyDelete
  10. Re:'how to protect your content from being "stolen" by other users'

    If you were going massively parallel couldn't you break things down to the point they're no longer really recognizable? Like each machine is only rendering a wall of a building. Or a single mountain. A single tree, etc. People could get parts of assets but not wholesale theft.

    Also you could always approach it as developing the crowdsource, p2p option for intial investment and proving out while you also develop a pay service that would be entirely secure. Which makes sense, if you want EA to use this someday. You know that leaking a single leaf of a single tree in the new Morrowind game could get some panties in a wad.

    I was also thinking that with the p2p option you could make a background/screensaver app to borrow free cycles that, maybe, people could use to earn render time.(People would dedicate machines if so) You could similarly get a lot of assets for free by offering render time for every X window, door, chandelier, w/e art assets people contribute. Might give you a lot of property quickly without a lot of initial investment, which seems like the key to making it attractive to the lucrative large investors.

    ReplyDelete
    Replies
    1. This is possible to some degree, but things cannot be broken down to very small levels. The advantage of procedural generation is to produce a lot of content from a small definition. A small portion of a world definition can still be a lot. For instance I may not be able to "steal" how your entire world looks from a single render request, but I could get one or two entire terrain biomes.

      Delete
  11. Hey there.

    Regarding what to do with the current tools you have and the possibility of making a game out of it: Would it be possible to, instead of making a game with this, concentrate on the world generation tool like you have until now but optimize the output so it could be easily imported into SDKs like CryEngine 3, UDK, Unity, etc. which would then be used to make the game itself.

    ReplyDelete
    Replies
    1. Yes you are right about that. For this tool to be useful it must output to these engines.

      There are so many out there. I may be able to support only one at the beginning. What would be your pick?

      Delete
    2. Hm. While I personally would prefer to see the CryEngine 3 be supported as IMO it's more promising, it seems to me that Unity and UDK are more commonly used at the moment. That can change of course, so I really can't say. I imagine the best way to get to a decision is to look at statistics and also check out the SDKs yourself (they're all free after all) to see which would work the best / which you prefer.

      Delete
  12. Miguel, once again you've proven voxels are amazing, and I am grateful for that to you.

    ReplyDelete
  13. To Miguel (and everyone else ^^)

    Hi!

    We are a group of people sharing the same vision of creating a volumetric/atom/voxel-based universe inhabited of artificial intelligent lifeforms running in virtual neural networks - "playable" in a MMO -like way.

    Interested people are most welcome to join us - we are still carving out the basic blocks of how to make the vision "real". Look us up on Google Groups (the group itself is called Virtual Biosphere).

    The first step we need to solve is getting access to using a multiuser voxel/atom engine.

    Can anyone help us with that? Lead us in the right direction?

    (We have Branislaw Siles from Atomontage in the group, but AE is not yet ready). So what to do - we want to get started :)

    All help is greatly appreciated!

    And again: Please join us :))

    Adam

    ReplyDelete
    Replies
    1. I checked what you want to achieve at your group's page:

      https://groups.google.com/forum/?fromgroups#!forum/virtual-biosphere

      I must say it is very ambitious. This is the dream of any virtual world builder. The problem I see is dreams are hard to achieve. To me it compares to putting a rocket in orbit, or making a computer that can play Jeopardy. This will require a lot of effort and money to succeed, otherwise you will need to severely compromise the results.

      For the time being I am struggling to complete my smaller, way less ambitious project. I really don't have the time to contribute to something like this.

      But thanks for the offer.

      Delete
  14. Therefore the team - and not "one mans project from his couch" as you write :)

    Still: everyone who wants to join us you are welcome to swing by and see if we can help each other!

    All the best
    Adam

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. MAN THIS ARE WOUNDERFULL NEWS !

    Been folowing your blog a year now - And this is reaaly amazing.
    Expecialy if enable us to Select parts of the Voxel Studio World and export to Obj with uvs texture or vertex color shaders out ...

    ...
    This would be also very Amazing if Voxel studio native format saved worlds could be loaded and integrated with game engines as Unity / Udk / Cry or even as a plugin inside 3d aplications max / c4d / Lightwave / modo ...

    But even if just exporting Selected geometry & textures out would be already amazing ...


    I'm the admin of a Open SOurçe Online Production 3D project / the SOULSSHINE.ORG / Soulsshine-online.deviantart.com ... ( our project is currently in constructions )

    And we been developing Literary and 3D/Interactive Graphic Novel Productions with our Online Team and this is reaaly what we would need to Kick up the Results of our Graphics ...

    We been using in our several open projects apps as Voxelogic Acropora In terms of voxels / Vue / Max / and even UdK / Even SPace Engine ... Of course terrain Engines as Worldmachine is our Day constant on all the production here ... but of course a flat 2d plane terrain is a flat nothing comparing to voxels , so we usualy use 3DCoat Voxels and Cropora to build and modify Our Virtual Worlds ...

    But as far as procedural generation features / there is nowere in the world a PROCEDURAL VOXEL STUDIO Like yours ...

    And seriously Your VOXEL STUDIO
    / WOuld be a Dream cometrue here ...

    PLEASE !
    Whenever you Need a SERIOUS BETA TESTER
    Please contact our STudio at Rubyworksproductions@gmail.com !


    THANK YOU !

    Reaaly hope to see this VOXEL STUDIO released SOmehow !

    GOOD LUCK !
    Thank you so much!


    -------

    RPGSI STUDIO ADMIN
    COIMBRA PORTUGAL

    ReplyDelete
  17. Replies
    1. Perhect! Fate robbed me of my feet, I live in Russia, in a very large and dirty city. Be in the world, created by you, wander through the waving grass, the white snow, the majestic mountains - it is my hope ... I'm looking forward to it! Translate by Google

      Delete
    2. This will happen sometime in 2013. I wish I knew the exact date, but I don't. It will be out when it is ready.

      Delete
    3. Thanks for eventual sharing this.

      Delete
  18. Still making my way through your blog... so you've built voxel studio yourself from the ground up? An wonderful piece of programming - looking forward to you releasing this... put my name down for sure!

    From an artistic point of view, something else that would be cool to see is stream/river/lake generation. I don't know if precipitation/dynamic water levels could be done (rivers are higher when fed by a large amount of rainfall on the feed areas to the river).

    Very eager to see how you get on with this :-)

    ReplyDelete
    Replies
    1. apologies, just been reading the architecture post and you discuss water generation in there. I'll read up properly before posting again!

      Delete
  19. Congratulations Miguel Cepero, I had never seen something so different and at the same time with high quality graphics, I live in Brazil and I was surprised by it. I do not know what you meant to develop further with their creativity, but if by chance for a game or a program I'm sure it will be magnificent. (Translated by Google)

    ReplyDelete
  20. I give this and a for awesome .i can't wait to get my hands on Voxel Studios game .

    ReplyDelete
  21. Summer 2013?
    Would give people / designers a lot of free time to develop fantasy worlds then.

    I can only imagine the possibilities..... player made versions of Game of Thrones world, Elder scrolls, DnD settings.... the list goes on.

    ReplyDelete