SculptAR: 3D sculpting in AR

One of the unique capabilities of augmented reality on smartphones and tablets is that, for the first time in the history of the world, hundreds of millions of people carry a device in their pockets capable of “6DOF” tracking. 6DOF stands for 6 Degrees of Freedom: the device’s movements can be tracked when it rotates around three axes, and when moves along these axes.

Sculpt of some balls to highlight how the cast shadows help to understand the 3D shape

So the question arose: what could be an application that makes use of this capability?
One nice idea is the ability to navigate an area or an object. See for example the Dichter and Bauer and GTS Future Energy apps that we contributed to. But of course, the applications don’t end with this. There are exciting gameplay scenarios that are just now becoming possible. For example, the ability to play a 3D puzzle game on your tabletop that uses this capability to create new types of puzzles and mechanisms.

A third application is the base for this experiment: what if it were possible to create directly in 3D space? ¬†This can be done in many ways… For example by giving the user the ability to create, pick up and place a variety of objects. These could then be stacked or clicked together. In this experiment an even more direct method is explored: the user can directly create and erase virtual ‘matter’ in the environment in front of him. This allows the user/artist to (in principle) create any shape he wants, without being hindered by physical constraints.

This experiment resulted in SculptAR*, and some pictures created with it are below.

Sculpt of a humanoid shape

The humanoid shape after some additions and removals of matter

The biggest challenge in creating an app like this is the design of a set of sculpting tools (and their associated mathematics) that feel natural, and allow the artist to actually express a vision. Currently there is only the ability to add and remove matter in a spherical volume of fixed size, in free space. To help the user get an understanding of where the matter will be added/removed, there is a pointer which indicates where the matter will appear. The pointer casts a shadow which helps a lot in gaining this understanding. In practice, after using the app a little bit, a sense of spatial awareness also develops.

The pointer with visible shadow cast on the object

Sculpt of an anemone

Some relatively easy extensions on the current (very basic) set of tools would be:

  • the ability to vary the size of the amount of matter added
  • the ability to vary the shape of the addition/removal tool (e.g. a cube shape, next to the current spherical shape)
  • the ability to use the existing matter as a base for addition/removal (instead of always working in free space)

Examples of advanced extensions would be:

  • ‘Real’ sculpting tools like ‘pinch’ and ‘flatten’
  • The ability to paint the surface, or more generally:
  • The ability to assign different materials to part of the surface (e.g. shiny, chrome, clay, luminous)
  • The ability to have an arbitrarily large volume to work in (right now this volume has a fixed size and resolution)
  • The ability to use stamping tools to work the surface

Such extensions haven’t been implemented in this experiment, but they are certainly possible.

Technical information

The experiment was developed with Unity Pro 4.3, using Vuforia with Persistent Extended Tracking. The ‘matter’ is created by generating an isosurface of a discrete 3D scalar field (i.e. a voxel grid), where the tesselation into a triangular mesh is performed in real-time using the Marching Cubes algorithm.

* This is the last time we use the ..AR version of a word for the name of an app, we promise!

2 comments on “SculptAR: 3D sculpting in AR

  1. Nabil on said:

    Hi, i can download this for UNity? is free?

    • Lex van der SluijsLex van der Sluijs on said:

      Yes, this has now been published as an app in the Apple App Store and Google Play Store called Clay: augmented sculpting, and it’s free!

      You can check out for more information and download-links.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

What is 7 + 15 ?
Please leave these two fields as-is:
IMPORTANT! To be able to proceed, you need to solve the following simple math (so we know that you are a human) :-)