SculptAR: 3D sculpting in AR
One of the unique capabilities of augmented reality on smartphones and tablets is that, for the first time in the history of the world, hundreds of millions of people carry a device in their pockets capable of “6DOF” tracking. 6DOF stands for 6 Degrees of Freedom: the device’s movements can be tracked when it rotates around three axes, and when moves along these axes.
So the question arose: what could be an application that makes use of this capability?
One nice idea is the ability to navigate an area or an object. See for example the Dichter and Bauer and GTS Future Energy apps that we contributed to. But of course, the applications don’t end with this. There are exciting gameplay scenarios that are just now becoming possible. For example, the ability to play a 3D puzzle game on your tabletop that uses this capability to create new types of puzzles and mechanisms.
A third application is the base for this experiment: what if it were possible to create directly in 3D space? This can be done in many ways… For example by giving the user the ability to create, pick up and place a variety of objects. These could then be stacked or clicked together. In this experiment an even more direct method is explored: the user can directly create and erase virtual ‘matter’ in the environment in front of him. This allows the user/artist to (in principle) create any shape he wants, without being hindered by physical constraints.
This experiment resulted in SculptAR*, and some pictures created with it are below.
The biggest challenge in creating an app like this is the design of a set of sculpting tools (and their associated mathematics) that feel natural, and allow the artist to actually express a vision. Currently there is only the ability to add and remove matter in a spherical volume of fixed size, in free space. To help the user get an understanding of where the matter will be added/removed, there is a pointer which indicates where the matter will appear. The pointer casts a shadow which helps a lot in gaining this understanding. In practice, after using the app a little bit, a sense of spatial awareness also develops.
Some relatively easy extensions on the current (very basic) set of tools would be:
- the ability to vary the size of the amount of matter added
- the ability to vary the shape of the addition/removal tool (e.g. a cube shape, next to the current spherical shape)
- the ability to use the existing matter as a base for addition/removal (instead of always working in free space)
Examples of advanced extensions would be:
- ‘Real’ sculpting tools like ‘pinch’ and ‘flatten’
- The ability to paint the surface, or more generally:
- The ability to assign different materials to part of the surface (e.g. shiny, chrome, clay, luminous)
- The ability to have an arbitrarily large volume to work in (right now this volume has a fixed size and resolution)
- The ability to use stamping tools to work the surface
Such extensions haven’t been implemented in this experiment, but they are certainly possible.
The experiment was developed with Unity Pro 4.3, using Vuforia with Persistent Extended Tracking. The ‘matter’ is created by generating an isosurface of a discrete 3D scalar field (i.e. a voxel grid), where the tesselation into a triangular mesh is performed in real-time using the Marching Cubes algorithm.
* This is the last time we use the ..AR version of a word for the name of an app, we promise!