Documentation
Hand interaction and physics
Let your audience reach into the scene: they can touch an object, pick it up and throw it with their own hand. Objects with physics fall, bounce a little and land.
What viewers can do
On a phone, AR Works follows your audience's hand in the camera picture. They point a finger at an object to touch it, pinch it or close their hand on it to pick it up, and open their hand to let go. If the hand is still moving as they let go, an object with Physics is thrown.
A finger on the screen works as a hand too, on any phone and whether or not a hand is tracked: press an object to touch it, drag it to carry it, and flick it as you lift your finger to throw it. A quick tap still fires On Tap.
A Tracking image with Hands in front turned on also turns hand tracking on for its scene, without an Interactive object: a hand that reaches in front of your content is drawn in front of it, and a hand that only holds the print stays behind it. See Tracking images.
To tell whether a hand is in front of your content, the phone needs to know how big the picture really is. A Tracking image has a Printed width for this. A scene’s own target image is taken to be 20 cm wide: if yours is printed at another size, use a Tracking image for that picture and set its Printed width.
Make an object interactive
Select the object and turn on Collisions in the Inspector. It is the master switch: the settings for hands and physics appear only while it is on.
- Interactive lets viewers touch and grab the object with a hand. One hand holds one object, and a held object is stopped by other objects with Collisions.
- Physics gives the object gravity. It falls until it lands on the Floor or on another object with Collisions, bounces a little, and round objects roll to rest. Without Physics, an object stays where it is let go.
- Can be pushed lets an open hand in front of the camera push the object along: a ball rolls, a box slides until the hand stops. It needs Physics. An object that can be pushed is harder to pick up, so turn it on only where pushing is the point.
- Pause animation appears under Interactive. An animated object that is Interactive can be picked up: its animation keeps playing in the hand, turning around the point you hold, and when you let go it glides back onto its path. With Pause animation on, all of its animation stops while a hand is on it, and when you let go it glides back to where it was and carries on from there. Objects inside it stop with it too, and carry on when it does.
To give falling objects something to land on, add a Plane shape and turn on Floor. The Floor is invisible in AR and hides anything below it, so content can sit on a real table or seem to come out of it. Each scene uses one Floor.
Note
Note
Choose the shape a hand touches
With Collisions on, the Inspector shows the object's Bounds: the invisible shape that a hand touches and physics uses. The selected object's bounds are drawn in the viewport as a thin green wireframe.
- The menu offers Box, Sphere, Cylinder, Capsule and Mesh. The Bounds menu starts on the shape's own type: Sphere for a sphere, Cylinder for a cylinder, Mesh for a cone or a torus, and Box for everything else, models included.
- Turn on Edit bounds to resize the bounds with handles in the viewport, or type exact values such as the Radius. Fit to object removes your changes, so the bounds follow the object again.
- Mesh uses the object's own shape. On an object that is Interactive or has Physics, a simplified wrap of the shape is used instead, so a crate built from a few cubes behaves better than one crate model with Mesh bounds.
Mesh bounds for a pipe or a bowl
For a ball to roll through a pipe or land in a bowl, the bounds need the same hollow shape, so choose Mesh. An object that never moves then uses its exact shape, up to 20,000 triangles an object and 60,000 in a scene. A more detailed one uses its box instead: Edit points, below, makes a simpler shape it can use. An object that is Interactive or has Physics can move, so it uses a simplified wrap, which is solid all the way through. Objects inside a Tracking image take no part in physics, so they never move.
Tip
Turn on Show collision bounds to see the shape that really collides, drawn in light green in the 3D view even when the object is not selected. Only you see it, in the editor: it never shows in Preview or on a phone. While it is on, the Inspector also offers these buttons:
- Convert to mesh shape turns Box, Sphere, Cylinder or Capsule bounds into a Mesh with points you can move, at the same size and in the same place.
- Edit points makes a simpler copy of a Mesh's shape, with up to 256 points you can move. It works on models of up to a million triangles.
- Use object's shape throws your points away, so the Mesh follows the object again.
To move points, select only that object. Its points show as small dots: click a dot to pick it, or Shift+click to pick more or to drop one. Then move them with the transform tools, just as you move an object. Each move is one step you can undo. Press Escape or click empty space to drop the pick. Your points are saved with the project.
React with logic nodes
Nine nodes work with hands and physics. You add and wire them like any other node, as Interactivity (logic nodes) explains.
- On Hand Touch and On Hand Touch End fire when a hand or finger starts and stops touching an Interactive object.
- On Grab and On Release fire when an Interactive object is picked up and let go.
- Set Physics and Set Interactive turn gravity and grabbing on or off for an object with Collisions.
- Set Can be pushed lets an open hand push an object along, or stops it. The object needs Collisions and Physics.
- Push Object gives an object with Physics a shove, and Reset Object puts an object back where the scene started it, at rest.
Tip
Try it in Preview
In Preview, the mouse is the hand: press an Interactive object to touch it, drag it to pick it up and carry it, and flick it as you let go to throw it. A drag that starts on empty space still orbits the view, and physics and the Floor work as on a phone.
In a project that uses hands, click Use webcam to track your own hand with the computer's camera. A small mirrored camera picture and a ghost hand appear; pinch or close your hand on an object to pick it up. The camera turns on only when you click, and its pictures never leave your computer. Click again, or close the Preview, to turn it off.
On phones
- Hand tracking needs a recent phone browser. Older iPhone and Android phones cannot run it: the viewer says so once, and a finger on the screen still works as a hand.
- Tracking runs only in scenes that use hands. The first one downloads the hand tracker, which the phone then keeps, so it starts faster next time.
- If the phone cannot keep the picture smooth, hand tracking switches itself off for the rest of the visit and says so once. The finger still works.
In a world tracking scene shown through the camera, the surface lock tries to keep your content pinned where it was placed, using the camera picture and the motion sensor. A white ring shows the surface a tap will use. It needs a level surface with visible detail, such as a desk with things on it, a floor or a worksheet. On a plain surface it says so, and the content turns with the phone but is not pinned. It does not map the room or work on walls.
Note
The camera stays on the device
Camera pictures are processed on the phone, or on the computer in Preview. Nothing is recorded, stored or sent. The hand tracker and the physics engine load from AR Works itself, not from another service.