Documentation

Hand interaction and physics

Let your audience reach into the scene: they can touch an object, pick it up and throw it with their own hand. Objects with physics fall, bounce a little and land.

What viewers can do

On a phone, AR Works follows your audience's hand in the camera picture. They point a finger at an object to touch it, pinch it or close their hand on it to pick it up, and open their hand to let go. If the hand is still moving as they let go, an object with Physics is thrown.

Viewers pick up and throw objects with their own hand.

A finger on the screen works as a hand too, on any phone and whether or not a hand is tracked: press an object to touch it, drag it to carry it, and flick it as you lift your finger to throw it. A quick tap still fires On Tap.

A Tracking image with Hands in front turned on also turns hand tracking on for its scene, without an Interactive object: a hand that reaches in front of your content is drawn in front of it, and a hand that only holds the print stays behind it. See Tracking images.

To tell whether a hand is in front of your content, the phone needs to know how big the picture really is. A Tracking image has a Printed width for this. A scene’s own target image is taken to be 20 cm wide: if yours is printed at another size, use a Tracking image for that picture and set its Printed width.

Make an object interactive

Select the object and turn on Collisions in the Inspector. It is the master switch: the settings for hands and physics appear only while it is on.

  • Interactive lets viewers touch and grab the object with a hand. One hand holds one object, and a held object is stopped by other objects with Collisions.
  • Physics gives the object gravity. It falls until it lands on the Floor or on another object with Collisions, bounces a little, and round objects roll to rest. Without Physics, an object stays where it is let go.
  • Can be pushed lets an open hand in front of the camera push the object along: a ball rolls, a box slides until the hand stops. It needs Physics. An object that can be pushed is harder to pick up, so turn it on only where pushing is the point.
  • Pause animation appears under Interactive. An animated object that is Interactive can be picked up: its animation keeps playing in the hand, turning around the point you hold, and when you let go it glides back onto its path. With Pause animation on, all of its animation stops while a hand is on it, and when you let go it glides back to where it was and carries on from there. Objects inside it stop with it too, and carry on when it does.
Collisions shows the rest: pick the Bounds, then tick Interactive and Physics.

To give falling objects something to land on, add a Plane shape and turn on Floor. The Floor is invisible in AR and hides anything below it, so content can sit on a real table or seem to come out of it. Each scene uses one Floor.

Note

An object with Physics that starts inside another one, such as a lid modelled inside a crate, stays put until it is lifted clear. A playing animation still wins over physics: an animated object does not fall when it is let go. An object inside another object takes no part in hand interaction or physics; its parent can. A press on it with the mouse or a finger picks up its parent when the parent is Interactive.

Note

On the Free plan a scene can have one Interactive object, and an object that a Set Interactive node turns on counts too. An object inside a tracking image or inside another object does not count. The editor says so when you try to add a second one, and a copy of the Interactive object comes with Interactive switched off. A project that already has more, for example from before a change of plan, keeps them and still plays them; you just cannot add another. When you import a .works file on Free, the first Interactive object of each scene stays Interactive and the others have it switched off. Pro, Education, teachers and students have no limit (see pricing).

Choose the shape a hand touches

With Collisions on, the Inspector shows the object's Bounds: the invisible shape that a hand touches and physics uses. The selected object's bounds are drawn in the viewport as a thin green wireframe.

  • The menu offers Box, Sphere, Cylinder, Capsule and Mesh. The Bounds menu starts on the shape's own type: Sphere for a sphere, Cylinder for a cylinder, Mesh for a cone or a torus, and Box for everything else, models included.
  • Turn on Edit bounds to resize the bounds with handles in the viewport, or type exact values such as the Radius. Fit to object removes your changes, so the bounds follow the object again.
  • Mesh uses the object's own shape. On an object that is Interactive or has Physics, a simplified wrap of the shape is used instead, so a crate built from a few cubes behaves better than one crate model with Mesh bounds.

Mesh bounds for a pipe or a bowl

For a ball to roll through a pipe or land in a bowl, the bounds need the same hollow shape, so choose Mesh. An object that never moves then uses its exact shape, up to 20,000 triangles an object and 60,000 in a scene. A more detailed one uses its box instead: Edit points, below, makes a simpler shape it can use. An object that is Interactive or has Physics can move, so it uses a simplified wrap, which is solid all the way through. Objects inside a Tracking image take no part in physics, so they never move.

Tip

Make a pipe a ball can fall through: select the pipe, turn on Collisions, choose Mesh bounds and leave Interactive and Physics off. A ball with Physics dropped in at the top falls through and comes out at the bottom.

Turn on Show collision bounds to see the shape that really collides, drawn in light green in the 3D view even when the object is not selected. Only you see it, in the editor: it never shows in Preview or on a phone. While it is on, the Inspector also offers these buttons:

  • Convert to mesh shape turns Box, Sphere, Cylinder or Capsule bounds into a Mesh with points you can move, at the same size and in the same place.
  • Edit points makes a simpler copy of a Mesh's shape, with up to 256 points you can move. It works on models of up to a million triangles.
  • Use object's shape throws your points away, so the Mesh follows the object again.

To move points, select only that object. Its points show as small dots: click a dot to pick it, or Shift+click to pick more or to drop one. Then move them with the transform tools, just as you move an object. Each move is one step you can undo. Press Escape or click empty space to drop the pick. Your points are saved with the project.

React with logic nodes

Nine nodes work with hands and physics. You add and wire them like any other node, as Interactivity (logic nodes) explains.

  • On Hand Touch and On Hand Touch End fire when a hand or finger starts and stops touching an Interactive object.
  • On Grab and On Release fire when an Interactive object is picked up and let go.
  • Set Physics and Set Interactive turn gravity and grabbing on or off for an object with Collisions.
  • Set Can be pushed lets an open hand push an object along, or stops it. The object needs Collisions and Physics.
  • Push Object gives an object with Physics a shove, and Reset Object puts an object back where the scene started it, at rest.

Tip

A box you can open: build a crate from a few cubes with Collisions, and give it a lid with Collisions, Interactive and Physics. Add a Plane with Floor turned on under the crate. Viewers can lift the lid off and drop or throw it, and an On Release node on the lid can play a sound.

Try it in Preview

In Preview, the mouse is the hand: press an Interactive object to touch it, drag it to pick it up and carry it, and flick it as you let go to throw it. A drag that starts on empty space still orbits the view, and physics and the Floor work as on a phone.

In a project that uses hands, click Use webcam to track your own hand with the computer's camera. A small mirrored camera picture and a ghost hand appear; pinch or close your hand on an object to pick it up. The camera turns on only when you click, and its pictures never leave your computer. Click again, or close the Preview, to turn it off.

On phones

  • Hand tracking needs a recent phone browser. Older iPhone and Android phones cannot run it: the viewer says so once, and a finger on the screen still works as a hand.
  • Tracking runs only in scenes that use hands. The first one downloads the hand tracker, which the phone then keeps, so it starts faster next time.
  • If the phone cannot keep the picture smooth, hand tracking switches itself off for the rest of the visit and says so once. The finger still works.

In a world tracking scene shown through the camera, the surface lock tries to keep your content pinned where it was placed, using the camera picture and the motion sensor. A white ring shows the surface a tap will use. It needs a level surface with visible detail, such as a desk with things on it, a floor or a worksheet. On a plain surface it says so, and the content turns with the phone but is not pinned. It does not map the room or work on walls.

Note

Hand tracking and the surface lock are new, and how well they work depends on the phone. The surface lock is still being tested, so do not count on pinned content on iPhone.

The camera stays on the device

Camera pictures are processed on the phone, or on the computer in Preview. Nothing is recorded, stored or sent. The hand tracker and the physics engine load from AR Works itself, not from another service.