MTouch Multi-touch SDK

The mTouch is the next generation of the world's most innovative multi-touch devices. Its unique capabilities and powerful technologies will change your tech lifestyle forever.

2

What is Tuio?

TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states. This protocol encodes control data from a tracker application (e.g. based on computer vision) and sends it to any client application that is capable of decoding the protocol. There exists a growing number of TUIO enabled tracker applications and TUIO client libraries for various programming environments, as well as applications that support the protocol. This combination of TUIO trackers, protocol and client implementations allow the rapid development of table based tangible multitouch interfaces. TUIO has been mainly designed as an abstraction for interactive surfaces, but also has been used in many other related application areas. Technically TUIO is based on Open Sound Control - an emerging standard for interactive environments not only limited to musical instrument control - and can be therefore easily implemented on any platform that supports OSC.

TUIO (Table-Top User Interfaces Objects) is a simple yet versatile protocol designed specifically to meet the requirements of table-top tangible user interfaces. Inspired by the idea of interconnecting various existing interactive surface interfaces such as the reacTable or AudioTouch. The TUIO protocol has been implemented using OpenSound Control and is therefore usable on any platform supporting this protocol.

Simply put: TUIO is just a standardized way of formatting OSC data.

TUIO Documentation (pdf)

Message Syntax:


Since TUIO is implemented using OSC it follows its general syntax:

/tuio/[profileName] set sessionID [parameterList] 
/tuio/[profileName] alive [list of active sessionIDs] 
/tuio/[profileName] fseq int32

Parameters:


The parameters defined in this section reflect the object properties we considered important for an interactive surface interface. A session ID number is assigned to each object. This is necessary to uniquely identify untagged objects across successive frames, and in the case where multiple objects tagged with the same classID are simultaneously present on the surface.

Some of these parameters (id, position and angle) are retrieved directly by the sensor. Others (speed, acceleration) are derived from these primary parameters using timing information. Computing these parameters on the low level side of an tangible user interface system allows a more efficient computation, since the necessary timing information does not need to be transferred to clients.

The parameters allowed in a set message:

s        sessionIDtemporary ob ject IDint32 
i        classID
fiducial ID numberint32 
x
yz        positionfloat32range 0...1 
a
bc        anglefloat32range 0..2PI 
X
,Z        movement vector (motion speed &ampdirection), float32 
A
BC        rotation vector (rotation speed &ampdirection), float32 
m        motion acceleration
float32 
r        rotation acceleration
float32 
P        free parameter
type defined by OSC packet header

 

Profiles:


We define a set of profiles, which apply to most table-style tangible user interfaces. This allows the tracking of objects and cursors on two dimensional surfaces and in special cases also in the 3D space above the table surface. If one of these predefined profiles does not meet a system’s requirements we also allow so-called raw profiles that send the raw sensor data, as well as free form profiles, which allow a user defined set of parameters to be transmitted.

2D Interactive Surface:

/tuio/2Dobj set s i x y a X Y A m r 
/tuio/2Dcur set s x y m r

3D Interactive Surfaces:

/tuio/3Dobj set s i x y z a X Y Z A m r 
/tuio/3Dcur set s x y z m r

Custom Profile:

/tuio/_[formatString] 
/tuio/_ixyP set s i x y 0.57

 

TUIO Homepage
OSC Homepage
Multitouch TUIO Simulator

TouchLib project

 

 

 

 

Building Applications for the mTouch

 

To build applications that function with the mTouch, you need to use a framework that is compatible with the mTouch System. The mTouch can launch and manage any .NET and/or  Flash /Adobe Air applications

 

 

 

4

 

 

.Net Applications

Breeze Framework: (Download) (Help File)

Breeze for WPF 3.5 is a full core framework for multi-touch showing all the internals. Unlike Windows 7, you can see how the rotate, move and scale actions are interpreted, how the centre of rotation is calculated and change it all to improve and extend.

Breeze has a load of cool features, some simple docs, a load of source code and a pretty example photo browser. Breeze for WPF 3.5 is ideal as a core for building multi-touch applications in WPF for Windows XP/Vista and 7, or as a reference implementation to create a new multi-touch application from scratch.

 

Getting Started

 

The Download link for the Breeze Framework provided above gives you a sample application called “TouchExample” that shows you how to scale, rotate, move, remove and add objects with the multitouch framework right out of the box.  This is a great sample and headstart to learning how to use the Breeze Framework.

 

Breeze Framework Sample Code

 

//Starts the breeze Framework

framework = TrackingHelper.GetTracking(canvas1, currentTrackingType);

framework.OnProcessUpdates += new FrameworkControl.ProcessUpdatesDelegate(this.DisplayPoints);

framework.Start();

 

//Registering an item for multitouch

ElementProperties sb_prop = new ElementProperties();

sb_prop.ElementSupport.AddSupport(TouchFramework.TouchAction.Slide | TouchFramework.TouchAction.Tap);

MTSmoothContainerRev sb_cont = new MTSmoothContainerRev(SLIDING_CLOSE_BUTTON, canvas1, sb_prop);

framework.RegisterElement(sb_cont);

SLIDING_CLOSE_BUTTON.ButtonDragged += new

 

//Add an event

SlidingButton.ButtonDraggedHandler(SLIDING_CLOSE_BUTTON_ButtonDragged);

 

//Event Function

private void SLIDING_CLOSE_BUTTON_ButtonDragged()

{

 

    //Event fired here

}

 

6

Adobe Flash/Air:

 

Requirements:


Flash Player 9+ (Download)
Java Runtime 6+ (Download)
Adobe Air (Download)

 

1) Download the attached file and extract to your desktop.
2) Open Flosc.jar (then click start)
3) Open Simulator.jar
4) Open Test.exe
5) Go back to Simulator.jar and draw within the circle, you will see your actions replicated into Test.exe
5a) Use SHIFT+Click for multi-touch events. (See Simulator Help for more info)

 

Explanation:

 


8

 

Step by Step:


1) The TUIO (OSC) data is SENT from the Simulator OR Touchlib’s “osc.exe”.

2) The FLOSC gateway CONVERTS the OSC (UDP) data into XML (TCP)

3) The Flash client uses an XMLSocket to RECEIVE and parse the XML data then renders the cursors within your application.

 

Notes:

  • Host: localhost ---- Ports: 3000/3333
  • Remember TUIO and OSC are referring to the SAME data.
  • The reason we need Step 2 is because Flash CANNOT read UDP data, thus we convert it to TCP
  • The Direct Binary method is not covered in this tutorial. But you can read more about it here:
    Binary Socket Connection in Flash.

 

Flash Tuio Libraries

AS3: alternative Flash AS3 TUIO/FlashTCP library (by Dean North)

AS3: alternative Flash AS3 TUIO library (by Ignacio Delgado)

AS3: alternative Flash AS3 TUIO library (by Johannes Luderschmidt)

 

Sample Applications and Sources

Getting Started with Flash Development and AS3

Building Your First Multitouch App in Flash

AS3 Lbraries & APIs

Digital Music Volvelles

Mapstouch (A Google Maps Multi-touch Application)

Flickr multi-touch photo viewing application

 

 

Allowing users to update application settings online

Before your application can communicate, and get user defined information from Mereltechnologies.com such as background color, images, videos, audio and other types of uploaded content and application settings, you will need to specify the parameters your application will accept when submitting your app for approval in the mStore. Upon approval of your application the mTouch team will set up your user defined variables for your application. When the application is downloaded and a user logs in or registers an account with Mereltechnologies.com, the user will be able to configure each one of those settings online. When your application is launched on the user’s mTouch and the user is logged in to their mStore Account all setting and files uploaded to the users account from the online portal will be downloaded and the application will start with the new settings and files.

 

Receiving Application Setting

When your application is launched and the user is also logged in to their mStore Account. The application settings are sent to your app via arguments. You can retrieve these arguments like this

Sample Arguments: "LOGGED_IN=true MAX_IMAGES=5 MAX_VIDEOS=1 MAX_MAGAZINES=1 ENABLE_KEYBOARD=yes BACKGROUND_COLOR=#000000 BROWSER_HOMEPAGE=http://www.yahoo.com";

 

C# Code

public Boolean LOGGED_IN = false;

private String BACKGROUND_COLOR = "";

void Window_Loaded(object sender, RoutedEventArgs e)

{

string[] args = Environment.GetCommandLineArgs();

 

if (args.Length > 1)

 {

                //Process the new configuration and save to configuration file.

                foreach (string arg in args)

                {

                    String[] s = arg.Split('=');

 

                    if (s[0] == "LOGGED_IN")

                    {

                        if (s[1] == "true")

                        {

                        LOGGED_IN = true;

                        }else{

                        LOGGED_IN = false;

                        }

 

                    }

               }

 

               if (s[0] == "BACKGROUND_COLOR")

               {

                        BACKGROUND_COLOR = "#" + s[1];                    

               }

 

}

}

 

Simulators

SimTouch Instructions:


1) Download and install SimTouch from http://code.google.com/p/simtouch/
2) Watch the video here on how to use SimTouch
http://www.youtube.com/watch?v=MkRXr2Y85c0
3) Use SHIFT+Click for multi-touch events. (See Simulator Help for more info)

SimTouch is an Adobe Air application that simulates native touch events on the Flash platform. SimTouch is a transparent app that sits on top of the application that you are developing and dispatches, TouchEvents to your application through a connection class.

Currently SimTouch only dispatches TOUCH_BEGIN, TOUCH_MOVE, and TOUCH_END events.

TUIO Simulator: (Download)