FreeVR: Virtual Reality Integration Library
FreeVR
Tutorials
User Guide
Programming
Functions

FreeVR

TUTORIALS

DOWNLOADS

GUIDES
    Administrator
    User
    Programming
    Function List
    Library Dev
    Socket Interface

MAN PAGES


All materials
Copyright © 2015
William R. Sherman

FreeVR: Library Developer's Guide

FreeVR: Library Developer's Guide

March 20, 2015 for FreeVR Version 0.6f
Written by Bill Sherman

Intro

This guide covers how to compile the library for various operating systems, followed by a description of how to add new input devices to the library (e.g. the Flock of Birds), or how to add a new type of visual renderer (e.g. Java3D, OpenSG).

The goal is to see the library working on a number of platforms, with a variety of graphics libraries and input devices. This is imminently achievable with a little help from the open-source community. At the moment, the library handles the following systems:

Operating Systems

  • Linux
  • Macintosh OS/X (Darwin) w/ XFree86
  • MS Windows using Cygwin
  • IRIX (SGI)
  • Net, Free, Open BSD
  • Solaris (compiled by John Stone)
  • HP-UX (compiled by John Stone)
  • Tru64 (compiled by John Stone)

Input Devices

  • X-windows
  • Linux joydev input device
  • Linux evdev input device
  • Generic Shared Memory reader Works with many tracker daemons
  • VRUI's VRDeviceDaemon (aka "vruidd")
  • UNC's VRPN system
  • ART DTrack systems (eg. TrackPack, SMARTTRACK)
  • Ascension Flock of Birds
  • InterSense Ultrasonic trackers
  • Polhemus magnetic trackers
  • FakeSpace Pinch Glove input device
  • Virtual Technologies CyberGlove input device
  • Magellan 6-DOF input device
  • SpaceTec SpaceOrb 6-DOF input device
  • SpaceTec SpaceBall 6-DOF input device
  • Static — a device with hard-coded inputs

Graphics Libraries

  • OpenGL under X (GLX)
  • The OpenSceneGraph (on top of GLX)
  • OpenSG (on top of GLX)
  • OpenGL under Cygwin (GLX)
  • SGI's Performer (also uses GLX)

Along with the library, the FreeVR distribution includes two simple applications: "travel" and "inputs". These are both helpful for testing and also (especially in the case of "inputs") also can be a VR facility utility.

The "travel" application includes functions that manipulate user-travel through a virtual world, with other colorful objects (the majority of them) represented fixed locations in the real world, and a white pyramid affected by travel relative to the virtual world.

The "inputs" application simply gives a live presentation of all the inputs received by the configured virtual reality system. This alone has great value, but "inputs" also serves well as an input server/repeater.

In fact, there are still a few significant things to do, things to test, and things to do better that would be nice to have implemented/tested prior to declaring the library as version 1.0. However, we've reached the point where the library is stable and useful, and thus would like to give others a chance to add features to the library. The items on my significant todo list are:

  • Implement an interface to the Oculus Rift
  • Implement an interface to the Razer Hydra
  • Implement a configuration utility program
  • Enhance the documentation with a more complete set of manpages
  • Implement a cluster-ready version of FreeVR
  • Implement input controls that can influence other (non-self) devices
  • Fork the simulation process

The full todo list is of course much larger, but at the moment, those items dominate the list. In fact, the code is populated with comments containing the string "TODO:" which can be used for searching for all the little things yet to be done.

One nice feature of "inputs" is that it renders some text strings that indicate the current values of many of the inputs. Thus, it works reasonably well as a verification/debugging tool for testing new input devices.

The "travel" and "inputs" applications also print out a lot of information, including all the values of the current configuration — twice. Once after vrConfigure() is complete, and another after vrStart(). There will be some differences because vrStart() waits until all the devices are open, by which time additional information will be available.


Compiling the library

The first step is to type just "make" to get a list of what OSes are available, and then type "make <os>". For example, the 64-bit Linux 2.6 version can be compiled simply by typing "make linux2.6-glx-64". If your OS isn't listed in the output of "make", then try make fulllist for a more comprehensive list of alternatives.

All the library C source and header files begin with a "vr_" tag. The other files might be used to compile an example application, or are otherwise useful. The "valtest.c", "travel.c" and "drawing.c" files are for creating the basic example applications.


Adding A New Input Device

This is the one area in which I expect (hope) many people will engage in programming to expand the FreeVR library. There are lots of interface devices out there that can be used/useful for interacting in an immersive way with an environment, and I hope most of them can be used with the library.

In the very first implementation of FreeVR a distinction was made between devices that provided controller inputs (buttons, valuators, etc.) and devices that provided tracking inputs (6-DOF positions sensors, 22-DOF hand position sensors (ie. gloves), etc.). However, it was found that there are some devices for which it is hard to separate the two — the Magellan and Tracker Daemon for two. Therefore, the library does not distinguish between types of input devices, but each input device can provide a number of different inputs of a variety of types.

The file vr_input.skeleton.c provides the basic outline for creating a new input device for FreeVR. Beginning with this file, a large jump start can be made in the creation of a new input interface.

The programming interface for an input device involves writing approximately six functions:

  • Creating the input memory structures
  • Opening the device
  • Polling the device
  • Closing the device
  • Reseting the device
  • Telling FreeVR how to access the functions that do the above four things.

Access to the functions for Opening/Polling/Closing/Reseting the device is done via callbacks. The initialization routine is specified in the file vr_input.opts.h. Information about each device is stored in a vrInputDevice structure which is initialized by the "GetInfo" function, and used as the main reference to the device within the library. The generic input device structure also contains a pointer to memory that is entirely dealt with by the device handling routines. This space is for containing device specific information such as file-descriptors, shared memory pointers, and any mode settings that the device interface may have.

New devices are "created" during the configuration phase of running a FreeVR application. As the config file is read, a string containing the type name is placed in a newly created vrInputDevice structure. A call is then made to the generic function vrGetInputDeviceInfo() with a pointer to the structure. This function then loops over all known input devices, and upon finding a string match with the config string and a device name assigned in vr_input.opts.h, calls the associated "GetInfo" function.

The main purpose of the "GetInfo" function is to create the five callbacks for the other four operations, and perhaps assign a "pre-open" version string. Prior to when the "open" callback will be executed, the configuration routines will have placed the name of the device, and any arguments for opening the device, etc. into the "name" and "args" fields of the vrInputDevice structure for that device. The "args" field in particular will be used by the "open" routine to determine things like filename of the device, shared memory addresses, baud rates, etc.

There are two functions provided for creating callbacks: vrCreateCallback(), and vrCreateCallbackNamed(). The latter is useful when debugging the FreeVR library, which is often useful when adding a new input device. A FreeVR callback can have up to five arguments. For the input device operations, the pointer to the device structure and a constant for the operation are the only two arguments necessary. The arguments to vrCreateCallback() are the function, the number of arguments to the function, followed by the values of those arguments.

Here is the code for the function vrMagellanInitInfo:

void vrMagellanInitInfo(vrInputDevice *devinfo)
{
	devinfo->version = (char *)vrShmemStrDup("-get from Magellan-");
	devinfo->Create = vrCallbackCreateNamed("Magellan:Create-Def", _MagellanCreateFunction, 1, devinfo);
        devinfo->Open = vrCallbackCreateNamed("Magellan:Open-Def", _MagellanOpenFunction, 1, devinfo);
        devinfo->Close = vrCallbackCreateNamed("Magellan:Close-Def", _MagellanCloseFunction, 1, devinfo);
        devinfo->Reset = vrCallbackCreateNamed("Magellan:Reset-Def", _MagellanResetFunction, 1, devinfo);
        devinfo->PollData = vrCallbackCreateNamed("Magellan:PollData-Def", _MagellanPollFunction, 1, devinfo);
        devinfo->PrintAux = vrCallbackCreateNamed("Magellan:PrintAux-Def", _MagellanPrintStruct, 0);
}

Resetting and closing a device tend to be fairly straightforward, so the main focus is on the functions necessary to Open and Poll an input device. The "PrintAux" function is another useful debugging aid. It allows one to find out the internals of the private (auxiliary) data structure within the input device driver.

Opening an Input Device

The first step is to create the structure containing the "auxiliary" data -- data that is private to this particular type of input device. By convention, we typedef this structure to a type name: _<input-type>PrivateInfo. This is where information such as the file descriptor, port name, etc. are stored.

Before a device can be opened, there will likely be some arguments that must be parsed to determine things like what serial port, or shared memory segment must be used to communicate with the device. Often there are other arguments too, such as the communications baud rate. The values of these arguments are typically stored in the private structure of the device.

The VR shared memory allocation routine (vrShmemAlloc0()) is used to create the structure memory, and is assigned both to the aux_data field of the device info struct, and to an auxiliary variable with the appropriate typecast for the private data.

	devinfo->aux_data = (void *)vrShmemAlloc0(sizeof(vrMagellanPrivateInfo));
	aux = (vrMagellanPrivateInfo *)devinfo->aux_data;
Note, devinfo is a pointer to the device's vrInputDevice structure.

Now, the arguments can be parsed and placed into this structure. There are some generic parsing routines for reading strings, integers, booleans and Floats. In the future, there will be more generic functions to accommodate the needed arguments by new input devices.

The current list of available argument parsing routines is:

	vrArgParseBool(<string of all args>, <arg name>, <storage variable>)
	vrArgParseInteger(<string of all args>, <arg name>, <storage variable>)
	vrArgParseFloat(<string of all args>, <arg name>, <storage variable>)
	vrArgParseString(<string of all args>, <arg name>, <storage variable>)
	vrArgParseFloatList(<string of all args>, <arg name>, <storage array>, <max size of array>)
	vrArgParseChoiceInteger(<string of all args>, <arg name>, <storage variable>, <choice names>, <choice values>)
The arguments for most of these are straightforward: the string containing all the given arguments for the particular device is first, followed by a string with the name of the argument being parsed, followed by a pointer to where the value should be stored (ie. a call-by-reference variable). All the functions return 1 if the argument was found, or 0 if not found. For example:
	if (vrArgParseFloat(args, "rotscale", &scale_value))
		aux->scale_trans = scale_value * TRANS_SENSITIVITY;

The functions vrArgParseBool() and vrArgParseInteger() both set the value of an integer, but the difference is that vrArgParseBool() will interpret the text strings of "on", "true" and "yes" as a 1, and "off", "false" and "no" as 0.

The function vrArgParseChoiceInteger() allows input devices to be given arguments that are specified from a list of strings. For example, the SpaceOrb input device can conveniently be held with the ball oriented upward, or oriented away from the user. To provide this choice in the config file, the following code is used:

	static  char    *orient_choices[] = { "up", "away", NULL };
	static  int     orient_values[] = { 1, 0 };

		vrArgParseChoiceInteger(args, "orient", &(aux->orient_up), orient_choices, orient_values);

Note that the list of choice strings is NULL terminated, and the list of values is shorter by one, as the NULL termination is not required. Also note that if the argument string "orient" is not found, or if none of the choices match the argument value, then the value of aux->orient_up will not be affected, and 0 will be returned.

Next, open the device — left as an exercise for the library developer. However, as a convenience, we now have some generic serial-port and socket handling routines that may make some of this easier. The main reason for doing this is to enable the device opening routines to work across various operating systems. Also, if the version of the device can be obtained, then copy this string to the version filed of the device structure.

There are plans for a variety of types of inputs possible. Thus far four types are implemented:

  • vr2switch — Binary switches (aka 2-way switches)
  • vrNswitch — N-ary switches (aka multi-pole switches)
  • vrValuator — Valuators
  • vr6sensor — 6-DOF Position Sensors

The "devinfo" vrInputDevice structure contains fields for storing the number of each type of input for a device, as well as a pointer to an array of structures for each type of input. So for each type, the number must be assigned, as well as a memory structure created and initialized. The function vrInputCreateType() creates and initializes arrays of the requested type of input.

Once all this is done, set the ready field to true, and return. If the ready field is never set to true, the app will hang, since it waits for all input devices to be ready before mapping physical inputs to logical inputs. In the Magellan and TrackD code, we use a private open field to set when the device is actually open, and then after all the input types have been added, set the ready field to match this:

	info->ready = aux->open;

Polling an Input Device

The most complicated part of adding the polling code of an input device to the FreeVR library is writing the code to communicate with the device, and parse it into input values. The task of assigning the incoming values to the structures used within FreeVR is fairly straightforward.

So, once you have the input values from the device, they can be assigned to the input structure with the vrAssign<type>Value() functions. These functions take two arguments (perhaps three in the future): a pointer to the specific input structure, and the new value (and perhaps an optional timestamp in the future).

Using these special assignment functions allows the library to constantly handle stuff like adding a timestamp when not given, calculating a delta-change for use by applications, putting the input on a queue if requested, and for inputs like sensors, doing the extra calculations that transform the raw input values into values more useful by the library and applications.

Recompiling the Library

Once the functions are written, it is pretty easy to add the new device to the library. The convention is to place all these functions into a file named vr_input.<input-type>.c without the need for a separate header file. You don't have to conform to this of course, but don't be surprised if it's been renamed in future releases of the library source.

Edit the Makefile, add the ".c" source file to the list of library source files — called FREEVR_SRC. Don't forget that every line in the list should end in a backslash — except the last line.

Edit the file vr_input.opts.h: add a function declaration for the vr<input-type>InitInfo() function, and add a new entry to the vrInputOpts structure in the manner of the current examples — ie. two entries: a string with the name of your device-type, and the name of your function for setting up the device callbacks. The only restriction is that it should precede the all NULL entry.

Here is an example for a library compiled with only the static device:

void vrStaticInitInfo(vrInputDevice *info);

#ifdef VRINPUT_ONLY
vrInputOptsType vrInputOpts[] = {
		{ "static",     vrStaticInitInfo },
		{ NULL, NULL }
	};
#endif

Recompile, and your device should now be included into the library.


Adding A Graphics Library Interface

Our original intent was to have a library that was designed for multiple graphics libraries, but we hadn't had the opportunity to hash out a good solution, so now that we have a second graphics library interface (Performer), we have separate libraries for each. The CAVE library[tm] also does this by having multiple libraries, one for each type of graphics interface — IrisGL, OpenGL, Performer.

For each graphics library, there are nine functions that the FreeVR library will need to call — one of which is just to setup a few callbacks and set the version string. These functions are (with the GLX declaration as example):

  • Get the window info (sets up the callbacks)
    void vrGlxGetWindowInfo(vrWindowInfo *);
  • Open a window for the screen
    void vrGlxOpenFunc(vrWindowInfo *info);
  • Close the window
    void vrGlxCloseFunc(vrWindowInfo *info);
  • Render to a window
    void vrGlxRenderFunc();
  • Render a NULL (empty) world
    void vrGlxRenderNullWorld();
  • Render the default (basic) simulator extras
    void vrGlxRenderDefaultSimulator();
  • Render text
    void vrGlxRenderText(char *string);
  • Place the User-travel transform on the stack
    vrGlxRenderTransform();
  • Swap between the Front and Back frame buffers
    vrGlxSwapFunc(vrWindowInfo *info);

Similar to the input device "GetInfo" routine, here is what the GLX vrGlxInitWindowInfo() function looks like:

void vrGlxInitWindowInfo(vrWindowInfo *info)
{
	info->version = (char *)vrShmemStrDup("GLX render window");
	info->Open = vrCreateCallback(_GlxOpenFunc, 1, info);
	info->Close = vrCreateCallback(_GlxCloseFunc, 1, info);
	info->Render = vrCreateCallback(_GlxRenderFunc, 0);
	info->RenderText = vrCreateCallback(_GlxRenderText, 0);
	info->RenderNullWorld = vrCreateCallback(_GlxRenderNullWorld, 0);
	info->RenderSimulator = vrCreateCallback(_GlxRenderDefaultSimulator, 0);
	info->RenderTransform = vrCreateCallback(_GlxRenderTransform, 0);
	info->Swap = vrCreateCallback(_GlxSwapFunc, 1, info);
	}

By convention, all these functions will be contained in a single C source file named in the form of vr_visren.<type>.c with a similar header file ending in ".h".

Opening a Window for Visual Rendering

One of the first things that will need to be addressed is to create a private data structure with info that pertains only to this type of visual renderer — eg. for X windows this includes the X host, server & screen values, as well as a pointer to the X "Display" structure for each particular screen. A routine for parsing the particular window options will need to be written to put the information into the public and private fields of the vrWindowInfo structure. The allocation of memory for the private data, and the parsing of arguments are typically the first thing done by the open-window routine.

Most of the parsing of the desired geometry, etc. should be pretty similar to what is in vr_visren.glx.c, so you can copy that code. For actually opening the windows and stuff, that's up to you.

Once the window is open, then you should invoke the user defined callback for initializing the graphics. This is the VisrenInit callback. Here is what the GLX version does:

	/* call global visren init function */
	callback = info->VisrenInit;
	if (!callback)
		callback = vrConfig->VisrenInit;
	vrInvokeCallback (callback);

	if (aux->doub_buf)
		glXSwapBuffers (aux->dpy, aux->win);
	else    glFlush ();

Closing the window

Closing the windows should make the calls that will remove the windows from the screen, and free the shared memory used by the private data structure.

The Rendering Process

The rendering loop is a multi-stage process of handling all the things that need to be done each frame. It would be possible to make the render process free of direct references to the graphics libraries, with callbacks for those functions that are specialized, versus the way it is currently implemented, which is that all stages are done directly in the specialized rendering function. The perspective matrix for the window is pre-calculated and passed to the graphics-specific renderer as an argument.

The rendering stages are:

  • (i) push gfx matrix/state
  • (ii) handle viewport — only partially implemented for GLX
  • (iii) handle color masking
  • (iv) handle viewmask — not currently implemented for GLX
  • (v) put perspective matrix on the render stack
  • (vi) call world_render
  • (vii) call simulator_render (if simulator window)
  • (viii) display the frame rate when flagged
  • (ix) restore gfx matrix/state — may also want to flush the pipe

For this, it might be best to look at the code in vr_visren.glx.c and follow along making changes where necessary, but I'll try to provide some commentary.

(i) push gfx matrix/state

This is the initialization stuff, including setting the eye buffers for multi-buffer stereo, setting the initial projection matrices to some innocuous default, and pushing the matrix stack.

(ii) handle viewport

The viewport is only partially implemented for GLX at the moment. For Full-mono, and left/right buffer stereo rendering, the proper values are set, but for stereo using partial window viewports, it has not yet been implemented.

(iii) color mask

The color rendering information associated with the current eye being rendered is used to set the glColorMask(). This is used for Anaglyphic stereoscopic rendering.

(iv) viewmask

The viewmask code has not yet been implemented for GLX (or any other library for that matter, since GLX is the only graphics library even partially implemented).

(v) put perspective matrix on the render stack

A call to vrCalcPerspMatrix() will calculate The proper perspective matrix for a particular window/eye pair is passed the graphics-specific render function. The matrix should be placed onto the graphics stack. This is then followed by multiplying the transform from window space to world space (as setup in the configuration phase). Typically for graphics systems that separate the projection from the model-view rendering matrices, the perspective matrix should be placed on the projection stack. However in some cases (such as hardware-fog rendering), it is better to put the perspective directly on the model-view stack. A future version of the GLX implementation will allow the programmer to specify which method should be used.

The calculation of the matrix in vrCalcPerspMatrix() is based on the type of mounting for each particular window. Choices are head-mounted, floor-mounted (ie. fixed), hand-mounted (ie. floating), and simulator. The only two implemented so far are the fixed and simulator methods. We had been using the method described by Deering in his SIGGRAPH '92 for the fixed windows, but when the Performer interface was being written we discovered that it is better to define the perspective as the frustum, not a full 4x4 matrix, so now we do a similar calculation based on some paper scribblings.

(vi) call world_render

Simply a callback to the application's render process (which may be based on user and/or window). The call to the render is surrounded by a push/pop matrix call to make sure the simulator rendering to follow will be in the correct position.

(vii) call simulator_render

If the particular window that is being rendered is of type "simulator," then in addition to getting its own perspective matrix, it also can render some additional features into the virtual world. At the moment, this feature is a head that floats around based on the location of the head of user #1.

There are some crude keyboard command that can be used to adjust the view position looking into the simulated world. This more or less mimic the numeric keypad keystrokes available for the CAVE simulator. These are:

  • '+' — move away from the origin
  • '-' — move toward the origin
  • '5' — reset view to center position
  • '7' — reset view to home position
  • 'Enter' — set current view as home position
  • '4' — rotate about the Y axis
  • '6' — rotate about the Y axis
  • '8' — rotate about the X axis
  • '2' — rotate about the X axis

  • '*' — toggle the simulator view
  • '/' — toggle the FPS display (also available in non-simulator windows)

As implemented in the GLX code, there is nothing specific to GLX, so it can just be cut and pasted into other methods.

(viii) display the frame rate when flagged

This simply puts a short text message on the window as a configured location that indicates the current frame rate for the current window. This portion of the rendering process can be disabled by a key command.

(ix) restore gfx matrix/state

Restore (ie. pop) the graphics matrix and/or state to how things were at the beginning of this iteration through the loop.

(x) swapbuffers or flush pipe

This step is actually handled through a separate callback that is called just before the next frame begins. This is done because it needs to be done at the same time for all render processes, so it is done after all processes in a sync-group are synced. For double-buffer windows, a swap buffer is performed, otherwise a simple flush of the graphics pipe is done (ie. all the commands in the queue are pushed through).

Rendering a Null World

The code for rendering a null (ie. empty) world is pretty simple. It's just a matter of clearing the window and returning. The reason this is necessary is that the render function itself is not responsible for clearing the window — since an application may have special methods for doing so. Thus, prior to the application's world rendering taking effect, the null world provides a way to keep the windows clear. It can also be used for other occasions is so desired.

Rendering for a Simulator View

In the CAVE library, simulator windows include an outline for the CAVE, a sphere for the head, and a representation of the wand. Each can be toggled off and on. The FreeVR library will provide a similar default view (outline, head & hand-held prop), but will allow the application programmer to entirely replace the routine for rendering the simulator. For the moment, the default GLX simulator renders a brown head with a nose, but it could use a lot of work.

Rendering Text

Text rendering is also handled specially because often there is special work for handling fonts, etc. required for text. Often it is necessary to store font information within the private data structure. Therefore, it is best to isolate this mess from the application programmer and handle it within the library, providing a special API.

Recompiling the Library

Once the functions are written, it is pretty easy to add the new device to the library. The convention is to place all these functions into a file named vr_visren.<visren-type>.c with a similarly named header file (ending in ".h"). You don't have to conform to this of course, but don't be surprised if it's been renamed in future releases of the library source.

Edit the Makefile, add the ".c" source file to the list of library source files — called FREEVR_SRC. Don't forget that every line in the list should end in a backslash — except the last line.

Edit the file vr_visren.opts.h: add a #include statement naming your header file, and add a new entry to the vrVisrenOpts structure in the manner of the current examples — ie. two entries: a string with the name of the graphics library, and the name of the function for setting up the device callbacks. All the other necessary functions are accessed via these callbacks (open window, close window, render world, null world, default simulator and text rendering function). The only placement restriction is that it should precede the all NULL entry.

Recompile, and the new graphics interface should now be incorporated into the library. Of course, to test this you'll have to modify the configuration settings to use the new interface.

Conclusion

Well, a little strange to call this a conclusion, given that development continues, but here we are at the end of the library programming guide document.

Obviously, not everything is implemented yet, however, there is enough that the library is quite usable, and even fairly stable. There are still ways in which the library can benefit from extensions, but it is complete in that full immersive applications can be developed with it.

So, comments on coding ideas and this document are very welcome. And code to implement new devices and graphics libraries is more than welcome — ie. we're requesting that you send us some.



© Copyright William R. Sherman, 2015.