SoftspaceAR Design Paper

An initial design for a spatial knowledge management tool

For the past few weeks I’ve been gearing up to build the first MVP of SoftspaceAR. My focus during this period was writing a paper that described what the product will be, why it works the way it does, who it’s for, and how they will use it. Today, I’d like to share that document with you.

Between SoftspaceVR and the prototypes I’ve built and released this year, we have a ton of conceptual, design, and technical building blocks to draw from. But there wasn’t any one thing I could point when somebody asked, “So what are you building, actually?”

More and more people have been asking this! My tweeting and writing has earned us some attention in various corners of the internet, and several very interesting opportunities have come our way. For those who ask, this document is my working answer to that question.

Most importantly, it’s a roadmap we will use to build the first version of SoftspaceAR that you can use in a real workflow. It will guide the next prototypes we build and release, each of which will move intentionally toward the design described here.

Abstract

A critical step in creative knowledge work is synthesis: the distillation of disjointed data into coherent ideas. As information problems become more complex, and good ideas increasingly valuable, individuals and groups demand better tools for managing and synthesizing knowledge.

We observe three trends in software aiming to meet this demand: spatial canvases (e.g. Muse, Figma), knowledge graphs (e.g. Roam, Obsidian), and collaboration (e.g. Zoom, Teams).

However, legacy tools remain constrained by their flat, bounded interfaces. Our design for Softspace proposes a collaborative spatial knowledge graph that transcends the 2D paradigm, offering creative knowledge workers the ultimate tool for thought.

I. Introduction

Knowledge Synthesis

The creative knowledge economy runs on great ideas, executed well.

A critical step in developing these ideas is knowledge synthesis: the work of recombining a large, disjointed collection of information into something simple, coherent, and valuable.

A designer synthesizes a wall of references into a beautiful product design. An entrepreneur synthesizes stacks of market data into a bold business strategy. A researcher synthesizes a myriad of observations about the world into an elegant explanation.

Synthesis is hard. To do it well, we have to hold many pieces of the puzzle in our head at once, and test innumerable combinations of ideas. Synthesis gets exponentially harder as the quantity and complexity of the information we’re working with increases.

Spatial Computing

By making the sharing, processing, and storage of information fast, cheap, and reliable, computers have become invaluable to knowledge workers. Yet when we use computers for knowledge synthesis, it becomes apparent that the UI is now the bottleneck.

Simplistically: laptop screens are too small and flat for working on big knowledge problems. They don’t let us see enough of the puzzle at once, and the way they show information makes remembering or reasoning about it difficult when it’s out of view—which is most of the time.

Spatial computing offers a solution. XR headsets let us see, remember, and think about far more information than before, by displaying it in immersive 3D. This may be a new paradigm for software, but it’s one to which our brains and bodies are exceedingly well-adapted.

Softspace

Softspace is an XR productivity and creativity app that gives creative knowledge workers a powerful new way to organize, develop, and communicate great ideas.

In our design, users work with conventional content types (e.g. text, images, PDFs, and websites) within a radical new paradigm: a collaborative 3D knowledge graph.

Through a workflow that combines elements of notetaking, mindmapping, and moodboarding, users build up spatial information workspaces that reflect the structure of their ideas.

Users then compose and export linear syntheses of their knowledge graphs, in the form of markdown files, for use in downstream workflows.

The immersive virtual workspace allows collaborators to step into the same information space for discussion and co-creation, regardless of physical distance.

II. Design

Items

The entity primitive in Softspace is the item. Items are single pieces of content data, or containers that hold other items. Items correspond to what are called blocks in some other knowledge-management tools, such as Notion or Roam.

Content Items

Conceptually, content items map best onto single files, although in cases such as text and URLs this correspondence does not strictly hold. Initially, content item types in Softspace include:

• text paragraphs
• static images
• PDF documents
• bookmarked websites

Future updates to Softspace may implement, among others:

• tweets
• videos
• podcasts

Container Items

As their name suggests, container items hold other items. The initial design for Softspace only specifies a single type of container: the topic.

Topics can either be expanded or collapsed.

• While expanded, the contents of a topic are displayed in a fixed ordinospatial layout that makes use of all three spatial dimensions while constraining contents to a single linear order. This ordinality makes topics mappable to conventional document formats (such as markdown), and usable as 3D notes.
• While collapsed, the text contents of a topic become hidden. However, backlinks from hidden text to other topics are visible as connections between the collapsed topic and referent topics. Images, PDFs, websites, and other non-text contents remain visible—they float free of the collapsed topic, but remain visually and spatially connected to it.

Topics have a title: a string value that identifies that topic within the workspace. Future updates to Softspace will implement topic aliases: alternative titles (to accommodate capitalization, synonyms, orthographical variants, etc.) by which one topic is identified.

When exporting a Softspace workspace, each topic is interpreted as a markdown file whose filename is the topic title, and whose contents are those of the topic.

For performance and technical implementation reasons, the initial design of Softspace will only permit a single topic to be expanded at a time. Later updates will allow multiple topics to be expanded simultaneously.

For performance and technical implementation reasons, the initial design of Softspace will not permit nested topics—i.e. topics cannot contain each other. Later updates will enable this.

Transclusion

Transclusion is a term coined by Ted Nelson in his 1980 publication Literary Machines. It refers to the concept of including a single piece of content across multiple contexts as live instances, so that a change in one instance is reflected across all instances.

Softspace implements transclusion by allowing item types to be contained by any number of topics. Adding a transcludable item to a topic does not remove it from its other containers.

Initially, this will be possible for item types which are highly atomic (semantically independent of their immediate context) and/or relatively immutable. Text items do not meet this criterion, and can therefore only be contained in a single topic at a time.

Transclusion allows topic containers to function as tags.

Backlinks

The popularity of the note-taking app Roam Research (and Roam-like apps) can be largely attributed to its use of the backlink as a core interaction primitive. A backlink is an in-line reference from text to a conceptual entity. In the case of Roam, backlinks point to notes. In Softspace, backlinks point to topics.

Our design borrows the [[]]-notation of Roam. Terms within a text item which are surrounded by double square brackets will be visually and spatially linked to the topic with the same title as the enclosed text. If no such topic exists, one will be created.

When a topic is collapsed, the backlinks from its hidden text items remain visible as indicator lines that connect the collapsed topic to referent topics. In future updates, these indicators will display the snippet of text which contains the backlink.

Spatiality

Whiteboarding apps like Miro have proven the tangible value of being able to lay out information spatially in a software tool. Many apps which feature a canvas for UI design purposes, such as Figma, are often used as general-purpose boards instead. New tools built on this basic pattern seem to emerge daily, from Muse to Heptabase to Apple’s upcoming Freeform.

Spatial interfaces are effective because our brains have evolved to be astoundingly good at perceiving, remembering, and interpreting where objects are in our environment.

But a spatial canvas displayed on a laptop screen suffers from three drawbacks:

1. 2D supports less spatial complexity than 3D, limiting spatial semantics.
2. The view cuts off at the edge of the screen, limiting contextual awareness.
3. The user is not situated within the workspace, limiting spatial memory.

The design premise of Softspace is a 3D spatial canvas within which the user is situated; therefore, it bypasses the above three constraints.

Further, our design incorporates three distinct modes for the spatial positioning of items in a workspace. These modes are optimized for different phases in the workflow.

Ordinospatial Layout

Within an expanded topic, items are laid out using a front-to-back, left-to-right, top-to-bottom system called an ordinospatial layout. All three spatial dimensions are used to arrange contents, but there is a definite order that makes each topic interpretable as a linear document.

This mode is best for the content a user is working on directly at that moment. Using this layout is conceptually similar to drafting a 3D note.

Force-Directed Layout

Items whose positions are not fixed within an expanded topic are subject to the force-directed layout system. This is a simulation-based layout system that automatically gathers related items closer together in space, and pushes unrelated ones further apart.

Related items are those with a semantic relation in the knowledge graph. The initial design specifies two such relation types:

• containment: the relation between a topic and its included contents
• reference: the relation between a text item and topics it links to via [[]]-notation

This mode is best for content that is not currently being worked on. Items move themselves into a spatial configuration that makes visible the relationships between them.

Cartesian Layout

Items not in an expanded topic can also be pinned in place, so that their position and rotation is no longer determined by the force-directed graph simulation. Instead, they remain at a fixed, user-determined Cartesian coordinate.

This mode is best for reference items that should be held in a specific spatial configuration.

Workspaces

The highest-level organizational unit of Softspace is the workspace.

Items are always created within a workspace. Initially, items will only have a single parent workspace; later updates may enable cross-workspace transclusion.

Users can create, manage, open, close, and delete workspaces using an in-headset UI. The initial design for Softspace only permits a single workspace to be open at a time.

Workspaces can either be local or cloud. Local workspace data is stored completely on-device, and are not multiuser-compatible. Cloud workspaces store their data to the cloud, and are multiuser-compatible. Softspace will launch with only local workspaces; cloud workspaces will be implemented shortly afterward.

User permissions are managed at the workspace level. Permissions roles can include owner, administrator, and guest access.

Workflow Integration

A key challenge when using mobile devices for productivity is the lack of a common file system. XR headsets are no exception. This deficit adds friction to the process of bringing files into and out of mobile software. If this friction is too high, it can feel like work gets “stuck” in the device, which understandably deters use.

Softspace is a designed to minimize this friction. It does this by:

• Prioritizing support for content formats that are common across knowledge workflows, such as images, PDFs, and markdown files
• Integrating with popular cloud file storage services like Dropbox, with automatic exports to maintain a readily-accessible copy of work outside the headset
• Implementing a full-featured in-app browser that makes the web easily accessible from within the headset

The goal is to maximize upstream and downstream compatibility with existing workflows, while retaining the unique advantages of this new computing medium. Users can quickly bring files into Softspace, work on them there, then easily access the contents of that workspace from their other devices at any time.

Common File Formats

Initially, users will be able to import:

• Text files (.txt)
• Markdown files (.md)
• Image files (.jpg/.jpeg, .png, .tif/.tiff)
• PDF files (.pdf)

Users will be able to export workspaces as a collection of 1) markdown files that correspond to its topics, and 2) image and PDF files.

Cloud Storage Integration

Cloud file storage access is available in the form of a simple 2D web app that is accessible in-headset. It allows users to log into their cloud storage accounts, select files and folders for import, and select folders to export workspaces into.

The first such integration will be with Dropbox, because of its large user base. We are also exploring Google Drive and local network drive access.

In-App Web Browser

A good web browser is central to almost all knowledge workflows. We implement a full-featured browser within Softspace that gives users access to the rest of the web, including the web app versions of complementary tools.

The in-app browser will allow the user to:

• Browse websites, including web apps
• Bookmark websites for future reference
• Save images files from the web into workspaces
• Snip any portion of the browser window into workspaces
• Copy/paste text to and from text items

Multiuser Support

Although by catalyzed the pandemic, we expect the importance of remote work to continue well into the future as companies seek top talent, and talent seeks geo-flexibility.

However, video-based remote collaboration tools fall far short of the creative magic that is possible when working together in-person. XR closes the gap by creating a true sense of social co-presence between collaborators in the same virtual workspace.

Softspace will not initially support multiple users, but its technical architecture has been designed from the beginning with multiuser collaboration in mind. We will enable this feature once cloud workspaces are implemented, as this is a key technical prerequisite.

Interaction Model

Spatial computing remains in its infancy, and is evolving rapidly and divergently.

The Softspace interaction model is designed to rely as little as possible on the specific features of today’s headsets, and to be highly portable across hardware and input paradigms.

Therefore, our design only assumes a head-mounted 6DoF AR device (passthrough or see-through) with high-fidelity hand tracking and bluetooth keyboard support.

Augmented Reality

Full-occlusion virtual reality is unsuitable (at least as the default mode) for a tool intended for use in professional settings. Blindness to one’s immediate physical surroundings gives rise to a sense of unease and vulnerability. This prevents many users from entering the state of flow that is necessary for doing their best creative work.

The design of Softspace is premised on an augmented reality paradigm that allows users to see their immediate environment. Virtual UI elements appear to float in this space. Initially, it will not be possible to anchor items to specific points in the physical environment, but this functionality will come with later updates.

Happily, even the low-resolution passthrough augmented reality of the Quest headset is sufficient to dispel the discomfort that arises from visual occlusion.

Hand Tracking

Currently, the most common input device for XR headsets is the hand controller. Optimized for gaming, this device is poorly suited to productivity use cases, because:

• It must be held in the hand at all times, precluding the use of a keyboard
• Its form, balance, and button placement are reminiscent of weaponry
• It adds two more devices to keep charged, remember to pack, etc.

Therefore, all non-keyboard inputs in Softspace rely only on computer vision-based hand tracking, which has already been developed to a very high level of usability and reliability.

Locomotion

The way that virtual objects are overlaid on the view of the physical environment in Softspace makes user locomotion through the workspace technically equivalent to the spatial repositioning of the workspace around the user. The only difference between the two is frame of reference.

Locomotion—or correspondingly, workspace repositioning—is initiated by forming a fist with one or both hands. This action “grabs” the workspace where the hand(s) is positioned. The user then moves the grabbing hand(s) to move, rotate, and scale the workspace.

Manipulation

Our philosophy for the design of the Softspace UI can be characterized by leveraged direct manipulation. We want to give users a feeling of high agency, effectiveness, and control when interacting with objects in the workspace.

Users highlight UI elements with a line-of-sight system, which uses an imaginary ray from the eye to the hand as its targeting vector.

Once an item is highlighted, pinching the tip of the index finger to the thumb is interpreted as a click action, which either causes the highlighted object to be used or grabbed.

Movements in the grabbing hand cause the grabbed item to be repositioned, with a leverage factor being applied to its motion along the user’s vector of view.

Grabbing an item with two hands enables rescaling and resizing.

Text Input

Given the centrality of natural language in knowledge work, text is a first-class content type in Softspace. Fast, accurate, frictionless text input is therefore absolutely critical.

Given these requirements, there is simply no viable alternative to the use of a physical (bluetooth) keyboard as the primary text input device. However, a backup virtual keyboard is available at all times.

Future releases of Softspace will explore additional text input methods, such as speech-to-text.

Art Design

The designer of XR software has much more control over the sensory input of their user than the designer of a 2D app does. While in Softspace, everything a user sees (and much of what they hear) is the result of decisions we will have made. This gives us great power to shape the user experience, but also comes with greater responsibility to ensure it’s a good one.

Two principles underpin the art design of Softspace: comfort and productivity.

Comfort

Comfort in Softspace has both an aesthetic and a performance component.

Given the full-immersion nature of XR, less is more. Our aesthetic design is restrained and minimal. We rely on a limited palette of colors and a familiar set of geometries. We prefer to subtract, adding only when necessary.

Performance-wise, maintaining a smooth 90fps on the Quest 2 (and 72fps on the Quest) is critical for user comfort. We therefore make creative use of a few simple meshes and shaders, to minimize compute load as the number of items in the workspace increases.

Productivity

Most XR software today is for gaming, entertainment, or other use cases where there is heightened value in sensory stimulation and excitement.

In contrast, we are building software for deep work. To support this, we wish to promote focus, creativity, and flow instead.

Softspace UI elements, passthrough color filtering, animation behaviors and velocities, and other aspects of its look-and-feel are calibrated to foster these qualities.

Example Images

Below are images from the prototypes we built to explore and validate various concepts and features. They are exemplary of the art design we will develop for the Softspace MVP.

III. User

The intended user of Softspace is a high-agency knowledge worker whose livelihood depends on her ability to quickly and effectively synthesize complex sets of information. She might be:

• An entrepreneur writing a product requirement doc
• A design researcher summarizing user interviews
• An independent analyst drafting a Substack post
• An architect crafting a deck about a new project
• A grad student outlining a chapter of her thesis

She currently uses a combination of analog tools (pen and paper, physical boards) and digital ones (Notion, Apple Notes) to collect information, make sense of it, and draft documents.

Crucially, she feels an acute frustration with the limitation of existing tools. She may be exploring spatial canvases (Muse, Figma) or structured note-taking apps (Roam, Notion) to help her manage and make use of her knowledge base.

These apps are steps in the right direction, but she wonders why there still isn’t software that gets close to the creative magic of a shared team project space.

Of course, to be able to download and use Softspace, she will need to have access to a compatible headset (e.g. a Quest 2).

IV. Flow

Workflow Phases

Formally, the intended user flow of Softspace can be divided into three phases:

1. Collection: adding relevant information to a workspace via cloud storage and the web
2. Construction: building up the knowledge graph by writing notes and composing topics
3. Collation: composing synthesis topics intended for export as linear outlines or drafts

In practice, we expected users to cycle through this flow many times, jumping between steps as they seek to make sense of a knowledge problem and explore different solutions to it.

Example Flow

For example, this is how somebody might use Softspace to draft a design proposal:

1. Collect images and PDFs related to the project into a Dropbox folder. Export notes from a note-taking app as markdown files into the Dropbox folder. This content likely includes:
• project brief
• reference images
• client interview notes
• previous project materials
2. Launch Softspace on headset, and create a new workspace.
3. Using the cloud storage UI, import the contents of the Dropbox folder to the workspace.
• Text and markdown files are converted into Softspace topic items
• Images and PDFs are converted to the corresponding content items
4. Build up a project knowledge map from these contents by:
• Creating and writing text blocks
• Creating topic and adding content to them
• Referencing various topics from within text
• Adding images using the in-app web browser
• Copy/pasting text using the in-app web browser
• Bookmarking URLs using the in-app web browser
5. Create a new synthesis topic which is intended for export. Compose a draft of the design proposal through a combination of 1) including content items already in the workspace and 2) writing new text that ties ideas and content together.
6. Using the cloud storage UI, select a Dropbox folder to export the workspace to. (This can be a one-off action, or be set to recur automatically.)
7. Copy the synthesis topic’s markdown file into a word processor for editing and formatting.