SoftspaceAR Prototypes 01–05

Building a spatial knowledge management tool in public

From January to November 2022, I built and released a series of prototypes apps to explore the design space for a general-purpose productivity tool that harnesses two key breakthroughs in consumer spatial computing: passthrough AR and hand tracking.

I published these prototypes to Twitter and Substack, and was able to grow a significant following among those watching the intersection of spatial computing and tools for thought.

This work resulted in additional funding and publishing opportunities for Softspace, which led to the August 2023 launch of Softspace v2024 on the Meta Quest Store.

Here is the entire Substack series documenting these prototypes.

April 14, 2022

Welcome to SoftspaceAR

Building the next Softspace in public.

Softspace harnesses spatial computing to help thinkers and makers better see, understand, and develop the ideas that form the heart of their creative projects.

To me, augmented reality has always made more sense than virtual reality for general-purpose productivity, because full immersion isn't ideal in many professional settings.

But see-through augmented reality (e.g. HoloLens) is really hard, and a truly mainstream headset built around this tech is still years away. Until recently, I believed that virtual reality was the only viable medium for Softspace for the foreseeable future.

Last summer, Oculus released their Passthrough API. After playing with it for a few days, I was shocked to realize that even today’s crappy passthrough is good enough to build great tools on.

It's a complete game-changer.

There's now a clear path forward for augmented reality software. Today's grainy grayscale passthrough will lead to tomorrow's hi-res low-latency full-color passthrough—which may even prove superior to see-through smartglasses.

The user experience advantages of AR for general-purpose productivity are massive.

One software update suddenly brought mainstream head-mounted AR into existence.

So we're starting work on SoftspaceAR.

Since 2018, we’ve built three major versions of the Softspace VR app, and shipped two. I could not be more proud of the love, energy, and vision that has gone into each.

If there were one thing I could change, however, it would be this: I would have released earlier and oftener, to cover more of the vast possibility space that we face.

Therefore: to build SoftspaceAR, we’re using an open-development model to quickly test a much wider range of ideas than we’ve been able to do before.

Over the coming months, we will release 5–10 prototypes to explore and (in)validate promising ways to harness augmented reality as a medium for thought.

These will be standalone apps that use the same core interaction paradigm, but are otherwise free to start from different points in the product idea maze.

This process is about learning, not polish. I'm aiming to release something every few weeks.

In fact… 🥁🥁🥁

The very first prototype is already here!

SoftspaceAR Prototype01 is an augmented reality Dropbox spatializer you explore with your hands.

Go ahead and get Prototype01 for free, then join our Discord and let me know:

🤔 What you think of Prototype01

🧠 How you'd like to use Softspace

💡 Cool ideas for future prototypes

I'll be publishing the prototypes, as well as what we learn from each, on this Substack.

I expect the first few prototypes to be all over the map, the next few to start converging on core underlying principles, and the last ones to build more deliberately on what came before.

At that point we'll be working toward the first release version of SoftspaceAR, which I intend to submit to the official Oculus app store by the end of September.

In short:

We're building an AR version of Softspace in public, and we'd love to have you join us by

🧪 trying out Prototype01

💌 signing up for updates

👯‍♂️ joining our Discord

April 14, 2022

SoftspaceAR Prototype01

Explore Dropbox in 3D with your hands.

Today we're releasing Prototype01. It's an augmented reality Dropbox spatializer that you explore with your hands.

Prototype01 implements several intriguing ideas that both our team and users have long wanted to test:

🕸 3D force-directed graphs

🗃 3D folder structure explorers

😎 Passthrough augmented reality

🤲🏼 Controller-less hand-tracked UI

Quick explanation of each below:

Force-directed graphs are an increasingly popular way to visualize information that consists of interconnected nodes. Apps like Roam and Obsidian (below) have recently added them, where they seem marginally useful. But what if you could navigate one in 3D, immersively?

Previous versions of Softspace all had some way to browse a connected Dropbox account from which to import files. But we've often gotten the question: what if the file browser were the entire app, and you stepped straight into your Dropbox?

Augmented reality always made more sense for productivity than full-occlusion VR, but mainstream AR headsets seemed too far away to build for. After playing with the Oculus Passthrough API, I realized even today's crappy passthrough is good enough to build a great app on.

Oculus has also been steadily improving the hand-tracking capabilities of the Quest 2. It's not perfect, but it's already workable, and the steep rate of improvement makes me comfortable committing to a controller-less future for the Softspace interaction paradigm.

Prototype01 combines these ideas into a single research demo.

Try It For Free

0. If you haven't already, get your Quest set up to sideload apps from @SideQuestVR. This is great thing to do in general, because it lets you try out great indie software that isn't available yet on the main Oculus app store.

1. Connect your Quest to your computer and install SoftspaceAR Prototype01 from SideQuest.

2. Open Prototype01 on your Quest. Log into Dropbox using the web browser that appears. Click on the page by holding up your hand and touching your index finger to your thumb. Type by pressing the keyboard with your index fingers.

3. Once you're logged in, navigate to a Dropbox folder that you would like to spatialize. This prototype only loads in image files (jpg, png, etc), and it will load everything in every subfolder, so it's best to select an image-rich folder that not too large (<1000 images).

4. To navigate around, you can "grab" the virtual space by making a fist with one or both hands. One hands is for simple moving, while two hands let you rotate and scale the space.

5. You can also move, rotate, and scale objects. Select them in the same way that you clicked on the web browser: by hovering over them with your hand and touching your thumb and index finger together.

6. The next time you open Prototype01, you'll still be logged into the same Dropbox account, but you'll be able to select another (or the same) folder to spatialize. Alternatively, you can log out of Dropbox to clear your data.

That's it! If you have any questions or thoughts, please join our Discord and share them with us.

10 May, 2022

SoftspaceAR Prototype02

Proof that you can do serious writing in a headset.

Prototype02 is all about writing.

Specifically, I wanted to prove that it's possible to write comfortably, productively, and enjoyably inside a passthrough augmented reality app.

Motivation

Writing might not seem like the highest-priority task for a spatial computing app. In fact, the idea of writing while wearing a headset might strike a lot of people as unnecessary—or silly!

After all, spatial computing’s great appeal is in 3D content, right? All the cool stuff you see in the Oculus ads involve zooming through outer space or tearing zombies apart with your bare hands. Isn’t using this tech to work with text a huge waste of its unique potential?

Also, there’s nothing terribly broken about existing ways of writing. Quite the contrary: a modern laptop running a well-designed text editor is a highly optimized writing machine. It epitomizes a near-universally-embraced paradigm. Why mess with success?

The thing is: the ability to work with text is critical for almost any serious productivity workflow—even ones centered on 3D content. From object names, to short annotations, to entire paragraphs, text is everywhere. Natural written language still plays an irreplaceable role in representing many of the ideas that we need to work with, regardless of domain or medium.

I wanted to tackle writing head-on and de-risk it before prototyping other aspects of SoftspaceAR. This isn’t to say that the writing experience needs to be perfectly on par with that of your MacBook (although I made some interesting discoveries about the advantages of writing in a headset; more on that below). It just needs feel good, and not get in your way when you want to do it.

I decided on a fast and dirty metric of success: Could I write for one solid hour in the prototype?

I initially thought about setting additional qualifying criteria, like word count, or some vague notion of writing quality. But after trying the first few development builds, I realized that the one-hour criterion implicitly covers a lot. It’s painful to write for that long if you feel like you’re going too slowly, or that you’re producing garbage.

Just to be clear: I don’t expect that people will really sit and write for hours on end in SoftspaceAR. It’s more likely that you would write shorter bits of text, sporadically, while working with other content. But I’m using “write for a solid hour” as a close proxy for “good for real writing”.

Mechanics

To write, you need something to write with, and something to write on.

The input part was an easy decision: I’ve tried many, many virtual keyboards, and none are good enough for serious writing. SoftspaceAR will simply require a physical (wireless) keyboard.

As for the writing surface, my goal was to keep it familiar. A naive user coming from a desktop text-editor should be able to pick this up and go. So I went with a simple fixed-width text box that expands downward to accommodate new content, like we’ve all seen a million times before.

I discovered how much we’ve come to take certain textual interactions for granted. I never thought much about these, but it sucks when they’re not available. Some examples: clicking on the text to position the caret (the flashing vertical line that tells you where you’re typing); click-dragging to select text; copy-paste; moving the caret around with the arrow keys.

Happily, most of these interactions are already beautifully implemented in the text-rendering package I use, TextMesh Pro for Unity:

Finally, the one “radical” design decision I made was to make the paragraph the object primitive (as opposed to the page). This is inspired by the way that paragraphs work in Notion and LogSeq, and I find this way of structuring documents much better aligned with how I think and work than one where documents are treated as bags of words with occasional line breaks. I wanted to test this out because I think this is how SoftspaceAR is going to treat text objects.

So in Prototype02, hitting the return key creates a whole new text box:

There are still some holes in the UI. For example, you can't use the arrow keys to navigate between paragraphs. I decided not to plug these because they don't prevent a user from achieving the stated goal of the prototype: letting you write comfortable, productively, and enjoyably.

How do I know?

Because the first draft of this piece was written in just over an hour, in Prototype02.

Discoveries

• Early builds had transparent backings for the text objects, because I assumed that it would be more comfortable being able to see things in the physical space around you even if they were behind a text box. In fact, I found it very hard to focus on writing when I could see through the text box—I guess my brain was overloaded by the conflicting visual information.

• Once I solve that issue by making the backings opaque, I was surprised to find how enjoyable it is to write in passthrough AR. I was expecting to have to force myself to switch from writing on the laptop to writing in the headset, but I actually found myself looking forward to settling down in front of a floating text box and just getting my thoughts out in front of me.
• One of the biggest improvements in my ability to write well in Prototype02 didn’t come from the design or code at all. I can’t really touch-type, and on a laptop I look at the keyboard all the time. The resolution on the Quest 2’s passthrough is too low to easily make out the letters on the keys, which basically forced me to figure out which letters are under which fingers. The first few days were really painful because my typing was so slow and error prone, but by the time I made the above time-lapse video I was typing away like a champ. Bug Feature!

Takeaways

• Prototype02 met the concrete criterion of being usable for writing for a solid hour. It was really helpful to explicitly define the metric for success of the Prototype before I set out to build it, because it helped me stay focused and know when to stop. For the next prototype, I’ll consider publishing the criterion for success even before building anything, as a way of “pre-registering” the experiment publicly.
• It took four weeks between the releases of Prototype01 and Protoype02. This is longer than I would like. My goal is to get this down to 2-3 weeks so I can cover lots more ground before starting to pull the various threads together into an MVP. But Prototype01 took closer to two months from start to finish, and a lot of the work for each in groundwork that speeds up the next cycle, so the trend is encouraging.

14 June, 2022

SoftspaceAR Prototype03

What if workspaces were markdown under the hood?

Prototype03 is about making Softspace deeply interoperable with your other tools.

Having a spatial canvas to organize your ideas on is amazing, but spatial layouts can be hard or impossible to translate to the other formats and devices in your workflow. And if the work you do in an app gets stuck there, you’re (justifiably) not going to use it very much.

With Prototype03, I wanted to figure out how Softspace might overcome this problem without simply becoming a VR version of an existing 2D app. Could we keep that immersive spatial magic, while reading from and writing to a file format that’s highly portable?

Like markdown?

Making Softspace workspaces interpretable as markdown files—and vice versa—would open up super exciting ways to fit Softspace into knowledge workflows. For example:

• You could throw research notes and images into a Dropbox folder, use Softspace to synthesize them into a project proposal, and then directly email that proposal to your team.
• You could work on your Roam/Notion/Obsidian/LogSeq/Athens/etc. personal knowledge repository in Softspace.
• You could start brainstorming for a Substack post in Softspace, then continue editing the post on your phone while riding the subway 😉.

All this sounds great—but there’s a catch.

Markdown, like all text formats, is inherently ordinal. Softspace, running in augmented reality, is inherently spatial. Ordinality requires that each element have a definite place in a linear sequence, but highly spatial layouts want to break free of such constraints.

Prototype03 reconciles these demands with a 3D layout system that is strictly ordinal, while letting you move a text block in any of the three spatial dimensions.

Words can only go so far in conveying how any 3D UI works, so I encourage you to get the prototype and let me know what you think (via Twitter, Discord, or email).

Or, you can do that after reading this deeper dive 💪🏽🧠👇🏼.

Layout Systems

Softspace is a tool for working with the ideas and knowledge that you already use in your projects. This knowledge takes the form of text, images, websites, PDFs, videos, and other content. To position all this content in 3D space, we need to use a layout system.

Ideally, this layout system places things according to how they’re related to each other in the underlying data structure. This way, you can understand those relations just by looking at where things are, and you can modify them by moving things around.

For example: the tasks in your favorite to-do app are laid out in the order that they’re saved in the underlying database. If you drag in item into another position, the database is updated to match. Order == priority, so this interaction lets you reprioritize upcoming tasks.

Across knowledge management tools, notes are a core format. Therefore, to make Softspace highly interoperable with existing knowledge workflows, it should have good support for the data structure that notes come in: text documents.

A defining characteristic of elements in a text file is their order in the document. We want Softspace’s layout system to express that ordinality. At the same time, we don’t want to give up all the benefits of spatiality by simply rendering a 2D document floating in front of you.

We need to invent a layout system that is strictly ordinal, yet meaningfully spatial.

Ordinality

The biggest advantage of ordinality is portability. Imagine an email you draft with speech-to-text, then finish and send on your laptop, before it’s read on someone’s phone. Its fixed sequence of information can adapt to very different contexts without losing much, if any, of its meaning.

The ordinality of HTML allows for websites that respond consistently to different screen sizes. The ordinality of natural language lets us turn books into audiobooks and back. (The non-ordinality of spreadsheets makes them a huge pain to work with on a phone.)

There’s another, deeper benefit to ordinality. Content that’s laid-out ordinally maps directly onto the temporal structure of human experience. That gives its authors a high degree of control over when ideas are encountered, and how they build on each other.

On the other hand, ordinal formats struggle as working environments for generating knowledge, because at larger scales, knowledge is not ordinal (it’s nested and networked).

As internet pioneer Ted Nelson puts it: “Writing is the process of reducing a tapestry of interconnection to a narrow sequence. And this is, in a sense, illicit! This is a wrongful compression of what should spread out.”

With all due respect to Ted: I actually think that writers do important and difficult work by charting navigable paths through that “tapestry of interconnection”. But he’s also right. We lose something crucial if our tools and methods obscure the true shape of that tapestry.

Spatiality

Softspace uses the emerging medium of spatial computing to let you work with that tapestry outside the limits of small, flat screens. We also believe that spatial layouts can unlock your innate superpowers of spatial perception, reasoning, and memory.

A maximally spatial layout system would let you place things freely in a 2D or 3D canvas. There might be collision detection, or snapping, but ultimately, every element’s position would be determined by an (x, y, z) coordinate. This is how most whiteboarding apps, like Miro, work.

Within this Cartesian playground, you get the full benefits of spatiality. You can construct richly meaningful configurations of information that express a lot of nuance and complexity. If you work in one for a while, you can really recall and explore these layouts in your mind’s eye afterward.

The problem with freely spatial layouts is that you can’t reliably put their elements into a coherent order. And without this ordinality, spatial layouts are hard to port to less-spatial contexts.

For example, a lot of meaning is lost when you export a Miro board to a linear document. Also, the process uses an “across-then-down” heuristic that creates unpleasant surprises in content order.

When you add a third dimension, the problem gets much worse. 3D Cartesian layouts, like those in previous versions of Softspace, are essentially impossible to export cleanly.

Ordinospatial Layouts

So ordinal data format are high portable, but obscure how information ties together. Spatial layouts reveal those interconnections, but aren’t very portable. Could we have the best of both worlds?

This prototype implements an experimental layout system that enforces a strict order to elements, but lets you position them relatively freely all three spatial dimensions.

Content (which are text blocks only, for now) flows down columns; columns spread horizontally across a plane; and planes stack front-to-back.

This system lets you move things along the x-, y-, and z-axes, but always results in a spatial configuration that is unambiguously interpretable—both by machines and people—as a definite sequence of elements.

There’s some “give” in the positioning of elements in the x- and z-axes, which lets you place things closer together or further apart. Physical distance is an important way of expressing relations between ideas in spatial layouts.

And of course: as you build up the workspace in Prototype03, it automatically writes your work to a markdown file. You can use SideQuest to retrieve it at: sdcard / Android / data / com.softspace.ar.prototype03 / Prototype03-Text.md.

Conclusions

The criterion of success for this prototype was to develop a 3D layout system that was 1) strictly ordinal, yet 2) meaningfully spatial.

✅ Prototype03’s layout system is strictly ordinal. The proof is that it can write out a markdown file representing any possible workspace you create in it.

❓ Spatiality, on the other hand, is a spectrum. The layout system is less spatial than a purely Cartesian one, but it still allows for relatively free placement of elements. Does this mediated spatiality convey enough of the benefits of the medium to justify working in a headset? I would say yes, but ultimately, only users can judge if this criterion has been met.

Future Ideas

• Mapping breaks in the x, y, and z axes to H3, H2, and H1 headings in markdown.
• Prototype03 only writes out a very simple markdown file, with paragraph blocks only. Future builds would need to support all the major block types.
• Instead of pointing to local headset storage, point to a Dropbox folder.
• Image file support
• In-headset browser support
• Tag / wikilink support

Improvements

• Hand interactions are disabled when a hand is not within a certain zone of activation in front of your head, to prevent accidental inputs, e.g. selecting things while typing. Thanks to Brian Eppert for this suggestion.
• Fatigue from having to hold your hand in the air to interact with elements is real. I'm working on ways to mitigate this, both by reducing the number of interactions that require raising your hands (e.g. making text boxes navigable using the arrow keys), and increasing the speed and ease of hand interactions (e.g. making positioning of objects snappier and more responsive). Thanks to Andy Matuschak for pointing out this issue.

That’s a wrap. As always—thanks so much for reading, and please don’t hesitate to reach out with your thoughts, comments, and questions!

30 July, 2022

SoftspaceAR Prototype04

See the true shape of your ideas (literally).

SoftspaceAR Prototype04 is out! You can download the APK here, then install it on your Quest 2 using SideQuest or the Oculus Developer Hub.

This prototype is an answer to the question: (How) Could a spatial UI let you work with, and work within, the true shape of your ideas?

To see how this question leads to the design of Prototype04, we need to take a step back and understand what this question even means.

In what sense do ideas have a shape, and why does their shape matter?

The Structure of Knowledge

What is the structure of knowledge? In other words: what are the actual things that constitute knowledge, and how are they related to each other?

There are many possible answers to this question. It could be that knowledge is one very long sequence of information; or a scattered field of points that don’t relate to each other at all; or a Borgesian state space of permutational nonsense through which we rifle for nuggets of Truth.

(I’m going to dodge the notoriously divisive question of what constitutes knowledge in the first place, as opposed to other kinds of information like data, or sensory experience. Let’s go with a Stewartian definition of knowledge for now. It’s the thing that PKM folks think about all the time. It’s what knowledge workers are busy working on. It’s the kind of information that tools for thought help you better create, understand, and harness.)

How knowledge is structured matters, because its structure defines what we consider to be legitimate strategies for seeking and creating it.

For example, if knowledge were actually a single long sequence of information (e.g. the revealed word of a deity), then we would seek knowledge within the canon, and create knowledge by expanding on canonical points.

If, like myself, you’re not religious, the biblical reference might make this model seem epistemologically unsound… but consider that this is how a lot of writing is produced. Pieces start out as a small list of points, each of which is then expanded into a collection of points, some of which need further elaboration—all while maintaining the overall single-sequence structure. Also: an encyclopedia!

Another candidate for the structure of knowledge is the tree: information is grouped into topics, which are grouped into larger topics, going from the specific to the general. This model is the basis for the Dewey Decimal System, which is used to organize books in libraries. It’s also how most people think of species as being organized: within an evolutionary Tree of Life.

If knowledge is structured as a tree, you should seek knowledge by determining the general category that a piece of information would be found in, then follow the path of ever-narrowing subtopics until you get to your answer. The work of creating knowledge would be the work of discovering new pieces of information and deciding which sub-sub-sub-branch it belongs to, or filling in categorical “missing links” between something higher up in the tree with something lower down. This is literally how librarians and taxonomists work.

The problem is that these models prove incomplete under closer scrutiny.

Biblical hermeneutics is the discipline of interpreting and elaborating on the received word, and is a critical bridge between what’s written in the Bible and important practical questions like “should capital punishment be allowed”? These interpretations are core theological knowledge, but they don’t simply slot into the existing sequence of the Bible. Rather, they can draw on information from across the text—or indeed, other authoritative interpretations—to arrive at their conclusions.

The librarian’s dream is a library with every book in its proper place, neatly arranged according to the single totalizing hierarchy. But underneath their calm, hardbound surfaces, books are seething with irregularities that break from the Dewey Decimal System. Footnotes and citations in the text refer to texts that could be on the next shelf down, or in a totally different category two floors up, forming a vast, nonlinear web that is at least as important as the DDS.

Even the Tree of Life, which seems to hold true for the vast majority of species that reproduce sexually, proves more convoluted than taxonomists would like. Consider the case of the hoatzin, a bird that defies all attempts to paint an orderly picture of the tree of bird species.

These examples hint at a different model.

If knowledge can be said to have a global structure, there’s a lot of evidence pointing to that structure being a network. Technically: it’s a graph in which pieces of information are the nodes and relationships between those pieces are the edges.

Why might we think this?

• Footnotes and citations in texts referencing other texts, or even other sections in the same text, form a nonlinear graph structure.

• Mind maps almost always take on a graph-like structure, with node-like concepts connected by lines. (Actually, while Googling for images of hand-sketched mind maps, I realized that the vast majority of examples are trees, which are a special subset of graphs. The distinction between the two is critical—but I would contend that people are being nudged toward drawing graphs as trees by the limitations of pen and paper; i.e. they are only drawing the tree-like taxonomical relationships, and ignoring other relationships, because then the mind map gets way too messy and illegible.)

• A new generation of graph-based note-taking apps (spearheaded by Roam Research) have become very popular, very quickly, in academic and professional communities that work intensively with knowledge.

• Anecdotally, a lot of people who were uncommonly good at working with knowledge also seem to intuit its underlying structure as graph-like.

Here’s Steve Jobs:

"Creativity is just connecting things… When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things.”

Also, Albert Einstein:

“The physical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be ‘voluntarily’ reproduced and combined… this combinatory play seems to be the essential feature in productive thought… the play with the mentioned elements is aimed to be analogous to certain logical connections one is searching for.”

Last, and never least, James Webb Young:

“An idea is nothing more nor less than a new combination of old elements [and] the capacity to bring old elements into new combinations depends largely on the ability to see relationships.”

The Spatial Knowledge Graph

With the graph-like structure of knowledge established, let’s get back to the original question:

(How) Could a spatial UI let you work with, and work within, the true shape of your ideas?

Ideas are knowledge, and the “true shape” of your ideas is the way that the graph-like structure of knowledge is manifested within a collection of ideas you’re working on.

Analogous to the way that language shapes what we can think, or architectural drawing techniques shape what we can build, our tools and methods for working with knowledge shape what we can come to know.

If knowledge really is graph-like, then the most effective strategies for seeking and creating it would be those that treat it as such. Tools that obscure or ignore its underlying structure would limit our ability to do knowledge work.

The issue with graphs is that they are notoriously difficult to represent in a usable way on the kind of bounded, 2D interfaces that computer monitors encourage. Graphs are large, nonlinear, and multi-scalar, three qualities that 2D UIs struggle with.

If our flat tools are limiting our ability to work with the graphs of our ideas, then they’re limiting how well we can understand and develop them.

SoftspaceAR Prototype04 proposes a new paradigm for representing knowledge graphs—one free of the limitations of 2D UIs by virtue of its use of spatial computing.

• This paradigm consists of a 3D spatial canvas in which user can create text blocks.
• Any terms in the text which are surrounded by double square brackets are interpreted as topics to which that text refers (this syntax is borrowed from Roam).
• Text and topics are our informational nodes, and the references from text to topic are one of two kinds of possible relation.
• The other kind of relation that is possible is inclusion: topics can be expanded, and text added to that topic as a way of defining that topic.

A 3D forced-directed graph simulation running in the background automatically tries to move related items together, while pushing unrelated ones apart. As the user creates, relates, unrelates, and deletes pieces of information, the system is constantly updating and displaying the overall shape of the knowledge contained within.

Prototype04 shows some of the advantages of working with knowledge graphs spatially:

• A spatial interface offers much for room to display the nodes of a graph. In fact, this even allows the graph to display individual text blocks as nodes, instead of the entire pages that many 2D graph views settle for.
• The global shape of the graph is generated from the local, node-to-node interactions. This global form is an important part of the meaning of the knowledge graph, because it suggests what the higher-level cluster or syntheses of the details could be.
• Object permanence makes it expected and intuitive that items would be includable in multiple contexts This makes it straightforward to implement true transclusion.
• While working on a given node, you can see related nodes, as well as second-, third-, or higher-order connections receding into the distance. This creates a sense of ambient context, so you always know “where” you are within your ideas.

How to use Prototype04

1. When you launch the prototype for the first time, a pre-constructed knowledge graph loads.
2. Click on any text block to start editing it.
3. Press Return at the end of a text block to create a new text block, and press Backspace when typing on an empty text block to delete it.
4. As you write on a text block, terms enclosed in double-square-brackets are interpreted as topic references.
5. Prototype04 will find or create the topic with the matching title (case-sensitive) and link it to the text block where you typed out the term.

6. As more topics are created, and more text blocks refer to different topics, the system automatically draws related items together and pushes unrelated ones apart—building up the shape of your ideas as you write them out.
7. So far, it seems like topics are just tags for drawing related text blocks together. But as their name suggests, topics are epistemic entities that can contain content.
8. Click on a topic’s title to expand it into a board. Click anywhere on a topic to create a pre-contained text block.

9. You can also grab an existing Text Item and add it to the topic board, or pull text items off and let them float away.

10. Clicking on the topic title again, or expanding a different topic, will collapsed the currently expanded one and allow the contained text items to float away—while remaining connected with a bolder connection line, to indicate a containment relationship.

11. A text item can be contained in any number of topics—the system has true transclusion.

12. Topics are created automatically when referenced for the first time, and deleted automatically if they are no longer referred to and contain zero text blocks.

13. Your work isn’t stuck in the headset! Each topic automatically exports itself as a markdown file, making it easy to take your work in Prototype04 out into the rest of your workflow. Inspired by Conor White-Sullivan’s highlight of Lenses in the Leo Editor, these .md files are called Lenses.

Known Issues

• Text blocks and topic titles that are far away become difficult to read—this is a major limitation. I have an idea for a text LODing system that would address this issue.
• Links are currently represented using a simple straight line between the linked items. These get in the way of text, and don’t tell you much when you can’t see the other end. They generally need design work.
• Prototype01 took eight weeks start-to-finish. P02 took four weeks; P03 took five weeks; and this prototype took me six weeks. I want to reverse this trend and get the build cycles back toward four weeks or less.

Next Steps

• Currently, topic boards use a single column layout, but there’s no reason why each topic’s internal layout system couldn’t be a full ordinospatial layout like the one in Prototype03.
• I think this paradigm will really shine with the (re)introduction of images and PDFs.
• The only way to create new content right now is by typing out text, but I fully intend to port over the very flexible in-app web browser from SoftspaceVR. This allows you to save out full-res images, copy/paste text, snip any part of the browser window into PNG, and create website bookmarks with a full-screen preview.
• Only one topic can be expanded at time right now, but we’ll probably want to let users keep several topics open at once; this raises the question of how to represent content blocks that are transcluded across multiple expanded topics.
• For development purposes, these prototypes write to, and read from, the headset’s local storage, which is a real pain to access. Production builds will use a Dropbox backend at first, and expand to other backend options later (IPFS anyone?).
• In this prototype, there’s a hard distinction between text blocks and topic items, but this distinction isn’t necessary. It would be far more elegant and flexible to allow any content block to become a topic container, so that ideas can be defined recursively.

Open Questions

• In practice, how and how much will people leverage the ability to truly transclude content across different topics? How should transcluded content be indicated as such?
• What is the right ritual for using a tool like this? Roam encourages a daily routine of use, with a privileged starting area (the Daily Notes) for collecting writing before the user knows where that writing belongs. How often should people use Softspace, and where do they begin when they don’t know where to begin?
• The ability to filter and query the graph will become critical. How should that work?

Acknowledgments

One of the most beautiful and rewarding consequences of building SoftspaceAR in public has been the incredible support I’ve received from others working in this space.

More than any previous prototype, Prototype04 could not have happened without the generous input of time and ideas from people who know a lot more about these questions than I do.

So I’d like to send a big “thank you” to:

• Andy Matuschak
• Conor White-Sullivan
• Jeff Tang
• Rob Haisfield
• Brendan Langen
• Rack Rusher
• John Underkoffler

That’s it for now! This post is the longest one yet, coming in at just over 2800 words, so if you made it this far—thanks and congrats 😄.

If you’d like to support this work, please consider following us on Twitter, trying out Prototype04, and joining our Discord to offer your questions, ideas, and criticisms.

8 November 2022

SoftspaceAR Prototype05

Toward a real tool for real work.

SoftspaceAR Prototype05 is out!

You can get the APK here, then use the Meta Quest Developer Hub or SideQuest desktop app to install it to your Quest, Quest 2, or Quest Pro headset. Please give it a go!

In February of this year, I started to build and release these prototypes to explore an exciting new possibility space: productivity and creativity tools designed natively for AR headsets.

In the announcement post for this project, I wrote that:

Over the coming months, we will release 5–10 prototypes to explore and (in)validate promising ways to harness augmented reality as a medium for thought… I expect the first few prototypes to be all over the map, the next few to start converging on core underlying principles, and the last ones to build more deliberately on what came before.

And that is indeed what is happening. This, the fifth prototype in the series, is the first one that feels like it’s building more on top of its predecessors than striking out on its own into new territory. It is heavily based on Prototype04, while also drawing on key lessons from Prototype03.

This doesn't mean that the experimentation is over. There are still aspects of the SoftspaceAR UX and product story that need prototyping and validation. But I've built out enough versions, and have tested them with enough people, to have a good feel for which ideas make sense, and which probably don’t.

In addition to a wide range of quality-of-life improvements and bug fixes which I will not bore you with, Prototype05 contains four major features that move SoftspaceAR toward being a real tool in which you can do real work.

These are: the web browser, images support, text item LODing, and multiple open topics.

Web Browser

Being able to access the internet is a core part of any contemporary knowledge workflow. Prototype05 implements a personal web browser to give users access to the rest of the web from within the headset. Given how many of the other tools that users rely on have web apps, this browser becomes a powerful and flexible window into existing workflows.

When browsing the web, if the user hovers over an HTML element that is identified as an image, a "download" icon appears to indicate that this image can be saved into the current workspace. Long click (or click and pull toward yourself) to save out the image.

Future versions of Softspace will enable website bookmarking, and the ability to snip portions of the browser window as images.

Images

As you can read in our origin myth, Softspace was born during a research residency I had in an art and design studio. Immediately before that residency, I had been in architecture school. Today, architects and designers remain a core audience for this tool. Therefore, visual research, reasoning, and communication are critical use cases for Softspace.

These use cases, of course, require the ability to work with image files.

Prototype05 ports over the image processing and displaying modules from SoftspaceVR. Right now, the only way to bring image files into the workspace is by saving them out from the web browser, but we're already at working on a Dropbox file importer.

Side Note: Image LODing

One of the (many) interesting (and annoying) new problems that spatial computing presents is the question of how to technically display a large number of images in a 3D space.

Unlike on a conventional scrollable 2D document, where only a limited fraction of a document's contents are visible at once, a 3D workspace allows the user to potentially see many hundreds or thousands of images simultaneously.

At the same time, users are able to view images from very close up; to avoid blurriness, the system needs to be able to render images at very high resolution.

Given the limited VRAM of any computing device, not to speak of the even lower constraints of a mobile device like the Meta Quest 2, it would be impossible to display all these images at the highest possible level of quality all the time.

Therefore, Softspace implements a novel Level-of-Detail system that uses the apparent angular size of each image to assign it an LOD value, which then determines which version of the image texture to load and display. These assignments are updated several times a second, and images transition seamlessly from 256px preview textures to 2048px full-res textures, as needed.

Text Level-of-Details

One of the issues users experienced in previous prototypes was that text became too small to read when the workspace was scaled down or move far away from the user. Once the text of the workspace became illegible, the workspace become nonsensical—a pretty visualization of something without meaning.

Prototype05 takes a first pass at rendering text bodies at different Levels-of-Detail, depending on text item size and distance relative to the user.

Just before a text block becomes too small to read, the text body is replaced with larger font. To prevent overflow of the text item bounds, this larger text is truncated. I find that being able to read the first few sentences or words of text items is enough to give me a much better sense of what different areas of the workspace are about.

Extra points: a more sophisticated version of this system could display an auto-generated summary of the text (e.g. using AI) instead of just using the first sentences.

Multiple Open Topics

In Prototype04, only one Topic item could be expanded at a time. If you expanded a Topic while another was open, the first would automatically be collapsed. The intention behind this design was to permit intuitive transclusion of content items (text, images) across multiple Topics.

However, a common point of feedback was that Prototype04 felt too messy and chaotic with all the Topics and text items floating everywhere. People who tried both Prototypes 03 and 04 tended to prefer 03's single fixed layout, especially while creating and editing content.

Prototype05 partially solves this problem by allowing multiple topics to be expanded and positioned next to each other. (If two topics share any content items, then they will still be mutually exclusively expandable).

I'm also working on a way for future versions of Softspace to have a single, global ordinospatial layout, like Prototype03 has.

Known Issues

An incomplete list of things I want to improve in upcoming prototypes:

• Arm fatigue. As originally pointed out by Andy Matuschak, and subsequently raised by many other users, the point-of-view based ray cursor is tiresome to use in a way that a mouse, or even the Oculus OS laser pointer, is not. I have been sketching out improvements on this front.
• Cursor icons. It’s about time to swap out the red debugging sphere that represents where the cursor is pointing in space with a proper set of icons.
• Topic ergonomics. Topic items currently maintain a vertical orientation, which can cause strain when you spend a lot of time writing text or moving images around on one. I’ve investigating more ergonomic orientations when a topic is actively being edited.
• Click-to-focus. Often, users want to get a closer look at a particular item without having to “swim” themselves over to it. I’m thinking about implementing a double-click-to-focus interaction that makes it much faster and easier to inspect things.
• More UI feedback. SoftspaceVR relied on hand controllers, which for all their downsides, did provide excellent haptic feedback when the user was interacting with things. In a tracked-hand interaction model, we need much more visual and auditory feedback to let the user know what they’re doing in the UI, or what will happen next.

As always, thank you so much for taking the time to read this! Your comments, ideas, and concerns are always welcome. You can help us on this journey by:

• Getting and testing this prototype 🧑🏽‍🔬
• Following and retweeting us on Twitter 🐦
• Joining the Discord channel 👯‍♀️

The Prototypes formed the basis of the SoftspaceAR design paper and the SoftspaceAR app which launched on the Meta Quest Store.