A lot of my spare time over the past couple of years has been spent creating experiments, tools, and prototypes related to indoor and outdoor mapping. In particular, I’ve become fascinated with indoor floorplans and finding ways to describe the features in such a way that you can then take that description and use it to generate full 3D visualizations.
My tools are now mature enough that I think it’s worthwhile demonstrating how they work with some test scenarios. I started with a basic, clean floorplan, selected at random from a large selection, so as to not pick something I know will look good or work well with the tools.
The first step is to run this through my “Floor Plan Analyzer” tool, which creates a “description” of the spatial aspects of the floor plan. A “description” in this instance means a simple data model which is then stored in a JSON file. Small items like doors, appliances, units, baths, sinks, etc are ignored for this part of the process. Analyzer works in a number of discrete passes. Each pass looks to gather information about one aspect of the floorplan. That information is then stored for future passes to use, allowing each pass to get more complex and figure out more difficult problems. The first pass, for example, is only trying to determine where the walls are. At the end of this pass, it makes this wall data available to all future passes. So, on our sample floor plan, we end up with something that looks like this:
On the one hand, Analyzer has done an excellent job of isolating the walls in our floorplan. It’s grabbed literally everything we would want it to find and it’s missed nothing. Even the little isolated piece of wall between two cupboard doors has been correctly identified as a wall. On the other hand, the contours which define the walls are not perfectly accurate. In order to get good results, Analyzer needs sharp edges, particularly when working with relatively low resolution imagery such as this sample. It gets those sharp edges by thresholding the image (making every pixel either black or white, depending upon whether its value is above or below our threshold value). This, in addition to some other operations performed on the image, makes our contours either a little larger or a little smaller than what the human eye would see. This helps to ignore deviations in wall edges caused by image compression and low quality imagery. But remember that we’re not actually using the wall information for our end result.
We can come back and do that later. Right now, all we want to know is which sections of the image are walls and which are not. This information is only stored for the use of future passes. So, for our purposes, correctly identifying the areas and not pulling in any false positives is by far the most important element.
Analyzer now views the whole dataset again, this time combining it with the information on the walls it detected. In subsequent passes, it looks for doors and windows, attempts to find the exterior contour of the building, and finally identifies individual rooms within the building. The exterior contour is extremely important if you’re planning to create a complex visualization where your 3D building is combined with larger map data (eg: from OpenStreetMaps) as it’s the only way to determine the relationship between your interior data and your OSM world data. But, that’s (literally) a subject for another post.
Each of the passes in Analyzer is a complex process involving mathematical operations, image manipulation, machine learning, and a whole bunch of custom algorithms. If there’s enough interest, I may write in more detail on those passes in a future blog. For now, let’s fast forward to the results of our analysis on the floor plan.
This is our final data from Analyzer. As you can see, it has identified all of the rooms in our building correctly. It hasn’t separated the kitchen/reception room into two pieces but there’s really no way that it could and no real reason that you would want it to. The doors and windows have been correctly identified also. They’re displayed oversized in order that the resulting image is easy to read. Internally, they are perfectly accurate as you’ll see when we come to creating a 3D visualization from this.
In the picture above, I’ve highlighted the kitchen/reception room to show the fidelity of the results. Red dots are the points in each shape and the yellow dots are mid-points along each edge. As you can clearly see, there is no extraneous data here. This is incredibly clean geometry with no messy extra points. This is really important when we come to build a 3D model from this. If the 2D geometry is not clean then the resulting 3D model won’t be clean either. For a simple visualization, this might not matter, but Visualizer is going to create geometry with holes for doors and windows and it’s going to UV map everything as well. That simply won’t work unless the data from Analyzer is super-clean.
So as we finish up in Floor Plan Analyzer, we have a 21kb uncompressed JSON file with the results of our analysis. It could comfortably be a lot smaller too. I include names for all of the entities (which will be utilized in future developments) and I also include two copies of the geometry, as I’m using the results in two different applications. Even with all of the non-essential data included in it, the file still compresses down to 3.25kb.
So what can we expect to generate from our 3.25kb JSON file? Let’s jump right into that. Import the JSON file into Floorplan Visualizer, a separate application which is built in Unity and can be used either in edit mode or in play mode and it creates this:
OK, so this is a lot. Visualizer has taken our JSON data and used it to create some 3D geometry. There is no user interaction whatsoever at this point. This is completely automated, taking the JSON as the sole input, and generating this scene.
The exterior and rooms have been used to tell us where walls should be. Exterior walls use a red brick material and internal walls use a yellow brick instead. Because we correctly identified all of the rooms in Analyzer, we could actually use different materials for each room. But that might look a little bit garish at this stage, so Visualizer is just applying the same material to all of the internal walls. Similarly, floors are split room by room so it’s possible to apply different materials and textures. They are also correctly UV mapped. In this example, Visualizer has assigned the bathrooms a tile material. It’s currently just using a very crude method of determining which rooms to tile and which rooms have carpet. I have plenty of ideas for improvement in this area. The next step will be using optical character recognition in Analyzer to retrieve room names from floor plans that have them and then applying materials based on the name of the room in Visualizer. A more advanced approach I have in mind is to supplement floor plans with photographs and have Analyzer create material “hints” in the JSON which Visualizer will then use to find the most appropriate material it can find.
Here’s another view of the same visualization:
The information we have on doors and windows has been used to cut openings in our internal and external walls. Simple intersection tests are used to ensure that our cuts are clean and that we can match up both sides of the opening. Once it matches up both sides of an opening, Visualizer now has all of the information it needs to create entities to fill those wall openings.
For windows, everything is currently completely procedural. Visualizer has the ability to create a window, complete with the surrounding frame from just a position, scale, orientation, and the number of divisions. All of that information can be derived from the JSON except the number of divisions, which is currently being done by a combination of size and shape. In the future, it will likely be integrated into the JSON.
Doors are done with a completely different system. It’s a pretty complex system and it probably deserves a post of its own some day, but essentially it’s a 3D version of nine-slicing. Technically, it’s not twenty-seven-slicing because there is no limit to the number of slices on each axis. So you can not only prevent the corners of the mesh being stretched, you can pick arbitrary regions in each axis and prevent stretching there too. What purpose does this serve? It means I can take a single mesh (untextured, currently, as UV map correction is a bridge too far for twenty-seven-slicing) and make it fit any combination of width, length, and depth. Furthermore, I can do it without unnaturally scaling certain elements in the mesh. For instance in the picture above, the door handles are not being scaled even though the doors are incredibly narrow. The indented panels in the doors are similarly unaffected by the scaling. It’s a really nice trick and one that I plan to make use of when I come to adding units, furniture, and appliances to my completed visualizations.
The most obvious application for this technology is real-estate visualizations but there are many other possibilities too. This is all based on research that I began in anticipation of a new project for a very large facility management company. Unfortunately, that project did not go ahead as planned but it’s still an excellent fit for facility management work. The facility maps that our team used to build for clients like Amazon Go and Swarovski were imported from BIM software like Revit or Sketchup and they needed a huge amount of manual processing in order to be usable. Amazon Go maps took something like ten hours to process because of the huge polygon counts involved. Swarovski’s maps were considerably more lightweight but they needed more detailed fixtures and furnishings which all needed to be done by hand. So their maps typically took even longer.
With my tools, even if you have to spend some time tweaking settings and training the machine learning algorithms on a new map, you could still expect to have a complete map in an hour at the very most. And it goes without saying that each subsequent map would get easier and easier as the algorithms are trained. Because the map can be generated at runtime from just a few kb of data, it would also be ideal for anything which needs to be a web application. You can embed the models and textures in the webplayer (and use Basis Universal textures for a tiny memory footprint) so that 99% of the data the app uses is cached with the player. No need to load chunky assetbundles or other external resources.
Creating these tools has been tremendously rewarding. I think because I had visual progress indicators at every stage of development, making it easy to see results quickly. Now I’m excited to explore the applications for this technology and see where it takes me.
Here’s a work in progress screenshot from the latest WIP version of Visualizer which takes hints from room names and fairly simple rulesets in order to create fittings and furniture within the visualized apartment:
