Streaming LAS files

[brycefrank]

@tmontaigu

Moving us over here for now since this is not lazperf-specific.

I don't know much about LAX files and what they allow us to do in terms of queries,
but if what we can get from the LAX file queries is a bunch of point indexes then we might be able to get something with acceptable performance in pure Python for LAS files.

So far my solution is to take advantage of the memory mapping that laspy provides. Here is a little example. This is not how the package is structured yet, but where I think I am headed.

Something I will be able to get from laxpy is a set of point indices that are in some quadtree cell. We provide the cell index as an argument:

my_cell_index = 512
my_point_indices = laxpy.query(my_cell_index)

.lax files are rather small relative to their "parent" .las files, so parsing these in Python is no problem so far, although I have only used fairly small .las files.

query returns a numpy array of all of the indices associated with that quadtree cell (not all of them are entirely within, see this video for an explanation). I can then just instantiate the laspy memory map and get the points. This seems to be reasonably efficient:

my_las = laspy.file.File('my_las_file.las')
my_las.points[my_point_indices]

So What is the Problem?

This is all fine and good, but we have to rely on lasindex itself to construct the .lax file. What would be nice is to do this ourselves via Python or Cython or something. That way it is a "complete package". If you refer to the video I linked above, Martin streams the points in one by one to construct the tree. As you have pointed out, doing this in Python itself is incredibly slow. For reference, lasindex can do this for ~600mb las tile in about 5-10 seconds...for the Python version I got tired of waiting!

The other solution is to just raise up our hands and load the entire point cloud into memory, construct the tree and write it to file. But this is silly, I think. The whole motivation behind creating the index in the first place is for querying large las files.

Any ideas you might have would be great.

[tmontaigu]

I’ll need to watch the video to have a more complete understanding of LAX files but what I can tell is that:

Yes, construting LAX files in pure Python in streaming mode is not an option as it is painfully slow. Loading the whole file is not a solution either as as you said, it sort of defeats the propose of LAX.

If we were to re-do it ourselves we definitely need a faster language like cython, Rust our Cpp, I’d be more in favor of C++ In case we want this to be accessible to other languages

As for using the mmap feature of laspy,
Yes this is what would give us the best performance when querying, however I see 2 drawbacks:

• a minor one: the file must be present on disk (and not on AWS S3)
• a major: LAZ support would be ‘broken’ because when laspy opens a laz files cannot mmap it, so it asks laszip to decompress the file in memory and then map the result. So effectively when a .laz is opened by laspy the whole uncompressed data is loaded in memory so the LAX file purpose is also sort of defeated. For this part we’ll need to interact lazperf to avoid loading the whole file

[brycefrank]

Thank you for your thoughts! This is very helpful.

If we were to re-do it ourselves we definitely need a faster language like cython, Rust our Cpp, I’d be more in favor of C++ In case we want this to be accessible to other languages

On this note, then I think calling lasindex (from lastools for example) is a fine solution for now. It is a C++ program already, and replicating it would be a good bit of effort. It is also a FOSS piece of lastools.

As for the LAZ support, it is a bit of a puzzler. I will poke around lazperf and see what options there are. I think, as a general note, LAZ support for Python interaction with LiDAR data is a larger theme that needs to get sorted out.

[hobu]

There is a class in lazperf that could be used to do per-point decompression of LAZ data. There is a branch in either the laspy or lazperf tree (I can't remember) where I implemented something, but I didn't get it finished.

[tmontaigu]

I agree that calling LAStool's lasindex is totally fine for now.

I don't think having a per point decompression will be hard to have with lazperf, as it was how it used to be done in the ptyhon wrapper wich rendered the process of decrompressing LAZ in python very slow (hobuinc/laz-perf#41).

Lazperf's VLRDecompressor takes as an init the whole buffer of bytes of the compressed points in the file. LAZ points are compressed in chunks (5000 per chunks by defaults if I'm correct) to avoid loading the whole point buffer in memory in order to process it maybe we could just send the bytes of one chunk at a time and process it per-point, this may be a workable and quick to test things out first)

[brycefrank]

This sounds do-able. In the next couple weeks I will plan on dipping into lazperf and learning a few of the internals.

To avoid a lot of work, I am wondering if we should think about combining your efforts in pylas into laspy first. Eventually I would like to "insert" laxpy into laspy at some point and I am structuring it such that that will be easy. I saw around GitHub that is something that you are thinking of doing an overhaul of laspy. Is it something you have started or are planning on doing soon?

[tmontaigu]

I tried to do an overhaul of laspy but I found that it was hard to get where I wanted with the current code so I restarted from scratch and ended up with pylas

[brycefrank]

In this case maybe I will think about merging this into pylas once it is a little bit more robust? It seems clunky to have laxpy as a standalone package. Thoughts?

[tmontaigu]

There are two possible choices:

• merge into pylas/laspy
• leave it as standalone and make pylas/lapsy (optionally?) depend on it

I don't know which is the best, and don't have any strong preference for now

[hobu]

For those still following, I have made an update to lazperf that will push wheels for all platforms as part of the PyPI release. This may change the calculus about LAZ support in laspy if someone can take on the rest of the task.