Problem with QGIS: 3D view, tiles keep on reloading

I had an issue with QGIS 3.20.3 where I opened a large .laz point cloud (Amsterdam, about 2 GB; took 7 hours to create the ept [Entwine Point Tiles] cache folder). I can see the points without problems, however when using the 3D viewer at certain angles it seems like the point tiles keep on reloading and dropping out.
Note the tile count at the top right. At some point it goes back and forth without ever loading everything in.
This also makes the generation of a QGIS animation with such an angle during the interpolation getting stuck and never getting past that frame. There is no error message, only the animation generation progress bar that never advances at a given point.