For those wondering why aliasing big tags takes so damn long and why we’re so reluctant to do it, here’s why.
The index on every image in the union of the two tag “sets” has to be manually updated in the search engine. This isn’t that slow; Elasticsearch can index about 5000 documents per second before the indexing throughput becomes noticeable in search. But we never get to that level.
The problem here is incredibly
inefficient processing of the images that are being built to be sent off. This might take 50-200ms
per image. Since one worker executes alias rebuilds at a time, at a best case scenario, that’s 50 seconds for 1000 images. Worst case, 200 seconds.
The obvious, quick-n’-dirty solution is to just add more workers and process the jobs in parallel. This doesn’t work like you expect; the server might have around 64GB of uncommitted RAM but it has to serve your requests too, and having its CPU cycles being stolen by alias rebuild jobs is not what we want to see happen. Especially with some annoying guy trying to DDoS us appside lately.
The correct and more scalable solution is to rewrite the system to process images in a bulk format and build a temporary identity map containing tag and user names while aliasing. This will prevent excessive database requests (the costliest part of indexing) and vastly speed up alias jobs.
Blah blah tl;dr shit’s fucked and I’m drunk again