Site Development Notification and Feedback Thread

@Background Pony #5126
This header should not be added since opening the image is not supposed to force a download of it.

@byte[]
Pretty sure inline does not force download, only attachment does.
Test it in your browser:
https://blogger.googleusercontent.com/img/a/AVvXsEhCFyjRbPDci-eCnDWOWDNTOfZ7V7XNojBQqdcJQWk8CKoBe9tkhi_xrixR53YK-UZV_MGHIEOzxSxWOM0B0OhqPVUX3B_cuENYOOl9RYqFQo_1jHJktT_BRGcQLaMsk2GjrubpIZMwcMfNYEQ5oWw3ORBWhLns-UHWecCpHk-1i-RnRBjYpolRY7i34A=s1100

@Background Pony #5126
Yes, inline is fine (you edited your post to reflect this).

Because some object storage offerings have recently become extremely compelling, and to be honest because it’s a pain to manage millions of image files on a local server, I am once again planning a move in the application code to take images from locally hosted to cloud hosted. Because cloud object storage solutions generally don’t allow symlinks and path-based header rewriting, this will involve changing bunch of image URLs into a permanent canonical format. I haven’t decided yet what this canonical format will be, but it definitely won’t support triggering a download of the image, and it won’t support tags in the filename.

This is not the first time I have proposed scrapping the duplicated file paths. However, to make it easier to swallow this time around, if the API downloaders are insufficient or inconvenient for your purposes, I will try to address what people are actually using the download with tags in the filename feature for: downloading bulk images from your browser.

This will generate a ZIP file on the fly from a selected source and download it as a stream; it will not take up all of your memory while it’s downloading. The source could be all of your open tabs, or it could be the contents of a search query (not sure if I should include this one or not). Importantly, you can configure whether you want short or long filenames, and included metadata or no metadata.

It will also be possible to download single files, but as before, will go through a clientside script that rewrites the file name for your local machine. (This feature depends on ServiceWorker availability and might not work if you are using private browsing mode.)

I understand that this solution won’t appeal to everyone. Leave your complaints below, because while there is some working code, this downloader is still essentially in the early prototype phase and is small enough to be rewritten if there is need for some major change.

@byte[]
How this “ObjectStorage” will play with TOR, proxies, VPNs, and IPs (unjustly) marked as “spam”? I won’t 403 you or something?
How about mass-downloader scripts that downloads gigabytes (or more) of images? It won’t ban or throttle you for “unusual” traffic?
What about hotlinking?

@Background Pony #5126
There should be no observable change in behavior for all of those situations. The main issue which will affect users is the filename changes.

@byte[]
Will it break old direct links, or there will be redirects?

@Background Pony #5126
Old links will almost certainly be broken.

I was honestly under the impression that being able to see an image onscreen in the first place meant that your device had already downloaded it; is the site serving thumbnails or condensed versions instead?

@TexasUberAlles
If you click (or tap, if you’re a phone poster) the image, it’ll download the full resolution. I learned this trick as a necessity to view details in some comics. The default is tuned to the resolution of your viewport, but you’d have to ask byte for specifics.

@TexasUberAlles
This refers to the difference between View and Download (one just opens the image, the other triggers a download prompt), plus the ability to offer an image with (some) tags in filename.

‘Save Image’ and ‘Drag ‘n Drop’ type of image saving shouldn’t be affected.

@PUBLIQclopAccountant
The site creates a number of “preview thumbnails” of various sizes to best fit the current window size; it avoids always transferring the full image size when it’s not necessary at all (ie. when someone’s on a 1080 screen and the image is 4k). Clicking on it will trigger loading the full size. This change won’t affect that.

@The Smiling Pony

@byte[]
Not having the ability to basically wget images with the tag names will definitely be a pain point for me.
Also, why don’t you do it on the server side?
I used to work on an application which served hundreds of millions of files to the “users” (other programs). Each request were internally redirected to the given object (azure blob storage), the user did not see how the item was stored. A lot of items was mapped from multiple urls to the same object. Even before the blob storage there was no such abuse of the fs what you mention. That’s what databases are for. Look up the path in that and serve the resource on whatever request URL regardless where it comes from.

Right now I’m running a smaller fan site (won’t name it) with around 500k items, each can be served on 3 different URLs, and there is only a single file for each item. Literally 1 line of inline perl in the nginx config to tell nginx what and how to remap. The generated html page contains a link which is generated by the server side code and nginx knows how to map the generated urls to the actual resource. That 1 single line of perl code even maps the objects to a simple tree structure to avoid having 500k files in a single folder (that would kill the fs and drag the iops down hard, does not matter what the backing storage is capable of). Yes, yes fs caching and stuff I know, there are reasons I don’t have them enabled/not sufficient amount.

And with this change (if there is no other option) you basically kill any other users of philomena. Not everyone want’s to have to use that specific object storage service.
Also buy your own hardware, and forget hdds. I just don’t get how a proper large nvme ssd’s millions of IOPS are a bottleneck. Especially if multiple ones are in a mirror mode where the reads can be distributed. Paying for an object storage service is more expensive. A lot more, especially considering the traffic.

I suspect that if you’re already at the point where you’re using wget, you can automate renaming them

Because then I can’t take advantage of CDN caching

The nginx configurations I have currently are similarly small; but it breaks caching

I already have the consideration of other philomena users in mind. They will get whatever the development environment gets, which would either be a file storage plugin or a local object storage server like minio. I would never require storage to be tied to an external service.

If you look at the links I provided, you will see that these object storage services are egress-free, and the relative cost is less than renting equivalent amounts of dedicated or block storage.

Great idea, I’d love to! Just one small problem: who will pay for the upfront cost of that?

Suggestion: downgrade hidden tags to mere spoiler tags specifically when browsing my:faves. I’ve also had hidden tags cause missing episodes from comics, but I can’t think of a good way to square that circle at the moment. However, it’s annoying when a favorite image disappears because someone added a tag you’ve hidden to it and you specifically chose to add it to your favorites collection so that you could enjoy seeing it again.

Not to mention that kind of kills scalability. Plus, cost of colocation can be higher than what I estimate the cost of operation would be post-move to cloud. Then in 5 years when we run out of storage on that shiny server we bought, we have to drop lotsofmoney to either upgrade or replace the obsolete server, also requiring untrivial amounts of manual labor to move everything.

Also, I suspect your site doesn’t get to the point where if you don’t cache you run out of server bandwidth instantly.

@byte[]
Will jpg and png still be supported?

Tag suggestions for uploads demo

NSFW variant

Loads the 50 highest frequency tags for a random set of images that match the tag input query so far. Needs some tuning as the results are not currently super relevant or ordered in a sensible way, and 50 is probably too many.

The tags are all clickable and add themselves to the input when clicked. Hopefully this will lead to less barebones tagging in the future when the feature is finalized.

The button is just for demonstration purposes, you won’t actually have to click anything to load the suggestions. The suggestions will load on their own every once in a while.

@byte[]
Nice to see this coming back as an official feature, the userscript for it has been broken for quite a long time

@byte[]
Will this only be on upload? Seems like it could get annoying in normal tagging.
Somehow I completely missed that there’s a fetch button, ignore me.

Not something I would ever use, but if it helps prompt people to stop uploading images with four or five tags it’ll be worth its weight in palladium.

@TexasUberAlles
Yeah, this is more a feature for new users that don’t know many tags yet. (Or old users that just never do tags)

@TheGlitchedWolf
For me the userscript was really helpful in making tagging faster, one click as opposed to typing tags out

Multiple sources