Database dump survey

@SviMik  
Reverse search code is public, see post here (and corresponding gist here)
 
Intensity information will likely be included

That’d be interesting.

@Joey  
Anonymize other interactions?

@Luna  
Upvote/Downvote counts are already available on the image metadata. I’m not really a huge fan of publishing hide counts.
 
Anonymizing them by assigning each user a unique ID isn’t good - at best someone could correlate data to create anonymous profiles on people, at worse they could use those correlations to actually start identifying people.

Wow, thank you!
 
I will be happy to see the following:
 

Sounds interesting. I’d use it just to find out things like which ponies get the most fan art.

never heard of it

What does that mean

For the data dump itself, I guess everything that’s publicly available should be in there. But perhaps the images could be in a separate dump? They take up way more space than metadata. The primary dump could include only thumbnails for quick perusing.
 
How big would the initial dump be? Will there be deltas available, so that one does not have to download everything again every day, but just few hundred megabytes of changes (or more if the images are included)? I could use these numbers, because I need to assess how big hard drives I’m going to buy.

I wouldn’t mind anyways about the changes..

@Joey  
I’m a mobile user so I didn’t really know I could see the time. Pardon that.

@JP  
As I understood it’s only a DB dump, it won’t include images. For sharing such amount of images, something like IPFS would be a good idea.

I don’t see any reasons why these dumps should be provided unless you are already facing some (probably minor) problems with site which can lead to it’s shutdown (i wish it wouldn’t shutdown).
 
The only useful thing about this dump is that other developers can create an alternative web interface for Derpibooru. And considering lack of most modern features (infinite scrolling, ajax loading, interactive javascript based interface), I’m pretty sure someone would find those dumps useful.  
Just make sure those dumps contain only data faced on your endpoint public API.

There seems to be some dictionary disagreement as to what dumping means. Do you mean copying, back-up-ing, deletion, or some other definition? Will this be for the “public-facing files” only, meaning that any hiccup leading to restoring from backups might mean the vanishing of material of a bootytastic nature?  
1- I guess that’s ok?

Dump in Database Computing is a file which hold data in it’s “raw” form (it is common that even human can view it as regular text file, and this file can be recognised by wide range of versions of same RDBMS), and you can restore table/tables/entire databases from dumps by importing them

@JP  
With the current amount of data that have been requested, it would likely be 1.5-2 GB for each one, though it might be smaller if LZMA compressed afterwards.
 
I’m not going to generate deltas because it doesn’t make sense in the format I’m offering dumps in. We’ll be offering the full data file each time.
 
It will not include image files. Those aren’t in our database.
 
@DBot  
We’re not planning any downtime events or a site shutdown. This is (1) us fulfilling our role as an archive site and (2) providing a separate path for developers to extract data from the site without having to page through the entire site.
 
@Mr Zero  
Dumps are used in the context of the pg_dump utility. PostgreSQL considers them different from backups, as they do not reflect the internal representation of the database files, and (in general) only contain heap data and the schema statements needed to recreate the database files. In contrast, a backup would also include the on-disk structure of indices, temporary tables, and the WAL as it existed at the time.

Sounds interesting. Would be down for that.

So its all anonymous and does not identify users?
 
What developers want these data and what do they do with them?
 
(If its for securing DB sure, otherwise its hard to understand why this is a thingie…)

@silbasa  
User information will be included, but anonymous posts will remain anonymous.

@byte[]
 
Full comments, forum postings votes and all? Is this safe to release? (Not trolling just wondering…)

@silbasa  
Comment bodies, forum posts, faves. Anything that is already accessible via the API should be present.

I’m not a developer, but I certainly would be interested on seeing that just out of curiosity; I might even learn something from it

@byte[]
 
Why do these devs need this information then?

@silbasa  
So you don’t have to scrape the API and deal with our ratelimits if you just want the whole thing anyway.

@byte[]
 
Thank you for your time. My understanding of this subject is to limited to have an opinion as I do not fully understands what it entails. But I trust you devs running the site. As long as it is safe and sane I suppose…