Eyes on Final Fantasy Forums > Reference > General Archive > Feedback Forum Archive > The deal with the thread pruning

[Citizen Bleys]

Alright; so you're probably wondering what the thread deletion has to do with today's downtime. It's a bit of a lengthy (and somewhat technical) explanation, so I thought it best to leave it out of the announcement in order to keep it brief.

All I tried to do this morning was upgrade the boards (completed in about two minutes) and rebuild the search index in order to improve speed. (The search index is used every time you open a thread, not just during searches). The first step to rebuilding the search index is to delete the old, obsolete index. Thus, once the rebuild is started, it MUST be allowed to procede to full completion; Monkeying around in threads while the rebuild is going on can lead to data corruption (I'm posting this during the rebuild, but I know precisely how far along the rebuild is coming: It's handling threads 11900-11925, and this is thread 20352, so I'm safe).

Rebuilding the index should take five minutes. Dunno if you noticed, but it's taking a bit longer.

Our database is now so huge that the rebuild keeps timing out. There are simply too many old, dead threads cluttering up the works. This has to be done. I've not consulted Scottie or any of the other staff about this--it has to be done, even if everybody doesn't see it.

vBulletin will allow me to prune threads by date of last post--hence, I'll be deleting anything in which the last post was more than 90 days ago. However, it runs through ALL forums, including the archive forums. I don't know a workaround (if you do, please post a reply in this thread ;p).

I'll do a full backup of the database, but for security reasons*, I cannot release the backup to the general public for download

*you shouldn't have to think about that very hard about that one. Everybody's password is stored in the database.

What I'd like to know from you, is
a) How often would most of you be able to handle a pruning operation? Once every month? Once every three? It's got to be done sometime.
b) do you know of any alternatives--Maybe an app that will save the threads? It should be simple enough to write--Just do an SQL dump of the table and then write a php script to parse it...But it's got to be a server-side app, and I have to have access to the source code, so that I can be sure that ALL it's taking from the database is the threads to be saved. We (EoFF staff) are not going to allow even the slightest chance that somebody's password will be taken out of the database.

[Griever]

I think once every two months would be sufficient.
As for the passwords, i think everyone'll just have to change them more often. Just regular changes, some people might accidently let them slip during a conversation or something. I did it once, but i was stupid back then.

[Super Christ]

Hmm - I have no luck finding any such utility or hack. Perhaps if, just before the pruning, an administrator adds one post to the end of each thread in at least the General Archive forum, and then after the pruning, deletes them. Just a thought. As far as threads in other forums, while there's a thread or two I'll miss, I won't miss them enough to really worry about it. Ahh well, that was just a thought.

[ShlupQuack]

Thank you, Bleys. =^_^=

[Nait]

Of course it is sad.

Oh, well. C'est la vie.

[Squally Leonharty]

Hmm, I think there's a way to save the archived threads. I'm not too sure, though.

I believe there's an option to prune threads PER forum, so why don't you do that? It's more work, of course, but at least you're able to save the Archive forum.

I think it's going to take you about 30 - 45 minutes to do all of the forums. Just that in one day. It's not that bad. It has to happen someday, right? Well, yeah... I got an idea how to backup the Archive forum, but I don't think it's possible to write a script for it, since the mysqldump doesn't allow partial dumps of tables (meaning certain threadids and posts). It only dumps the whole tables. I'll look in backup.php and try to get the code from there and edit it.

Edit:

Bleh, I skimmed through the whole code and it looks like it is possible after all. But it's gonna be very hard, and I can't really bother with that. I'm not THAT skilled. Oh well. I guess you'll have to prune per forum.

[Citizen Bleys]

And Squally says the obvious, and it hits me in the head like a hammerblow after a night of hard drinking. Of sake.

Yeah, we can prune by forum. *sheepish look*

[Squally Leonharty]

*makes fun of Bleys*

Just kidding. At least you know it now. Have fun with pruning.