Bugs and things to fix

[hushpiper]

Sooooo for fun, I decided to do a brand new install of simplemachines, import only the content-related tables (none of the logs or settings ones) from the broken database, and see what that did. It worked: none of the posts that were previously missing are missing in this install.

FWIW, there was an option to enable UTF-8 in the installation process, so I tried it twice to see what it did. It made no difference either way. I then tried it again, with all brand-new files freshly unzipped, but this time importing the full dump from the broken database, and the posts disappeared again.

[Sunflower]

Sooooo for fun, I decided to do a brand new install of simplemachines, import only the content-related tables (none of the logs or settings ones) from the broken database, and see what that did. It worked: none of the posts that were previously missing are missing in this install.

FWIW, there was an option to enable UTF-8 in the installation process, so I tried it twice to see what it did. It made no difference either way. I then tried it again, with all brand-new files freshly unzipped, but this time importing the full dump from the broken database, and the posts disappeared again.

Wow.  I love seeing a solution come together.  If the Known World had you folks working in the lab, I think they'd have a Rash vaccine by now! 

[Eich]

Sooooo for fun, I decided to do a brand new install of simplemachines, import only the content-related tables (none of the logs or settings ones) from the broken database, and see what that did. It worked: none of the posts that were previously missing are missing in this install.

FWIW, there was an option to enable UTF-8 in the installation process, so I tried it twice to see what it did. It made no difference either way. I then tried it again, with all brand-new files freshly unzipped, but this time importing the full dump from the broken database, and the posts disappeared again.

Wait, so... we have a solution?  Or does the problem still have a chance of happening?

[hushpiper]

Well--it's progress, anyway. I've narrowed it down to a single db table: the settings table. If I import a shiny clean version of that table from a fresh install into a broken version of the site, bam, the posts show up. Problem solved.

The issue is that doing that screws with... all kinds of things. The theme, in particular, goes all screwy because it's both installed and not installed at the same time. Post counts go wonky, member counts go wonky. So I really don't want to do it on the live site. If possible, I'd prefer to see if I can fiddle with it and find the precise settings entry that's causing the problem so we can spot fix it instead of brute forcing it and causing who-knows how many problems. Will report back when I have more info.

[JoB]

I've narrowed it down to a single db table: the settings table. [...] The issue is that [reinitializing it] screws with... all kinds of things. The theme, in particular, goes all screwy because it's both installed and not installed at the same time. Post counts go wonky, member counts go wonky.

I take it, then, that that table's holding data about the global forum's settings, rather than the users'? Like ... (the recent changes to) the profanity filtering ... ?

[Puts elaborate theory about SMF UTF8 conversion and the DB holding both pre- and post-conversion versions of posts and SMF doing on-demand conversion back into the mental drawer]

[hushpiper]

I take it, then, that that table's holding data about the global forum's settings, rather than the users'? Like ... (the recent changes to) the profanity filtering ... ?

[Puts elaborate theory about SMF UTF8 conversion and the DB holding both pre- and post-conversion versions of posts and SMF doing on-demand conversion back into the mental drawer]

*stares at you*

Holy son of a ever-[insert long-winded string of things for that filter to work on here]-!

Yes. Yes, the profanity filters are stored in the settings table. And they are one of a handful of entries (along with member counts, post counts, newest user, etc) that I completely ignored when I went through that table row-by-row last night to try to see which one was making the difference. Went to the settings and took out the most recent addition to that filter: problem solved. Posts are back.

And holy bleep this is like Kiss Kiss Bang Bang, where in all the stories the detective has two cases but by the end they turn out to be the same case, cuz guys! Guys! This is what was causing the issue Sunny was talking about where words like "reciprocating" turn into "recipussycating" and so on. The substitution was treating the *'s used to bleep out bad words as wildcards. I can only assume this was causing some kind of strange bug with its post conversion that possibly we should let SMF know about, just in case. JoB, you'd probably have more relevant ideas than me.

Gonna change the news line to let people know.

[Piney]

Three cheers for hushpiper and JoB, forum heroes!    Really glad to see this all came together well.

[Richard Weir]

Hurray! Hurray! Hurray!

Thanks and congrats to HP and Job!

[Eich]

You two are a regular crime solving duo!  This is amazing!  And it was solved before the new year, too!   
Waaaaaaah! 

JoB, if you want admin status too, just say so (I'll try to get new titles for the both of you.  Solovei suggested "Slayer of Silence," which I quite like.).  Thanks so much guys, all of forumdom is in your debt, and you will be welcomed into the halls of Forum-Valhalla!

[Solovei]

Yaaaaay you guys are amazing! I could hug both of you right now!

[Fen Shen]

Great work and thank you so much! You are an awesome duo!

[Nimphy]

The Forum's Second Crisis (if the spammers can be counted as the First) is now behind us, thanks to our detectives! Yay!

EDIT: Now that I think of it, maybe it could be the third. There's still that time when Eich forgot to log in...

[Sunflower]

Yaaaaay you guys are amazing! I could hug both of you right now!

Yes!  It's a New Year's Eve miracle.  Laurel crowns of victory for both of you!  Thanks so much!

Signed,

A grateful (and verbose) writer of posts and PMs

[JoB]

*stares at you*
Holy son of a ever-[insert long-winded string of things for that filter to work on here]-!

So far for why I didn't inquire about the profanity filter - the other recent change besides the UTF-8 topic - right from square one ...

The substitution was treating the *'s used to bleep out bad words as wildcards. I can only assume this was causing some kind of strange bug with its post conversion that possibly we should let SMF know about, just in case. JoB, you'd probably have more relevant ideas than me.

Ehhh you might want to note that, while I can guess my way to how some application interfaces with the semi-known field of SQLDBs, I've never touched SMF (or any other forum software) and have no idea whatsoever what those profanity filter rules might actually look like. (Right now, I'm wondering how regex-style "*"s even can act as wildcards, given that DBs typically use "%" and the "*"s were in the texts rather than the filter expressions ... !?)

JoB, if you want admin status too, just say so

You, too, should take note of my actual level of ignorance (e.g., I have no idea what functions admin status even would unlock in the first place) before handing me the keys to the Minuteman silo ... !

(I'll try to get new titles for the both of you.  Solovei suggested "Slayer of Silence," which I quite like.)

Considering that the actual result (for now?) is that the profanity filter is back to its original overzealousness, "Four-Star Chaperone" might be more accurate ...

I could hug both of you right now!

... I didn't know they make arm extensions.

[Makes Mikkel look like a bantamweight fighter]

[hushpiper]

I'm totally doing the "the clues were right in front of me all along!" thing right now. I was scouring through the code just yesterday, I saw the censorText references over and over and--nope, didn't think it was important. Lesson learned: when working on two issues at once, don't be afraid to assume they're related.

As for the question about how it can use regex-style * as wildcards instead of DB-style %... I don't think the database is actually involved here. As far as I can tell, the post information is stored in the database uncensored: when a page loads, it calls the display.php file which calls the load.php file, which pulls the information from the database and runs it through the censorText routine which--dun dun dun!--uses regular expressions. And, apparently, doesn't filter them properly for special characters within the text to be filtered. (Or maybe it does? I'll be honest, the code in that section make my eyes cross. Maybe it only has problems when you try to loop filters like Eich's substitution spell was doing.)

Why that then zeros in on 64-character links of all things, I don't know--I saw some link scrubbing routines in another file, but I think those were meant to be preparatory to inputting into the db and thus unrelated? Either way, there's very little error catching written into the load/display sections, so presumably it hit the censorText routine, ran into problems and failed to finish loading the post but continued anyway--leading to blank posts and no error logs.

Yay theories?

The issue with the over-strict profanity filter is easy enough to fix though, assuming I understand the issue correctly. So far we've been bleeping out bad words with asterisks, which have the downside of apparently also acting as wildcards. (Which is a cool thing that could probably be exploited, actually...) Assuming looping filters is permissable, all we'd have to do is replace * with a character that doesn't have special uses in regex, like - maybe. In the meantime, subject to further tweaking, I've checked the option to have it replace whole words only, so pussycat is safe while p**** should be bleeped.

P.S. Ooooohhh I love the sound of Slayer of Silence! It's all cool and mythological whilst also being goofily over the top. Trust Solo to come up with the best titles!