The Stand Still, Stay Silent Fan-Forum

About the Site => Website Help and Rules => Troubleshooting Archive => Topic started by: hushpiper on December 09, 2015, 03:07:10 PM

Title: Server Migration II: Revenge of the Something-Or-Other (SOLVED)
Post by: hushpiper on December 09, 2015, 03:07:10 PM
Due to Life Things on the part of yours truly, the forum will be switching hosts. My husband has graciously volunteered the server provided to him by his work for the purpose, and the forum will be living there for the forseeable future. The move to the new server will be happening on Friday, December 11th promptly--*coughs*--at 8:00 AM US Mountain Time (GMT-7), and shouldn't take more than a half-hour.

Here's what you need to know:

The forum will be unavailable for the duration of the move. This may involve a friendly little message popup telling you we're migrating, or it may just involve an error message; both of these are normal.

The server move involves a DNS change. What this means is that even after we've finished the migration, it may take some time for the forum to show up for you. This should take no more than 4 hours; for the vast majority of you, it will be considerably faster. For a few of you--especially those of you who live far away from cities and/or have small internet service providers--it may take longer. If you still cannot see the server after about 2:00 PM mountain time, try clearing your DNS cache: https://www.whatsmydns.net/flush-dns.html

After some time has passed--let's say 24 hours, for things to settle--then you can start looking for bugs. (I know some of you are pro at that. *eyes you all suspiciously*) The new server has a slightly different architecture than the old, which may cause problems. We've been testing it for the past few days and don't see any big issues, but keep an eye out for anything suspicious over the course of the next week or so.

We're gonna do our best to keep this transition smooth. If you find an issue, report it in this thread, or PM myself or another admin. If you have a question--ditto.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Rabbit on December 09, 2015, 03:10:43 PM
December 11th?
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: urbicande on December 09, 2015, 03:11:06 PM
We're gonna do our best to keep this transition smooth. If you find an issue, report it in this thread, or PM myself or another admin. If you have a question--ditto.

And if we can't get to the server, make SURE to post it on this thread ;)

Good luck with it!
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: viola on December 09, 2015, 03:11:54 PM
December 11th?

Yes december 11th
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 09, 2015, 03:20:03 PM
Goshdangit dates, why do you torment me so? Yes, the 11th.

If you're not able to get to the forum at all (as opposed to seeing bugs on the forum), ping somebody in IRC or on the comic page comments. There'll be people watching both.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 09, 2015, 04:01:08 PM
The server move involves a DNS change. What this means is that even after we've finished the migration, it may take some time for the forum to show up for you. This should take no more than 4 hours; for the vast majority of you, it will be considerably faster. For a few of you--especially those of you who live far away from cities and/or have small internet service providers--it may take longer. If you still cannot see the server after about 2:00 PM mountain time, try clearing your DNS cache: https://www.whatsmydns.net/flush-dns.html
Assuming that you have the usual "full" control over the domain's DNS entries, you could set up another, temporary FQDN (say, merryxmas2015.ssssforum.com) pointing to the new IP when you start the migration. With that, people could even work around a provider's cacheing forwarder being broken and ignoring the TTL, if need be.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 09, 2015, 06:35:28 PM
Assuming that you have the usual "full" control over the domain's DNS entries, you could set up another, temporary FQDN (say, merryxmas2015.ssssforum.com) pointing to the new IP when you start the migration. With that, people could even work around a provider's cacheing forwarder being broken and ignoring the TTL, if need be.

I assume you mean that could be done so that people could use the merryxmas2015.ssssforum.com subdomain to access the forum instead? That would be doable--we could even point it now--but the forum software requires that you access the forum using the URL it has in its config. So even if the merryxmas2015 subdomain was pointed to the correct folder, most forum objects (and all links) would still try to load from the ssssforum.com domain, putting us back where we started.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: starfallz on December 12, 2015, 08:53:15 AM
I think email notifications haven't been going out? There have been replies to some topics I watch that I haven't been notified about.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 12, 2015, 10:28:23 AM
I think email notifications haven't been going out? There have been replies to some topics I watch that I haven't been notified about.

Yep, good catch. That should be fixed now.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 12, 2015, 11:00:34 AM
I think email notifications haven't been going out?
Looks like a general e-mail-out problem again. I tried sending myself one from the forum ~90m ago, no dice.

(Disclaimer: I do not actually control the mail server it's going to, but I have yet to find it having been the problematic part when expected e-mail doesn't arrive there.)
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 12, 2015, 11:26:00 AM
Looks like a general e-mail-out problem again. I tried sending myself one from the forum ~90m ago, no dice.

(Disclaimer: I do not actually control the mail server it's going to, but I have yet to find it having been the problematic part when expected e-mail doesn't arrive there.)

The SMTP settings in SMF were still set for the old server, rather than the new--I thought I'd fixed those during the migration yesterday, but it must've not taken properly due to the propagation (i.e. when I submitted the new settings it sent the POST request to the old server...). Moral: DNS propagation is fun, kids! So fun!
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: starfallz on December 12, 2015, 11:59:19 AM
I'm getting notifications now. Thank you Hush!
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Amity on December 12, 2015, 02:14:08 PM
I don't really exercise the forum features very much, but I haven't noticed so much as a hiccup -- nice work, hushpiper!
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: jmf (john_cephalopod) on December 13, 2015, 08:27:00 PM
I found a bug.
When I want to change my avatar by uploading an image from disk, it says:
Quote
The attachments upload directory is not writable. Your attachment or avatar cannot be saved.
Hosting it on an other site, e.g. imgur, works though.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 13, 2015, 11:09:11 PM
I found a bug.
When I want to change my avatar by uploading an image from disk, it says:Hosting it on an other site, e.g. imgur, works though.

Confirmed and fixed; was a stray setting that got overlooked in the move. Thanks for the heads up!
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Haiz on December 21, 2015, 08:53:47 AM
Minor thing, but every time im going on forum on my phone since the server move, a message telling me the website may be unsafe or that it doesnt have some sort of certificate pops up. I just click "ok, continue anyway" but yeah. Shrugs
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 21, 2015, 09:29:58 AM
Minor thing, but every time im going on forum on my phone since the server move, a message telling me the website may be unsafe or that it doesnt have some sort of certificate pops up. I just click "ok, continue anyway" but yeah. Shrugs
The server's using the same cert as before ... Qualys (https://www.ssllabs.com/) would like the CA chain reordered and longer, self-generated DH moduli, but neither should prompt a browser popup ... ? ???
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 21, 2015, 11:31:17 PM
This is most likely a mixed-content error, I fiddled a bit with some of the rewrites I had previously been using to force https on links that had been written with http. Which is to say, the cert itself is fine, the browser's just complaining. Thanks for letting me know Haiz, I'd forgotten about that one! It'll get fixed as soon as I figure out how I want to handle that. :P
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Haiz on December 22, 2015, 01:16:02 AM
It looks like this (sorry for the norwegian):

(https://41.media.tumblr.com/53b1ff0bd2cdedcf66ea1f12ae6fa91e/tumblr_nzq699vD0Q1r6soqlo1_400.png)
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: viola on December 22, 2015, 01:50:04 AM
It looks like this (sorry for the norwegian):

(https://41.media.tumblr.com/53b1ff0bd2cdedcf66ea1f12ae6fa91e/tumblr_nzq699vD0Q1r6soqlo1_400.png)

Security warning
There are problems with the security certificate for this site.
The name of the site does not agree with the name on the certificate.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 22, 2015, 08:14:50 AM
Security warning
There are problems with the security certificate for this site.
The name of the site does not agree with the name on the certificate.
Haiz, it would be helpful if you were to use that "view certificate" button the next time and tell us what's shown as the cert's "subject"/"issued to". The forum server's cert would report a "common name" (CN) of "www.ssssforum.com" and/or a "distinguished name" (DN) including that CN.

I currently see three possibilities, but neither seems to match the symptoms 100%:

[scratches head]
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Haiz on December 22, 2015, 06:01:24 PM
(https://41.media.tumblr.com/60cc858454b3fc91601e9467d7b8ed82/tumblr_nzs7rqaRd31r6soqlo1_400.png)
(The "sideinfo" just says that this page is ssssforum.com but I think you know that)

might be my phone is just old or something. It's really no problem, just a thing that happens
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 22, 2015, 09:22:53 PM
(https://41.media.tumblr.com/60cc858454b3fc91601e9467d7b8ed82/tumblr_nzs7rqaRd31r6soqlo1_400.png)
[Checks DNS and WHOIS]

Well, that still doesn't tell us which part of the forum's content refers to the external content in question and under what differing name, but it's 99% certain that it's the cert of some other server that causes this warning. (Also, you shouldn't see this warning during the login process or in the board/thread listings but only once you look at actual content within a thread - correct?)
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 23, 2015, 09:18:41 PM
Huh, that is completely not what I thought it was. So--basically what's happening here is your phone is complaining that something is trying to load using the forum host's SSL certificate, rather than the one that's installed on our website. Which... shouldn't be happening. The host cert shouldn't even be accessible, considering it's installed on a whole other IP. Maybe there's something in there that's hard coded to the old IP address? *scratches head*
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: JoB on December 24, 2015, 10:11:56 AM
Huh, that is completely not what I thought it was. So--basically what's happening here is your phone is complaining that something is trying to load using the forum host's SSL certificate, rather than the one that's installed on our website. Which... shouldn't be happening. The host cert shouldn't even be accessible, considering it's installed on a whole other IP. Maybe there's something in there that's hard coded to the old IP address? *scratches head*
If it were the old IP address, Haiz wouldn't get the server cert of the new host ... ?

I note that both alternate functional "hostnames" I can dig up (162.144.176.230 and ssss.personaldefensecenter.org - the IP's PTR RR points to 162-144-176-230.unifiedlayer.com, but there's no further RR for that), when used for HTTPS, take you to the SSSS forum without redirecting the browser to one of the "proper" hostnames (www.ssssforum.com and ssssforum.com) first, complete with "wrong cert" warnings. Maybe Haiz somehow bookmarked a URL with one of those?

(Dunno how much we want to care about SEO, but both in that context and generally, having a website nudge the visitors to one "official" hostname is considered a good thing.)

If you have logs that record either the HTTP 1.1 "Host:" header or the "Referer:" header, you should be able to find out what hostname Haiz is actually using (because the reverse proxy apparently rejects HTTP 1.0) ...
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: Haiz on December 24, 2015, 10:37:35 AM
uuuuhhh i don't know what you guys are saying but it's REALLY ok, i was mostly just alerting you guys in case of a bug, but it's honestly not an actual problem. it's just a popup i can easily click ok on.
Title: Re: Server Migration II: Revenge of the Something-Or-Other
Post by: hushpiper on December 24, 2015, 08:55:10 PM

No worries! I'm glad you're still able to use the forum without problems. We're just chewing over it because it could indicate another problem I hadn't noticed--it doesn't do to have the forum behaving unexpectedly!


Well seeee, the cert in that screenshot could refer to either the old server or the new one--both have the same host, and the server-wide cert installed by the host is identical. So that's no help.

As far as using the IP to access the website--well, yes, that would certainly cause those errors, among many others. I did have some things in place on the old server to handle redirecting requests like that, but they got changed in the move. It doesn't quite fit, but since that did change during the server move it's a good place to look for the culprit. The forum software trends to be very finicky when it comes to SSL and its related rewrites.