Re: Web Server Upgrade how long is this taking? we just moved few weeks ago from a Host, now we moving to another server?
No. To explain things a little better, if it is possible at this point, what has been replaced is actually the web server software itself. Wait, did you just ask how the website actually operates? Here it comes...
This software contains added support for many, many more connections, handles dynamic content about 6 times faster (which most of the website consists of), and gives a 50% performance increase to PHP, which is the scripting language upon which the website is built. If this is a way to eliminate any mystery of what is going on, I hope that it can help in this reply.
So, in fact, the actual main server we use that houses the secured database, and in fact houses the primary web server, is in fact the same, and is of an extremely high capacity to the point that there is now currently tons of free memory. Normally I would rather not go into specific details on such things, but we have 24GB of DDR3 EEC and 16 processor cores on hand. If we ran out of memory, after that, paging would still take place on server-grade solid state HD technology, which wouldn't really be that slow. The problem is that dynamic content eats up a lot of CPU cycles, especially when thousands of people go to it at the same time. Imagine if your computer had 600-1000 people trying to connect to it every second. This is what we have to deal with.
With the new web server software, certain things are done better. A memory I/O cache is created and it integrates with measures we already have in place to make things faster. The software is specifically constructed to serve dynamic content better. When combined with hardware and software optimization, this creates much faster page loading. There is also a file-based cache where we are able to utilize temporary space on a separate drive for this purpose. Further, the number of connections per system per second can be limited fairly easily, so that flooding attempts and denial of service attacks are dealt with easier.
So, in fact, this is all a software upgrade, and the current server, which is of good quality and which has sufficient cores, with an excellent connection and support to match, is fine for the purpose of hosting this website. The problem really seemed to be a bottleneck at the web server level and an inability to handle an enormous amount of traffic. Add to this delays that we were detecting from overseas users due to the overhead problems and there we were.
Just brought up this, because I dunno but there should be some talking and thinking before we do stuff. I dunno if the admins have met yet, but some points to consider.
We talk every day by phone, text, and staff forum. We could strip the site's features to bare bones, and keep the website's main page very light weight and simple, but this would look somewhat boring and this is not really an option. We could remove the unique images for every forum and this would make the site load in under a second. But wouldn't this look horrible? Generally, the front page of a site should have a lot of important information that is easily accessible. And it shouldn't look half bad either.
It is good to have things like social groups and album pictures shown, even though they slow down page rendering significantly. So, upgrading the technology is a better idea from this perspective. We got to a point that we needed an enterprise-level solution to a enterprise-level problem. Rather than buy another physical server or something, which would be a somewhat wasteful use of resources.
How are we moving all this information around from server to server?Link Removed - Invalid URL
That depends on what information you are talking about. We have access to, and are now once again using, a content delivery network. Content such as images, JavaScript, Cascading Style Sheets (CSS), and anything else we want to really put there that doesn't change all the time is placed there automatically. Certain folders can also be sent there manually. We can't put everything there, but we could put up all static content like support files, attachments, and so forth. This distributes the content around the world. In fact, here is where those places are on a map:
North America:
Europe:
Asia/Pacific:
The ones in blue are still being built, so in fact, super-fast content delivery in Asia isn't available yet, but those people can still get to the data here by using longer routes. They have to get to our main server in Chicago anyway, but it would be faster for them when we distribute the content across multiple servers.
Every time you access this site you are using a database, possibly thousands of times, depending how long you are on. And this database houses all content ever created by anyone here since something like 2006.
This content delivery network allows us to host less stuff from one place - the content we know really won't change for a very long while. Not any database information, but the things I mentioned before. We also use Google to host some dynamic libraries created by Yahoo that we need and to monitor visitor trends to figure out what we need to do next.
This is all possible at relatively low cost, as CDNs have become less expensive over time. Hosting large support files, though, is a big set back. Its not possible for us to host a popular file for very long, since our bandwidth, like most other sites, is metered. So its a good thing that there are places like Softpedia and Peer-to-Peer networks. Peer to Peer networks (P2P), or BitTorrent, get a very bad reputation because of the abuse and piracy associated with them, but when used to host legitimate files that are digitally signed or generally good to have on hand, like support files, freeware, and so on, this is very useful to have.
Overall, everything has been done to make the website as fast as humanly possible. We are running the latest version of practically everything, and to my knowledge, we were getting a 2 second load time from an island in the Mediterranean Sea earlier today. Content pages load much faster because they have less database calls, scripts, and more text. For a webpage, this is not bad.
As the website's content continues to grow and number of unique visitors continues to increase, we will have to keep taking action. It won't be the last time some upgrade happens, but we shouldn't need additional equipment for awhile. Keeping the website running all the time is actually monumental task, and it has become harder as time has gone on simply due to the fact that our requirements keep getting bigger. The point is to store as much information as possible so that you can go back and always find what you are looking for. Possibly forever. For so long as there is technology to discuss and a place to reach us. And non-bankruptcy!
I hope this explains most of how everything works. It has all been set up to actually provide a unique experience. Like I wrote before, we could remove all or half the features, and probably not need all of this extra stuff, but people actually use the features. People like gaining experience points, using albums, using a live search, having a sidebar, seeing a map of the world with dots of people on it, being able to see in real-time what people are posting, and so forth. Providing tutorials, videos, blogs, and software solutions are ways to keep the site interesting, instead of it just being another forum site.
But the content and people who are on the site will always have the most important part, whether they realize it or not. A site like this doesn't operate if no one posts anything. Do you realize that with every post you make, you are potentially adding a page to the website?