PaidVerts

Fgold escreveu:Os problemas técnicos já vão para uma semana e o Natal aproxima-se.... Por isso há que acrditar na chegada do...

Here's part of an email from Lucas, just FYI:

We have already implemented simple function to read from slaves. Also adaptaded bunch of code.
Actually most of MTV reads runs on slaves servers now (forum, tickets, portfolio, results, game stats, gameplay summaries) and few other.
Also a lot of PV stats.
We do only very few reads from master (e.g deposits, as they need to be fast, and replication sometimes is delayed due big load).

But as I said, 95% of slow queries (reads) is disturbing our new servers (haproxy balances between 2 slaves), so that's good.

===========================

The main problem why sites were slow was because database locks. Some query could block whole ads table for e.g 2 seconds, then it was adding up and done, we were all stuck.

So don't expect miracles just after start, should be steady enough, but we'll work on it few more days to minimize all possible locks. We'll see how it goes, lags should be much less, just don't expect performance tripled (3 database servers).

===========================

The ad view process will resume ASAP. Anytime from now on is good, as its well past peak traffic hours... And the first issue will almost certainly be tomorrow."

Everything is back online now… likely still a few loose ends to tie up. But we’ll get to those as we find them.

– All ads were extended, so you shouldn’t have lost any of those.

– And Paul has extended the super user upgrades to compensate you for the downtime.

– There is however an issue with some accounts, as our recycle cron was working for 1 hour during the downtime, so a few people will have lost a few ads. And we can’t get those back. So please submit a support ticket using the “Other” section, only if you have lost more than 1000 BAP… And SloSumo will refund the BAP in the next few days.

Paul has mentioned we may need additional www servers, as the load on those last night was super high. We’ll look into that ASAP, as they’re really easy to add.

I am aware of the captcha difficulty issues. We’re working on a solution for that at the moment. I’ve asked the programmers to look into Googles NoCaptcha … and we’ll likely run that alongside SolveMedia — issuing half the ads with one, and half the ads with the other… That will help resolve the issue in two different ways; i’ll explain more as we deploy it.

I am issuing a low value ad issue today… just to get the ball rolling again, and test everything out. The next ad issue (either tomorrow or Sunday) will be twice the size of todays issue. And then from Monday onwards we’ll be 100% back to normal with full ad issues.

The programmers have edited the members main page, to remove the latest game wins; as that is a problematic query that was slowing the members page down a whole bunch. So right now it only shows forum posts – and is a lot faster.

The programmers are keeping an eye on our WWW servers and LoadBalancer bandwidth; we’ll expand those as soon as they look like they’re nearing capacity… The good part about Yan, is he is actually checking this stuff all the time now.

—

Still chasing up Neteller… the merchant account was fully approved in mid November. But we’re struggling to get hold of the login info … As soon as we get that, Paul will install it.

Credit Cards and everything else, is a project for another day. Carlos should also have some additional bank accounts setup for us to use soon… And we’re going to have to introduce some basic ID requirements for bank deposits/cashouts – as the banks are being a real pain of late. It’s almost like they’re scared of money… This anti moneylaundering regulations and what not, are utterly ridiculous now. They’ve completely paralyzed the banks.

BitCoin really needs to grow! As the old system has drowned itself in paperwork and doesn’t function well anymore.

Share Split will happen when it happens… likely at least a week away.

Anyway… super busy! But we’re back on track, and things are looking good. Next year will be huge for us I’m sure.
Jo

Yea our traffic is extreme…
95th percentile for Bits per second is 130mbps.

I think we’re going to have to get started right away on our datacenter relocation. Apparently Iweb has a new datacenter that we should move to; as they permit 10+GBPS ports.

I wonder if we can set that up to perform not this Sunday, but next Sunday… It’ll involve another half day of downtime; but it should be more controlled this time as we dont have to adjust the code.

* And as a temporary measure, we’re going to order a few more WWW servers.

But yea, we still need a lot more… we just had ~10k users online. But that’s nothing; there are 55k+ active daily users. And 400k+ registrations. If even 10% of those users come online; we’ll be done for!

We’re still working on server scaling stuff… An extract from one of the messages from Yan:

… To counter this we have implement the VPR (Virtual Private Rack), bad news is VPR only available at NE.

So to scale we are force to migrate to NE. Now the migration i would need to talk with the vendor and see how we can handle things.

What id recommend but let me talk first with vendor is the follow:

We order 8 servers at NE in a vpr, all plain os.

We then initiate the website migration and db. we test everything and if we are set, we move the dns to new ip, cause we cannot transfer ips.

By the end of next week we close everything at CL, get credited for new servers ( 30days money back ).

The solution Yan is talking about there will give us options to scale a lot better. As currently we have an issue with ‘ Direct Routing ‘ that can’t really be fixed in our current format.

Anyway.. we’re working on a Solution, and i’ve asked them to aim for any downtime to take place next weekend… We’ll likely need a few hours to copy the database and what not; then it’ll be as fast as your DNS resolves to the new servers…. I wonder if we can do something about that to make them resolve more quickly – perhaps increase the refresh rate?

We're placing an order with Iweb for 11 servers (and to move datacenters in the process) to a virtual private rack with a 1GBPS port scalable to 20GBPS+

400,000+ user accounts now, 55,000+ active & viewing ads every single day... Traffic is peaking at over 10,000+ users online at the same time. And we're selling approximately $500k in PTC advertising per week. Getting serious now

This will take about a week to setup, then next weekend we'll have a bit of downtime as we switch over. That should give us everything we need to scale for the near future.