PaidVerts

joaomoney1 escreveu:Pagamento PayPal correu como habitual
Pedi ontem à noite 11/30/2014 22:33
Chegou hoje 12/01/2014 14:52

We've turned off the ad view page for the moment, as we have too many users for our current server configuration. And if you're all viewing ads, it's killing the website... We're working on the necessary to scale our servers & coding to cope with our huge growth of late, and as soon as that's ready, we'll put the ad view process back online.

* We have extended all active ads by 48hours (and we can extend further if necessary) , so you won't lose any ads as a result of this maintenance. The ads will be waiting for you when we get everything fully functional (and speedy!) again.

----

I'll copy a few programmer messages to show you where we're at:

Daniel K. a day ago
- reff daily stats to redis - frequent lock on stats table
- partitioning issues table
- investigated some other things which should be moved to redis, to be continued tomorrow

Daniel K. about 14 hours ago
- preparation for DB repliacation
- audit results analisys
- moving ad interaction to Redis

Alex (last night)
Ok quick update before I head out to bed.
I have successfully set up 2 Mysql 5.6 databases replicating. Simple replication works fine.
The next test is to create a quick CodeIgniter testbench that will allow writes on DB1 and reads on DB2. Then stress test it all to see if it can cope with load.

Good night all

Alex... This morning:
Good morning,

The replication has been successful and I am now in a position where I write on DB1 (master) which then replicates on DB2 (slave), then I can read back from DB2.
That way we can theorically offload the big page generation like portfolio onto a different server, keeping the main one free. Backups will also be able to move onto a slave.
Now there are several other things I need to try, including a replication failure - how can we recover from it...

I'll keep you updated, but I think we should order 1 new DB server to be slave. What do you guys think?

Daniel
That's a very good news

In the worst case, we can use this old server which is still connected. But, I also prefer to have a faster one. With more cores for operations and a bigger disk of course.

So, maybe we should simply order a new DB server, with configuration copied from the current one ?

Alex
That was my thinking yes. We keep using the main server as we are, change it to be a master server and add a new server which will be the slave. Ideally we need the same server in terms of hardware and software.

In the long term, it may be possible to have more than 1 slave and use some kind of sharding in order to query the right database. For example, let's consider we have 1 Master and 3 Slaves.
Everything gets written on the Master which is as we are now, but we want to perform the hardest reads on the slaves by sharding the connections. We could use the user ID modulo number of slave servers... Meaning that User ID 1 will get its data from Slave 1, User ID 2 from Slave 2, User ID 3 from Slave 3, User ID 4 from Slave 1, User ID 5 from Slave 2, etc...
And if we need to add more servers, then we change the modulo divider - simplez

...

To which I have just replied requesting the programmers place the order with iweb for the new equipment. And i've insisted that they buy not one, but 2 additional dedicated servers, so that we have some spare capacity to tap into as of when.
* As i'm certain that if for nothing else, we can make use of the 3rd database server as we build multiplayer poker... That's definitely going to require extra capacity in one form or another; so I don't mind spending a few hundred bucks a month to have a spare server lying around, connected & ready for instant utilization as the programmers require it.

....

I've also told the programmers to turn the ad view process back on as soon as they think the site can handle it... But it's still offline for the moment, so I suppose there is more work to be done.

But Alex seems to be actively helping us right now; so that's great news. And I would imagine we'll get a very speedy solution to our problems - provided iweb doesn't create a delay installing new equipment.

------

And just to reassure people...
Our problems are good problems, we simply have too many users viewing ads for our current server architecture and coding. But that's a straight forward fix, just takes a few days to deploy.

The system is completely liquid.
There are $464k in user account balances (that can potentially be withdrawn)... but last time I checked we were sitting on almost $900k in cash assets. There's $440k in PayPal alone.

So this is no ponzi that's lost its momentum, and is going to bankrupt as a few people panic & cashout... We've simply incurred a small problem, like hundreds before, these last 3 years... For which we'll get it sorted, and be back to business as usual before you know it.

We just require a bit of patience... the website is still online, you can cashout your balance at any point. We're good... there's just some work to be done, like always!

Jo

To repeat, don't worry about your ads... When the ad view process is opened, investigate your account THEN, and you'll see everything will be fine. Right now, it's disabled and the programmers are working. So ignore everything.

----

From Lukas

We have received database audit yesterday, it's 48 pages in pdf, about different problems on our database server.
First batch of changes was problem in our config, I have changed it, so should be a bit better now.
Second one was missing indexes in few tables, Paul took care about it. E.g we had problem in our pv purchases table, so direct ad purchases should be a bit more reliable now.
Third one, and the biggest was problem with issuing ads.. obiously. Database was getting locked and in the meantime other quries were blocked too.
Likely we'll move whole process to Redis, but it's in Paul/Daniel hands now.

----

Also, we've been contacted by someone who works for iweb offering their help, in terms of a sys admin // sorting out our server configs etc to better scale the system... We'll investigate that and see what he has to offer.

I don't have an ETA for resuming ad views, if we can get the new DB servers installed early tomorrow; then perhaps as early as tomorrow evening they maybe operational. But that might be wishful thinking; likely Wednesday.

But I'm not actually sure whether we're 100% waiting on new servers + replication to resume things. We might be able to turn the ad view process back on before that.

Either way... a day or two and we'll be back to business as usual!

The programmers & Yan are having a real complex conversation about how to setup the 2 new database servers (3 in total), and the replication in general. Plus what's required to adapt our current code to all sorts of complex terminology.

Anyway, they're working on that right now. Carlos has signed & paid for the two new servers and I believe Yan is trying to expedite the installation process.

Then we're going to require up to 7-12 hours of complete downtime - as we copy a 200GB+ database to our two new database servers... And then there'll likely be potentially a few more hours of testing the new system with the database running across multiple servers, a new load balancing software and all sorts.

This really is a huge change behind the scenes, but the users won't really know any different. Other than hopefully the website will maintain its current speed at scale.

Let's see how the day progresses... but if I were you, i'd assume nothing is going to happen today. And come back tomorrow. I'll make sure all ads have at least 24hours recycle time on them whenever we come to re-open the ad view process. So you won't miss any.

But fingers crossed! This is a huge evolution in our 'foundations' and it'll allow us to grow to epic proportions in the New Year.

I've not been contacted by any programmers, other than yan for a server admin position.

Lukas has set a feature to automatically lock news posts after 7 days, to stop people digging up last months posts.

And right now, we're waiting 2-3hours for the servers to be connected. And then we'll commence the downtime I think.. And fingers crossed, overnight we'll be back in business.

* And what are the odds, that we get such a flood of traffic when the ads reopen, that the site crawls again, regardless of 3 DB servers lol.

https://www.paidverts.com/member/latest_news.html - I've added a task to the programming queue, to enable shareholders with more than 100k shares, to update the PV news page from the shareholder control panel. (As I always forget this page)

---

And SloSumo has submitted an idea to improve the support tickets -- via a more interactive form. That asks you a bunch of questions about your issue, before you submit it, to collect as much data as possible -- and even answer you automatically in some cases.

We'll let him refine that idea, and deploy that ASAP.

----

And Paul says the Polish translation is ready. We'll deploy that after the new servers are up and running.

And Neteller should be ready soon. Our account is ready / fully merchant verified ; just trying to get them to resend the login info & Paul will install it.

Sorry! Site is currently under maintenance

Maintenance began at 5:50 GMT. We're currently replicating our databases to facilitate our massive growth. This will take 7-14 hours at least. You will not lose any ads during this process. Check our facebook page for additional updates.

Update from Paul:
"I've configured both servers, installed mysql and imported dbs"

So the database transfer was a lot quicker than expected. Now Yan is talking about adjusting the balancing pool to only point to the slaves (the two new databases that replicate the data for read only) - to take the burden off the master database that handles the writing.

I don't have an ETA for you just yet... but so far so good!