PaidVerts

"The website is as good as dead at the moment... I strongly suggest coming back in 2-3hours, give the programmers time to work.... And seeing as I can't post the news in the forum; I'll post it here for the moment;

Improving server performance and general website availability is our top priority right now. As I'm pretty sure that our physical ability to permit more users on the website is the only thing holding back our growth.

I think PV can double its active users within the next month, if the website will just work, and be nice and speedy in the process.

We've talked to incapsula about our TCP setting, to try and reduce all these Error 20 messages that you keep seeing. And we've asked them to check things on their end, as the delay via going through their proxies is usually 0.2seconds, but during the middle of the day it seems to jump up to 2seconds. So we're trying to find out whats going on there.

We're also doing a bunch of things behind the scenes regarding caching certain data, images and all the rest of it to generally try and speed things up / reduce the server load.

Today we should be upgrading to the latest version of MySQL, and also switching something over to NGINX. Both of which should significantly improve our server performance... We actually began this last night, but our database is so huge, it took forever to properly backup, and migrate over to the dev server for testing (currently its over 25GB).

We're also hunting down other random issues; like strange quantites of active connections to our server from two IP addresses (one of which being our database server).... Something is not right there; but we need to figure out what exactly. Some bug / some process is doing something wrong, and hanging up the server.

There's quite a lot going on, which you'll likely never see, beyond whether or not the website loads / is fast or not! But our traffic might not actually be source of the problem; as our server loads seem fairly consistent throughout the day. There's some process that's screwing things up we think.

Todays PV ad issue is in motion; I'll post the numbers later once that's completed.

My PM bulk pay is also installed, so I've cleared all but a dozen cashouts there. Waiting on more PM funds to sort the rest out. (Those users can either wait, or request to be paid via another method; up to them -- but non are over 7days yet, so no biggy)

And I'm still chipping away at PayPal cashouts as fast as the released funds permit.

We've also added a "Latest News" page to PaidVerts. Where we can post random updates, and try and direct people to the MTV daily forum news for the PV numbers etc.

In addition to the pending account history rebuild, I've also added a low priority task to create a "history balance graph" -- which will show the range of your balance each day (min-max) and plot that over however long... Then as you select a range in that graph, it'll display all the history entries for that time period.
This feature is meant to help the admin check pending cashouts; as we can quickly see where your balance spiked; and jump straight to that part of your account history. But we'll throw it out to the user side also.
* And later, we'll build in a daily summary thing, which will compile your daily history into a bunch of statisics for easy viewing. Show what incomes came from where, and where you spent / invested / cashed funds. All in quick stats; save you having to crawl through thousands of referral commission entries - it'll just summarize them all.
Jo "

From the programmer:
So, everything’s down because of mysql server. The reason is no space left on the main partition.

We’re moving DB backups to another, 900GB empty partition.

-----
We have 3 severs already. And today's ad issue hasn't really happened yet. Only for the bottom 3 bap groups.

I don't know how long the maintenance will take. Hopefully sometime this evening we'll be back up.

But 40million hits a day is challenging! Especially with a site as complex as pv/mtv.

Todays news is a work in progress, I’m not 100% sure where we're at. Or what's happening today, or whether there'll be anymore significant downtime.

We have definitely upgraded to the latest version of MySQL. And we've also moved something or other to a new partition, as the reason everything went to hell last night was we basically ran out of space on our MySQL server. Or something like that.

Today we are installing NGINX in some configuration or another (again, I don't have a clue what I’m talking about, just re-hashing bits of emails I’m looking in on as the programmers communicate to one another) . And on the back of that, the programmers will need to re-write all the crons and make some optimization stuff to utilize the latest version of MySQL to our full advantage.

So right now, I’m in wait and see mode! I don't really know what the plan is for today just yet. Once I catch up, we'll get everything rolling!

What a messy couple of weeks. Non stop chain of problems, all of which seem to involve downtime. Weird; as we had 2+ years online prior to this without any major downtime events... But all this stuff needed doing; so it's good to get it out the way I guess. And now isn't a bad time, as our funding is pretty healthy. And things are generally going pretty well; we're just working to solidify our foundations to properly facilitate enormous growth, which should be imminent... But this is proving a bit tricky / lot of work!
------------
This is part of the roadmap towards the inevitable scaled hosting solution we'll require, with multiple servers, replicated databases and load balancing.

1. Latest version of MySQL
The programmers wanted this when we first launched PV; but Alex resisted, as it might be a pain with all the existing MTV code which was built around an older version of MySQL.

2. NGINX
No idea what this is, but apparently its better / faster / more scalable.

3. And the final step will be to move our database to NoSQL. As that should be easier to work with as we get bigger. And then we can upgrade the hosting, and the programmers can deal with a whole new set of problems based around replication  

But yea... we're starting to get bigger! And running into big issues. But this is good; with scale will come a lot more money. And we can do bigger and bigger things.

But as a company, we're really going to start to grow in the next 12months I think. We'll need to hire a lot more people in a wide range of areas. Then we can get groups of people working on their own division of MTV; and building our products. And then we'll really start to grow our empire!

Tomorrow there will be a big ad issue.
Jo

Our server upgrades are about done for the moment, we've got the latest version of MySQL installed, along with NGINX. The programmers are still in the process of re-writing all our crons to maximise their performance. But so far, the site seems to be holding up.

We're also chasing up iweb, who appear to have short changed us on a hard drive... Giving us half the capacity we should have; and when making multiple backups of a 25gb+ database; we need the space!

Todays PV numbers:
$361,000+ ad packs sold to date!

BAP Group …........ Percent (x100) …........ Dollar Value …......... Participants …......... Ads Per User

20 to 1599 …............... 0 …............... 0 …............... 12796 …............... 0
1600 to 6400 …............... 0.065 …............... 845 …............... 5992 …............... 0.1410213618
6400 to 12000 …............... 0.03 …............... 390 …............... 1223 …............... 0.3188879804
12000 to 24000 …............... 0.045 …............... 585 …............... 1089 …............... 0.5371900826
24k to 48k …............... 0.07 …............... 910 …............... 880 …............... 1.0340909091
48k to 96k …............... 0.085 …............... 1105 …............... 531 …............... 2.0809792844
96k to 180k …............... 0.085 …............... 1105 …............... 305 …............... 3.6229508197
180k to 360k …............... 0.08 …............... 1040 …............... 229 …............... 4.5414847162
360k to 720k …............... 0.065 …............... 845 …............... 124 …............... 6.814516129
720k to 1.5m …............... 0.08 …............... 1040 …............... 67 …............... 15.5223880597
1.5m to 3m …............... 0.13 …............... 1690 …............... 58 …............... 29.1379310345
3m to 6m …............... 0.1 …............... 1300 …............... 27 …............... 48.1481481481
6m to 20m …............... 0.08 …............... 1040 …............... 12 …............... 86.6666666667
20m + …............... 0.085 …............... 1105 …............... 2 …............... 552.5

* The 20m+ group got a nice boost, as one of the 3 users in that group purchased vacation mode! So that's bumped your earnings up from $350 or so, to over 550!

Anyway, let's hope for a bit of stability on the hosting side of things for a while. As i'm pretty sure that's all we need to double our active PV users. Things are going very well there, if we can just keep the site online - that should grow a whole lot in the next few weeks.

Other news: Cashouts are rolling as best we can, PayPal is nearly caught up. Programmers have spent the entire week on server issues; not managed to deploy any new features in a while. But we'll get to that! SSL is still in progress. And i'm still playing catch up on the support tickets. Same old!

Jo