Forums

Is the service offline at the moment?

Not working for me either. Doubtless PA staff are busy working on it but an update on progress either here or on Twitter would be helpful.

It's working for me

Even the blog is down: http://blog.pythonanywhere.com/

New news? My site is still not working.

Some parts are working (eg PA home page and also my consoles) but the PA blog and my website fail with the OperationalError at / (1105, '(proxy) all backends are down') message.

my site is down as well.

All seem to be working again now.

sigh still down...

I still have the issue. Guys, could you please fix your service, this is really a blocking one.

I seem to be getting this one too.

Am finally back online...

Apologies for not posting here earlier. At 6:06am UTC there was an outage on one of the Amazon database instances we depend on. It lasted for about 40 minutes. We're investigating what caused this and will escalate to Amazon if necessary.

Normally our system can recover automatically from situations like this, so we would have had a 40-minute outage but then things would have been OK. However, in this case the problem triggered a number of other failures that cascaded through the system, breaking a number of customers' web apps, and it required significant manual intervention to put things right. Servers started coming up again at around 11am, and by 12:30pm everything was back up and running again.

Everything should be OK now -- let us know if you're still seeing problems -- and we're trying to work out what caused the original issue, and also what triggered the error cascade that turned this from a 40-minute outage to what was for some people over 6 hours. We'll post more when we know more.

Hi Giles,

Service seems to be problematic again: Unable to access mysql from my app, am able to connect from console. Have tried to reload but am getting the same error. OperationalError: (2003, "Can't connect to MySQL server on 'mysql.server' (111)")

I have the same problem

Looks like we have rolling outages. I just had trouble loading my Dashboard...:-(

Morning folks. Looks like the same issue as yesterday's outage. We were able to get to it sooner though, and do the required restarts. We've narrowed down the probable cause too, it's to do with our daily backup job on one of the RDS instances.

Things should be back to normal now. Can you confirm?

Or at least, back to normal for most users. Will keep digging.

OK, that should be better now?

@jacksonville, @befa, @a2j -- I've checked your web apps and they all seem to be functioning normally. Let us know if there are any other problems.

Works now, thx

Excellent, thanks for confirming.

well, it is not working for me... I have that error 1105.... Anything going on?

All the servers' connections to MySQL are now fine. Try restarting your web app? Or if you're having problems from a console, try closing it and starting again with a new one?

(1105, '(proxy) all backends are down')

hi, im getting the following error and in the logs it shows Exception Location: /home/byrenx/.virtualenvs/django16/lib/python2.7/site-packages/MySQLdb/connections.py in init, line 193 Ive also tried mysqldump and it doesnt seem to work as well, and also tried reloading my web app and restarting my console but the same error shown. can you help me out?

We're getting the error message as well on all 3 of our webapps, and it is persisting after restarting the app.

even if try to restart my app its still not working. I am paying much attention to this error line.

Exception Location: /home/byrenx/.virtualenvs/django16/lib/python2.7/site-packages/MySQLdb/connections.py in __init__, line 193

I think there might be an error to database connection. I think the msql server perhaps

Im also having the same error (1105, '(proxy) all backends are down') After restarting web app, still the same problem.

My app is down for this error too. 3 for 3 the last three days being down most of the day for some issue

They've all been in the PM EST hours too... so they don't get fixed until morning english time. PythonAnywhere needs to get an on call dude at these hours. Hell, I'm at the datacenter at 3:00 AM for my job sometimes. It's business, and this sucks.

yeah there have been a bad string of days :( not good

My web app does not work last 12 hours. Restarting web app not help. I have an error "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in init super(Connection, self).init(args, *kwargs2) OperationalError: (1105, '(proxy) all backends are down' MySql database worked fine.

Same still over here, but somewhat confused. Consoles work, Web2Py backend works, but still after App reload, front-end does not show with "(1105, '(proxy) all backends are down')"

And Frontend works as well. Don't know what you did, but thanks :)

Hi there -- sorry for the slow response. We have people on call 24/7 but our system is failing in ways that our alerting isn't picking up, so we're not getting paged. I'm investigating. As there seem to be multiple threads about this issue, I'll post updates on this thread.

So, here's an update on where we are with this.

We recently moved data centers, from Amazon's us-east1a "availability zone" to us-east-1c. The latter zone supports new, faster instance types, with SSD storage and much better processors.

Unfortunately, since we moved there, we've had a significantly higher rate of errors connecting to our database instances. The problems tend to impact specific servers -- eg. one of the web servers in our cluster might have a sudden spike in connection errors while the others are unaffected.

Obviously we're chasing Amazon hard to find out what's going wrong. They've so far not found any root cause, but we're going to keep pushing until it's fixed.

In the meantime, we've been working on changes to PythonAnywhere and improved alerting for our support team so that we can mitigate the effect of these problems on our customers. Each outage we've had has been due to a different failure mode of our systems in response to database outages. We've added various kinds of redundancy, and improved our monitoring so that we're alerted to various kinds of new problems. On Thursday we introduced a completely new set of alerts (in addition to the previous ones), which would have alerted us to the problems on Wednesday. Unfortunately last night, the system found a new way to fail which bypassed those alerts.

Naturally we're upgrading the alerts again and putting procedures in place so if this happens again, at the very least we'll all be paged and will be able to log in and fix the issue without lengthy downtime.

So, once again, our apologies for the outages. We're working on it, and hope to have it fixed ASAP. We fully appreciate that whatever the underlying cause, and however far downstream it might be from our operations, it's solely our responsibility to provide a stable service to you.

Getting this error OperationalError: 1105, '(proxy) all backends are down'

Backend and DB work, but frontend results in error 500

Me too

All worked yesterday. Today is an error

Seems to be offline again...

Still offline, same error

Any idea when it will be resolved?

Offline for 8 hours, this is bad :(

One of the web servers decided to stop talking to the database. We believe it's related to an ongoing issue we're having with AWS where some DNS lookups just fail for no reason. We have monitoring for whether the database connection on the web servers is working, but it looks like it doesn't trigger for this failure mode. I'll be spending quite a bit of today beefing up the monitoring and trying to work out how to make the database connections more robust against DNS failure.

We've come up with something that should make things a little more resilient. If you change your database host from mysql.server to yourusername.mysql.pythonanywhere-services.com (eg. for me it would be giles.mysql.pythonanywhere-services.com) then you'll bypass this particular bit of our infrastructure.

This might be more reliable until we get the problem fixed (and might even be a better solution in the long term).

-

Hi Giles,

I changed my db address as suggested and it worked for a bit. I'm now getting the following error: 2014-07-26 09:11:22,226 :OperationalError: (2005, "Unknown MySQL server host 'jacksonville.mysql.pythonanywhere-services.com' (111)") Thanks,

Just so everyone knows, @jacksonville helped us get this sorted out and it fixed similar issues for other users.

Hey,

I am facing this issue for my site. Can someone help?

Thanks, Chirag

I have the same issue, since about 15 minutes ago.

EDIT: I managed to connect with my DB by setting the host value to: yourusername.mysql.pythonanywhere-services.com

I have a couple of apps running on PA but only one of them has this issue in the past 15 minutes or so.

getting same issue

doing the yourusername.mysql.pythonanywhere-services.com fixed it for me.

doing the yourusername.mysql.pythonanywhere-services.com fixed it for me.

Looks like it was the same problem -- really sorry about that, we thought Amazon had fixed the underlying problem, but obviously it's not quite as thorough a fix as we thought. It should be sorted now -- I've rebooted the service that was broken -- but going forward it looks like the best workaround is to bypass the system that's affected by this error by using the direct yourusername.mysql.pythonanywhere-services.com. TBH now that we've beta-tested that DB addressing scheme thoroughly, it's probably ready to be the official way of doing things anyway (it's actually slightly more efficient). So we'll change the text that tells you which hostname to use, and we'll get an email out to everyone suggesting they change their code, so hopefully this won't happen in the future.

Thats good to know. Thanks Giles.

Hi Giles, I am facing the same issue in one of my apps

One of the services on the web server that your app was on had some trouble. We've fixed it now so your app should be working.

Same problem again (https://www.pythonanywhere.com/forums/topic/979/#id_post_15288)

It was the same problem again. Are you using the yourusername.mysql.pythonanywhere-services.com pattern for your MySQL server connection string? It should be proof against this kind of problem (and our next release will make it the official way to connect).

I wasn't using the "yourusername.mysql.pythonanywhere-services.com" pattern for MySQL server connection string. I was using 'mysql.server'. I changed it and it's working properly right now. Thanks!

:) we'll be removing the proxy and switching it out for something more reliable soon....

After our system update today, the username-based hostname is now the official way to connect to your MySQL database. The old mysql.server system still works, but is now deprecated.