502 error - uWSGI process 61 got Segmentation Fault : Forums : PythonAnywhere

502 error - uWSGI process 61 got Segmentation Fault

Hi II/m unable to debug this issue:

Sometimes my request receives 502 and in server log shows up this:

2019-11-16 10:56:14 !!! uWSGI process 61 got Segmentation Fault !!!
2019-11-16 10:56:14 *** backtrace of 61 ***#012kaifis www.webapp.com uWSGI worker 2(uwsgi_backtrace+0x2c) [0x46529c]#012kaifis www.erbapp.com uWSGI worker 2(uwsgi_segfault+0x21) [0x465661]#012/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f52bd0624b0]#012/usr/lib/libpython3.7m.so.1.0(+0xe7545) [0x7f52b9b49545]#012/usr/lib/libpython3.7m.so.1.0(+0xe7962) [0x7f52b9b49962]#012/usr/lib/libpython3.7m.so.1.0(PyList_New+0x4f) [0x7f52b9b1fd0f]#012/usr/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4703) [0x7f52b9ad0183]#012/usr/lib/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0xa64) [0x7f52b9be04e4]#012/usr/lib/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0xa6) [0x7f52b9afb966]#012/usr/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7f16) [0x7f52b9ad3996]#012/usr/lib/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0xa64) [0x7f52b9be04e4]#012/usr/lib/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0xa6) [0x7f52b9afb966]#012/usr/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x75ec) [0x7f
2019-11-16 10:56:15 DAMN ! worker 2 (pid: 61) died :( trying respawn ...
2019-11-16 10:56:15 Respawned uWSGI worker 2 (new pid: 67)
2019-11-16 10:56:15 spawned 2 offload threads for uWSGI worker 2

I have created a web app with the same code and it works fine. Do you know where could be the problem? Thanks for your time.

(U haven't found any pattern in those 502, sometimes almost all request received 502 and 10 minutes later only about half)

deleted-user-2193179 | 15 posts | Nov. 16, 2019, 11:07 a.m. | permalink

I see that you have 2 web apps are using different virtualenvs, so I'm guessing that one of the virtualenvs is broken in some way try deleting and re-creating the virtualenv for the web app that does not work.

glenn | 9718 posts | PythonAnywhere staff | Nov. 16, 2019, 12:41 p.m. | permalink

I have re-created the broken one and it is still broken. Then I have re-created the working one and it is still working.

Also, those errors can be reproduced on Django's official administration site. When I'm refreshing the admin site I would get approximately every third refresh Pythonanywhere's "something went wrong" site. with Error code: 502-backend

deleted-user-2193179 | 15 posts | Nov. 16, 2019, 2:02 p.m. | permalink

The virtualenv for the web app that is showing that behaviour has not been changed since November.

glenn | 9718 posts | PythonAnywhere staff | Nov. 16, 2019, 4:43 p.m. | permalink

I have removed the whole environment and created it again.

I'm not sure what does behaviour has not been changed mean. But I have re-created everything in that environment yesterday. With

rmvirtualenv pcht
mkvirtualenv pcht --python=$(which python3.7)
pip install -r requirements.txt

And all the files in virtualenv look newly created:

09:21 ~ $ ls -lL .virtualenvs/pcht/bin
total 176
-rw-rw-r-- 1 kaifis registered_users  2089 Nov 16 13:36 activate
-rw-rw-r-- 1 kaifis registered_users  1031 Nov 16 13:36 activate.csh
-rw-rw-r-- 1 kaifis registered_users  2185 Nov 16 13:36 activate.fish
-rw-rw-r-- 1 kaifis registered_users  1137 Nov 16 13:36 activate_this.py
-rwxrwxr-x 1 kaifis registered_users   248 Nov 16 13:38 black
-rwxrwxr-x 1 kaifis registered_users   249 Nov 16 13:38 blackd
-rwxrwxr-x 1 kaifis registered_users   249 Nov 16 13:38 chardetect
-rwxrwxr-x 1 kaifis registered_users   243 Nov 16 13:38 coverage
-rwxrwxr-x 1 kaifis registered_users   243 Nov 16 13:38 coverage3
-rwxrwxr-x 1 kaifis registered_users   243 Nov 16 13:38 coverage-3.7
-rwxrwxr-x 1 kaifis registered_users   291 Nov 16 13:39 django-admin
-rwxrwxr-x 1 kaifis registered_users   153 Nov 16 13:39 django-admin.py
-rwxrwxr-x 1 kaifis registered_users   262 Nov 16 13:36 easy_install
-rwxrwxr-x 1 kaifis registered_users   262 Nov 16 13:36 easy_install-3.7
-rwxrwxr-x 1 kaifis registered_users   247 Nov 16 13:40 epylint
-rwxr-xr-x 1 kaifis registered_users   150 Nov 16 13:36 get_env_details
-rwxrwxr-x 1 kaifis registered_users   259 Nov 16 13:40 iptest
-rwxrwxr-x 1 kaifis registered_users   259 Nov 16 13:40 iptest3
-rwxrwxr-x 1 kaifis registered_users   252 Nov 16 13:40 ipython
-rwxrwxr-x 1 kaifis registered_users   252 Nov 16 13:40 ipython3
-rwxrwxr-x 1 kaifis registered_users   237 Nov 16 13:40 isort
-rwxrwxr-x 1 kaifis registered_users   249 Nov 16 13:36 pip
-rwxrwxr-x 1 kaifis registered_users   249 Nov 16 13:36 pip3
-rwxrwxr-x 1 kaifis registered_users   249 Nov 16 13:36 pip3.7
-rw-r--r-- 1 kaifis registered_users    72 Nov 16 13:36 postactivate
-rw-r--r-- 1 kaifis registered_users    74 Nov 16 13:36 postdeactivate
-rwxr-xr-x 1 kaifis registered_users    69 Nov 16 13:36 preactivate
-rw-r--r-- 1 kaifis registered_users    75 Nov 16 13:36 predeactivate
drwxrwxr-x 2 kaifis registered_users  4096 Nov 16 13:39 __pycache__
-rwxrwxr-x 1 kaifis registered_users   243 Nov 16 13:40 pygmentize
-rwxrwxr-x 1 kaifis registered_users   245 Nov 16 13:40 pylint
-rwxrwxr-x 1 kaifis registered_users   251 Nov 16 13:40 pyreverse
-rwxrwxr-x 1 kaifis registered_users 11624 Nov 16 13:36 python
-rwxrwxr-x 1 kaifis registered_users 11624 Nov 16 13:36 python3
-rwxrwxr-x 1 kaifis registered_users 11624 Nov 16 13:36 python3.7
-rwxrwxr-x 1 kaifis registered_users  2348 Nov 16 13:36 python-config
-rwxrwxr-x 1 kaifis registered_users   247 Nov 16 13:40 symilar
-rwxrwxr-x 1 kaifis registered_users   240 Nov 16 13:36 wheel

PS: They haven't changed since last November? That is weird because I created this webpp with that environment like 3 months ago.

deleted-user-2193179 | 15 posts | Nov. 17, 2019, 9:39 a.m. | permalink

I'm sorry. I don't know what happened in my brain when I looked at the date. I think I just couldn't believe that we're already in November, so I thought "That must be long ago" and not "That was today"

The only thing that I can see that may be influencing it is that sentry is complaining about the transport you're using being incompatible with our environment for web apps. Perhaps try disabling sentry and seeing whether anything changes.

glenn | 9718 posts | PythonAnywhere staff | Nov. 17, 2019, 1:01 p.m. | permalink

Nevermind ;).

I have enabled sentry after I started experiencing those issues. Hoping it will give me something more helpfull than OS Error.

deleted-user-2193179 | 15 posts | Nov. 17, 2019, 4:11 p.m. | permalink

Hi it's the issue is still there and is affecting our production instance. The sentry was setup after this issue started and the virtualenv is freshly installed.

There could be a prohlem with the database but this issue happens randomly - so it's weird data issue or something else.

I have also observed that when I go on the djangos admin page and load user detail. I get 502 and second time it displays the detail.

deleted-user-2193179 | 15 posts | Nov. 20, 2019, 8:09 a.m. | permalink

Since you have a working version and a non-working version, it must be something that is different between them. I would suggest that you try to find which of the differences is causing the issue and then we can hopefully work out a solution from there.

glenn | 9718 posts | PythonAnywhere staff | Nov. 20, 2019, 1:01 p.m. | permalink

It seems that the issue was caused by a bug in MySQL connector. I switched to mysqlclient and everything is fine.

deleted-user-2193179 | 15 posts | Nov. 20, 2019, 5:03 p.m. | permalink

Interesting! Glad you worked it out. I think the mysqlclient is probably the best option now in general -- MySQL Connector was useful back when there was no other Python 3-compatible MySQL library, but that hasn't been the case for a few years now.

giles | 12096 posts | PythonAnywhere staff | Nov. 20, 2019, 5:07 p.m. | permalink