Forums

Hello,

i run a web app with pandas and tensorflow and i see that when building a sequential model with keras i get my website crashed. on the error log i do not see any errors and on the server logs i get:

2018-01-29 22:36:47 W
tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow
library wasn't compiled to use SSE3 instructions, but these are
available on your machine and could speed up CPU computations.
2018-01-29 22:36:47 W
tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow
library wasn't compiled to use SSE4.1 instructions, but these are
available on your machine and could speed up CPU computations.
2018-01-29 22:36:47 W
tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow
library wasn't compiled to use SSE4.2 instructions, but these are
available on your machine and could speed up CPU computations.
2018-01-29 22:36:47 W
tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow
library wasn't compiled to use AVX instructions, but these are
available on your machine and could speed up CPU computations.
2018-01-29 22:36:47 WSGI app 0 (mountpoint='') ready in 18 seconds on interpreter 0x1eb55e0 pid: 6368 (default app) 2018-01-29 22:36:47 *** uWSGI is running in multiple interpreter mode *** 2018-01-29 22:36:47 gracefully (RE)spawned uWSGI master process (pid: 6368) 2018-01-29
22:36:47 fork(): Resource temporarily unavailable
[core/master_utils.c line 729] 2018-01-29 22:36:47 metrics collector
thread started 2018-01-29 22:36:47 pthread_create(): Resource
temporarily unavailable [core/utils.c line 3816] 2018-01-29 22:36:47
!!! unable to spawn stats pusher thread !!! 2018-01-29 22:36:47
chdir(): No such file or directory [core/uwsgi.c line 1590]
2018-01-29 22:36:47 VACUUM: unix socket
/var/sockets/dkaraflos2.pythonanywhere.com/socket removed.

is this related with any memory issues? i cannot figure it out

Thanks in Advance!!

that is because we limit user processes to <128. Is your tensorflow library trying to create more than 128 subprocesses? Also note that we do not allow threading. Finally- if your webapp takes more than 5min to respond, that response is probably going to terminated/cut off (and so the whole thing wouldn't work anyways).

Dear Conrad,

i run a Neural network using keras with tensorflow as backend. I do not know how to figure out how many processes tries the tensorflow library to create. my bigger sapshot of my server log is the following (as i can see, the number of processes is 15)

018-01-30 08:13:46 fork(): Resource temporarily unavailable [core/master_utils.c line 729] 2018-01-30 08:13:46 metrics collector thread started 2018-01-30 08:13:46 pthread_create(): Resource temporarily unavailable [core/utils.c line 3816] 2018-01-30 08:13:46 !!! unable to spawn stats pusher thread !!! 2018-01-30 08:13:46 chdir(): No such file or directory [core/uwsgi.c line 1590] 2018-01-30 08:13:46 VACUUM: unix socket /var/sockets/dkaraflos2.pythonanywhere.com/socket removed. 2018-01-30 08:16:09 *** Starting uWSGI 2.0.14 (64bit) on [Tue Jan 30 08:15:34 2018] *** 2018-01-30 08:16:09 compiled with version: 5.4.0 20160609 on 22 November 2017 21:24:09 2018-01-30 08:16:09 os: Linux-4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 2018-01-30 08:16:09 nodename: giles-liveweb1 2018-01-30 08:16:09 machine: x86_64 2018-01-30 08:16:09 clock source: unix 2018-01-30 08:16:09 pcre jit disabled 2018-01-30 08:16:09 detected number of CPU cores: 2 2018-01-30 08:16:09 current working directory: (unreachable)/etc/uwsgi/vassals 2018-01-30 08:16:09 detected binary path: /usr/local/bin/uwsgi 2018-01-30 08:16:09 *** dumping internal routing table *** 2018-01-30 08:16:09 [rule: 0] subject: path_info regexp: .svgz$ action: addheader:Content-Encoding:gzip 2018-01-30 08:16:09 *** end of the internal routing table *** 2018-01-30 08:16:09 chdir() to /home/dkaraflos2/ 2018-01-30 08:16:09 limiting number of processes to 15... 2018-01-30 08:16:09 your processes number limit is 15 2018-01-30 08:16:09 your memory page size is 4096 bytes 2018-01-30 08:16:09 detected max file descriptor number: 123456 2018-01-30 08:16:09 building mime-types dictionary from file /etc/mime.types... 2018-01-30 08:16:09 536 entry found 2018-01-30 08:16:09 lock engine: pthread robust mutexes 2018-01-30 08:16:09 thunder lock: disabled (you can enable it with --thunder-lock) 2018-01-30 08:16:09 uwsgi socket 0 bound to UNIX address /var/sockets/dkaraflos2.pythonanywhere.com/socket fd 3 2018-01-30 08:16:09 Python version: 3.6.0 (default, Jan 13 2017, 00:00:00) [GCC 4.8.4] 2018-01-30 08:16:09 *** Python threads support is disabled. You can enable it with --enable-threads *** 2018-01-30 08:16:09 Python main interpreter initialized at 0x7b75e0 2018-01-30 08:16:09 your server socket listen backlog is limited to 100 connections 2018-01-30 08:16:09 your mercy for graceful operations on workers is 60 seconds 2018-01-30 08:16:09 setting request body buffering size to 65536 bytes 2018-01-30 08:16:09 mapped 333952 bytes (326 KB) for 1 cores 2018-01-30 08:16:09 *** Operational MODE: single process *** 2018-01-30 08:16:09 initialized 35 metrics 2018-01-30 08:16:09 Using TensorFlow backend. 2018-01-30 08:16:09 WSGI app 0 (mountpoint='') ready in 35 seconds on interpreter 0x7b75e0 pid: 19013 (default app) 2018-01-30 08:16:09 *** uWSGI is running in multiple interpreter mode *** 2018-01-30 08:16:09 gracefully (RE)spawned uWSGI master process (pid: 19013) 2018-01-30 08:16:09 spawned uWSGI worker 1 (pid: 19081, cores: 1) 2018-01-30 08:16:09 metrics collector thread started 2018-01-30 08:16:09 spawned 2 offload threads for uWSGI worker 1

For a webapp, you would have to run tensorflow single threaded. For example, see this.

Dear Conrad,

I have the same symptom (). Could you tell me whether my problem has the same root cause as the above one, i.e. Tensorflow must be run single-threaded? If yes, would be so kind as to point me where I can insert the code from the stackoverflow answer you linked to?

The answer lists Tensorflow code: session_conf = tf.ConfigProto( intra_op_parallelism_threads=1, inter_op_parallelism_threads=1) sess = tf.Session(config=session_conf)

but I use Keras with backend Tensorflow, not Tensorflow directly. So the only line I use is import Keras

Also, my stacktrace looks differently from the one in the above post, so maybe you can detect another root cause - would be really really great (I spent two full days AND nights for this now...):

2018-12-01 10:04:43 Using TensorFlow backend. 2018-12-01 10:04:43 2018-12-01 10:04:06.124509: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2018-12-01 10:04:43 /usr/lib/python3.6/site-packages/keras/engine/saving.py:269: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.#012 warnings.warn('No training configuration found in save file: ' 2018-12-01 10:04:43 MobileNet model loaded. 2018-12-01 10:04:43 ResNet50 model loaded. 2018-12-01 10:04:43 InceptionV3 model loaded. 2018-12-01 10:04:43 WSGI app 0 (mountpoint='') ready in 41 seconds on interpreter 0xe56dc0 pid: 1 (default app) 2018-12-01 10:04:43 *** uWSGI is running in multiple interpreter mode *** 2018-12-01 10:04:43 gracefully (RE)spawned uWSGI master process (pid: 1) 2018-12-01 10:04:43 spawned uWSGI worker 1 (pid: 50, cores: 1) 2018-12-01 10:04:43 metrics collector thread started 2018-12-01 10:04:43 pthread_create(): Resource temporarily unavailable [core/utils.c line 3853] 2018-12-01 10:04:43 unable to start offload thread 1 for worker 1 !!! 2018-12-01 10:04:43 spawned 1 offload threads for uWSGI worker 1 2018-12-01 10:04:43 Sat Dec 1 10:04:43 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 10.0.0.98) !!! 2018-12-01 10:04:43 Sat Dec 1 10:04:43 2018 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET / (10.0.0.98) 2018-12-01 10:04:43 announcing my loyalty to the Emperor... 2018-12-01 10:04:50 terminate called after throwing an instance of ' 2018-12-01 10:04:50 std::system_error 2018-12-01 10:04:50 ' 2018-12-01 10:04:50 what():
2018-12-01 10:04:50 Resource temporarily unavailable 2018-12-01 10:04:50 2018-12-01 10:04:51 DAMN ! worker 1 (pid: 50) died, killed by signal 6 :( trying respawn ... 2018-12-01 10:04:51 Respawned uWSGI worker 1 (new pid: 54) 2018-12-01 10:04:51 pthread_create(): Resource temporarily unavailable [core/utils.c line 3853] 2018-12-01 10:04:51 unable to start offload thread 1 for worker 1 !!! 2018-12-01 10:04:51 spawned 1 offload threads for uWSGI worker 1

You could also try this: http://help.pythonanywhere.com/pages/MachineLearningInWebsiteCode/

thanks @glen but i need to work with the TensorFlow backend, Theano is not an option. So my question can be condensed into:

How can you prevent Keras with backend TensorFlow to produce the following error log messages:

pthread_create(): Resource temporarily unavailable [core/utils.c line 3853]

unable to start offload thread 1 for worker 1 !!!

Yeah, I got rid of the pthread_create() error by finally finding the code that forces single-thread mode for Keras thanks to by

from keras import backend as K
import tensorflow as tf
session = tf.Session(config=tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1))
K.set_session(session)

So the server log confirms single-thread mode:

2018-12-01 15:24:08 *** Operational MODE: single process ***
2018-12-01 15:24:08 initialized 38 metrics
2018-12-01 15:24:08 Using TensorFlow backend.

But now I have the next problem: Broken pipe!

2018-12-01 15:24:08 *** uWSGI is running in multiple interpreter mode ***
2018-12-01 15:24:08 gracefully (RE)spawned uWSGI master process (pid: 1)
2018-12-01 15:24:08 spawned uWSGI worker 1 (pid: 47, cores: 1)
2018-12-01 15:24:08 metrics collector thread started
2018-12-01 15:24:08 spawned 2 offload threads for uWSGI worker 1
2018-12-01 15:24:08 Sat Dec  1 15:24:08 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 10.0.0.98) !!!
2018-12-01 15:24:08 Sat Dec  1 15:24:08 2018 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET / (10.0.0.98)

This must be related to the wsgi setup - anyone knows what the last two lines about broken pipe mean and how to solve them?

It means that the client disconnected before the response was sent back. It's usually not something to worry about unless your views are taking a long time to complete.