Forums

PA 3.6 interpreter unfamiliar with updated asyncio/aiohttp?

Hi! I have several scripts using asyncio and aiohttp, which run well at home but not on PA. Although I have both modules installed and execute the scripts with PA 3.6, I get the following errors as if it were a 3.4 or 3.5 interpreter:

1) Coroutines defined by async def some_func() are not recognized as such. Instead, upon calling that function, the interpreter prompts: A Future or coroutine is required

2) When I decorate the coro with @asyncio.coroutine, the interpreter recognizes it but doesn't understand the await call inside (it works without await):

j = await r.json()
          ^
SyntaxError: invalid syntax

3) The interpreter stumbles over the async with statement wrapping aiohttp commands:

async with aiohttp.ClientSession(headers=headers) as session:
         ^
SyntaxError: invalid syntax

Thank you for your support!

Hmm, that sounds odd. How are you running the scripts?

I tried it from the console and as a task: same errors.

How did you run it in the console, though? Did you use the python3.6 command, or just python3?

Regarding the scheduled tasks -- I assume it wasn't with one of the ones you currently have, which are all using Python 3.5...?

Console: I just click on "Run" from the editor. The scripts begin with !# python3.6 and after the console is started, it shows Loading Python3.6 interpreter...

Scheduled tasks: Correct. Those are old scripts that don't use asyncio and aiohttp. The scripts that raise the problems are started with 3.6. I just deleted them from the tasks since they don't run anyway.

If I try to run the scripts with 3.5, I get an error because the modules are not installed for that version. So, yes, 3.6 from all I can tell.

I have put a scratch.py script in my home folder for you, which is a very condensed version of the scripts that I would like to run. It causes exactly the same errors (while it runs exactly as intended on my home PC):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#! python3.6

import asyncio
import aiohttp

async def download_html(url, session):
    async with session.get(url) as r:
        return await r.text()

async def manage_download():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.Task(download_html('http://google.com', session)) for i in range(20)]
        return await asyncio.gather(*tasks)

loop = asyncio.new_event_loop()
results = loop.run_until_complete(manage_download())
for i in results:
    print(i[:50])
loop.close()

Thanks! I was just going to ask if you had a minimal repro. Taking a look now.

Hmm. So it doesn't look like it's the version of Python that we have installed that's causing the problem. I created a new account (with unrestricted Internet access) and ran your script without problems:

18:21 ~ $ pip3.6 install --user aiohttp
Collecting aiohttp
  Downloading https://files.pythonhosted.org/packages/75/27/d76236fb1620eba35d6f5c60eaa08de36256a6c8e6516b3e132868b51509/aiohttp-3.2.1-cp36-cp36m-manylinux1_x86_64.whl (761kB)
    100% |████████████████████████████████| 768kB 967kB/s 
Collecting yarl<2.0,>=1.0 (from aiohttp)
  Downloading https://files.pythonhosted.org/packages/4a/da/aec3974c0de3f7cde440bc5f984145cdc2c3ac2060944b1233e392e2e6db/yarl-1.2.4-cp36-cp36m-manylinux1_x86_64.whl (252kB)
    100% |████████████████████████████████| 256kB 1.5MB/s 
Collecting attrs>=17.3.0 (from aiohttp)
  Downloading https://files.pythonhosted.org/packages/41/59/cedf87e91ed541be7957c501a92102f9cc6363c623a7666d69d51c78ac5b/attrs-18.1.0-py2.py3-none-any.whl
Collecting idna-ssl>=1.0 (from aiohttp)
  Downloading https://files.pythonhosted.org/packages/c4/3b/facf5a5009e577e7764e68a2af5ee25c63f41c78277260c2c42b8cfabf2e/idna-ssl-1.0.1.tar.gz
Collecting async-timeout<4.0,>=3.0 (from aiohttp)
  Downloading https://files.pythonhosted.org/packages/96/0f/e6357458c87fb4ed8f3df215773f3caad40968f10e05552cbd8bd28415e4/async_timeout-3.0.0-py3-none-any.whl
Collecting multidict<5.0,>=4.0 (from aiohttp)
  Downloading https://files.pythonhosted.org/packages/cc/30/508a22a28dfb50cf9079cd9d0cf9b0d7dbae5afdf9823977351cbd548897/multidict-4.3.1-cp36-cp36m-manylinux1_x86_64.whl (476kB)
    100% |████████████████████████████████| 481kB 1.4MB/s 
Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/dist-packages (from yarl<2.0,>=1.0->aiohttp)
Building wheels for collected packages: idna-ssl
  Running setup.py bdist_wheel for idna-ssl ... done
  Stored in directory: /home/gttest20180515a/.cache/pip/wheels/67/97/96/5745320e00648bb29ebaa62a555b613f34a94ccefeb8fc7d45
Successfully built idna-ssl
Installing collected packages: multidict, yarl, attrs, idna-ssl, async-timeout, aiohttp
Successfully installed aiohttp-3.2.1 async-timeout-3.0.0 attrs-18.1.0 idna-ssl-1.0.1 multidict-4.3.1 yarl-1.2.4
18:22 ~ $ cat > test.py
#! python3.6
import asyncio
import aiohttp
async def download_html(url, session):
    async with session.get(url) as r:
        return await r.text()
async def manage_download():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.Task(download_html('http://google.com', session)) for i in range(20)]
        return await asyncio.gather(*tasks)
loop = asyncio.new_event_loop()
results = loop.run_until_complete(manage_download())
for i in results:
    print(i[:50])
loop.close()
18:22 ~ $ python3.6 test.py
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
<!doctype html><html itemscope="" itemtype="http:/
18:22 ~ $

I got the same result when I ran it from the editor.

So clearly something very odd is going on here. What happens if you run your script from a bash console using python3.6, like I did above?

Also, what do you get if you run which python3.6 in a bash console?

Very odd indeed. :-o

20:35 ~ $ python3.6 scratch.py
Traceback (most recent call last):
  File "scratch.py", line 16, in <module>
    results = loop.run_until_complete(manage_download())
  File "/home/pimpcoltd/.local/lib/python3.6/site-packages/asyncio/base_events.py", line 296, in run_until_complete
    future = tasks.async(future, loop=self)
  File "/home/pimpcoltd/.local/lib/python3.6/site-packages/asyncio/tasks.py", line 516, in async
    raise TypeError('A Future or coroutine is required')
TypeError: A Future or coroutine is required
sys:1: RuntimeWarning: coroutine 'manage_download' was never awaited
20:38 ~ $ 
20:38 ~ $ 
20:38 ~ $ which python3.6
/usr/bin/python3.6
20:38 ~ $ 
20:38 ~ $ 
20:38 ~ $ python3.6 test.py
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    results = loop.run_until_complete(manage_download())
  File "/home/pimpcoltd/.local/lib/python3.6/site-packages/asyncio/base_events.py", line 296, in run_until_complete
    future = tasks.async(future, loop=self)
  File "/home/pimpcoltd/.local/lib/python3.6/site-packages/asyncio/tasks.py", line 516, in async
    raise TypeError('A Future or coroutine is required')
TypeError: A Future or coroutine is required
sys:1: RuntimeWarning: coroutine 'manage_download' was never awaited
20:40 ~ $ 
20:40 ~ $ 
20:40 ~ $

When I try to reinstall asyncio and aiohttp modules:

20:46 ~ $ pip3.6 install --user asyncio
Requirement already satisfied: asyncio in ./.local/lib/python3.6/site-packages
20:46 ~ $ 
20:46 ~ $ pip3.6 install --user aiohttp
Requirement already satisfied: aiohttp in ./.local/lib/python3.6/site-packages
Requirement already satisfied: idna-ssl>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp)
Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp)
Requirement already satisfied: async-timeout<4.0,>=3.0 in ./.local/lib/python3.6/site-packages (from aiohttp)
Requirement already satisfied: attrs>=17.3.0 in ./.local/lib/python3.6/site-packages (from aiohttp)
Requirement already satisfied: multidict<5.0,>=4.0 in ./.local/lib/python3.6/site-packages (from aiohttp)
Requirement already satisfied: yarl<2.0,>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/dist-packages (from idna-ssl>=1.0->aiohttp)
20:46 ~ $

OK. What does pip3.6 show asyncio aiohttp show?

11:17 ~ $ pip3.6 show asyncio aiohttp Name: asyncio Version: 3.4.3 Summary: reference implementation of PEP 3156 Home-page: http://www.python.org/dev/peps/pep-3156/ Author: UNKNOWN Author-email: UNKNOWN License: UNKNOWN Location: /home/pimpcoltd/.local/lib/python3.6/site-packages Requires: --- Name: aiohttp Version: 3.2.1 Summary: Async http client/server framework (asyncio) Home-page: https://github.com/aio-libs/aiohttp/ Author: Nikolay Kim Author-email: fafhrd91@gmail.com License: Apache 2 Location: /home/pimpcoltd/.local/lib/python3.6/site-packages Requires: multidict, idna-ssl, attrs, async-timeout, chardet, yarl 11:17 ~ $

I think the problem is that asyncio package. asyncio is part of Python 3.6, so that package is probably not helping. I would suggest deleting the contents of .local, then re-installing only aiohttp and seeing if that helps.

Excellent! Thanks a lot! Both the test script and the real one run well now -- except for the fact that the latter gets killed. I guess that's a download volume issue (1.5 GB), which I've encountered on other occasions: Any option to raise the download limit?

We don't kill processes for going over a download limit. We kill processes if they use to much RAM. You should be able to write your code so it uses less RAM.

OK, I see where the RAM issue comes in. But my 16 GB notebook can cope with the script, so I'm surprised that I'm crossing a line here. Any option to raise the limit?

We do have some options, but it does cost more -- essentially, you'd need a private server, and one with 15GiB RAM comes out at $250/month. Let us know if you're interested!

No worries, I've changed the script already -- turning it partly slower and partly faster, so I'm fine with the status quo. That said, I find it a bit akward to not know the exact limit and to see scripts killed out of the blue with no opportunity to save the data: a waste of paid cpu usage.