Forums

Broke something upgrading scrapy

Hi

Hope someone can help. Ran pip install --upgrade --user scrapy, all good until cleaning up then:

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/usr/lib/python2.7/shutil.py", line 303, in move
    os.unlink(src)
OSError: [Errno 13] Permission denied: '/usr/local/bin/easy_install'
Storing debug log for failure in /home/richyvk/.pip/pip.log

now scrapy --version gives me:

Traceback (most recent call last):
  File "/home/richyvk/.local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 122, in execute
    cmds = _get_commands_dict(settings, inproject)
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
    cmds = _get_commands_from_module('scrapy.commands', inproject)
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
    for cmd in _iter_command_classes(module):
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
    for module in walk_modules(module_name):
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 68, in walk_modules
    submod = import_module(fullpath)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/commands/bench.py", line 3, in <module>
    from scrapy.tests.mockserver import MockServer
  File "/home/richyvk/.local/lib/python2.7/site-packages/scrapy/tests/mockserver.py", line 6, in <module>
    from twisted.internet import reactor, defer, ssl
  File "/home/richyvk/.local/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
    from OpenSSL import SSL
  File "/home/richyvk/.local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/home/richyvk/.local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/home/richyvk/.local/lib/python2.7/site-packages/OpenSSL/_util.py", line 3, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/home/richyvk/.local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 60, in <module>
    class Binding(object):
  File "/home/richyvk/.local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 108, in Binding
    libraries=_get_libraries(sys.platform)
  File "/home/richyvk/.local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 97, in build_ffi_for_binding
    extra_link_args=extra_link_args,
  File "/home/richyvk/.local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 105, in build_ffi
    ffi = FFI()
  File "/home/richyvk/.local/lib/python2.7/site-packages/cffi/api.py", line 52, in __init__
    from . import cparser, model
  File "/home/richyvk/.local/lib/python2.7/site-packages/cffi/cparser.py", line 7, in <module>
    import pycparser
ImportError: No module named pycparser

Any ideas what I've done wrong here??

deleted-user-557803 | 10 posts | Dec. 19, 2014, 4:09 a.m. | permalink

An update, uninstalled and reinstalled scrapy with pip and everything seems OK now. Still no idea what I did wrong the first time??

deleted-user-557803 | 10 posts | Dec. 19, 2014, 8:54 a.m. | permalink

Hi richyvk,

I tried installing scrapy and came across the same error as you. Looking at it more detail, the error comes from a package dependency called distribute that easy_install is trying to install into the global python site packages (/usr/local/lib/python2.7/dist-packages) instead of the local site packages (~/.local/lib/python2.7/site-packages), even though we have passed it --user.

This seems to be a bug in the distribute/pip setup code. Also followed you and uninstalled and reinstalled scrapy, after which I got no error as well. However, I believe this leaves you with the earlier version of scrapy (try runing pip show scrapy).

One method where I could easily get the most updated version of scrapy installed was to create a virtualenv. Take a look at our wiki here for info. Another method I can think of is to manually install distribute to your local packages.

Conrad

Staff conrad | 4232 posts | PythonAnywhere staff | Dec. 19, 2014, 10:46 a.m. | permalink

Hey Conrad

Many thanks for the reply. I seem to have latest version install now after reinstalling. pip show scrapy gives:

Name: Scrapy
Version: 0.24.4
Location: /home/richyvk/.local/lib/python2.7/site-packages
Requires: pyOpenSSL, six, queuelib, lxml, w3lib, cssselect, Twisted

I'm happy with this. I know I could have used a virtualenv but didn't want to keep having to activate that everytime. I have no code that uses older versions of scrapy.

Do you guys have plans to upgrade the default scrapy version? It's quite different to the version you currently have. My existing code is totally broken using the default version you have, modules are all differently named etc.

Thanks again Richard

deleted-user-557803 | 10 posts | Dec. 19, 2014, 8:30 p.m. | permalink

Good point- the concern is that somebody else might have code dependent on this older scrapy though. I just put in a ticket to get scrapy upgraded for 2.6/2.7 and installed for 3.3/3.4 and I'll let you know if the change goes through!

Keep in mind it might take a couple weeks to a month or two though.

Staff conrad | 4232 posts | PythonAnywhere staff | Dec. 21, 2014, 6:13 p.m. | permalink

If the only concern is the hassle of activating a virtualenv in your console, you could just modify your .bashrc to activate a "default" virtualenv when you log in?

deleted-user-39880 | 669 posts | Dec. 22, 2014, 12:41 p.m. | permalink

Any progress upgrading scrapy for 2.7?

deleted-user-587466 | 1 post | Jan. 11, 2015, 8:34 p.m. | permalink

Seems it takes a few months??

deleted-user-557803 | 10 posts | Jan. 11, 2015, 9:20 p.m. | permalink

Hi Danweeks,

Currently installing scrapy inside a virtualenv works fine/without any problems. (just run pip install scrapy)

Unfortunately we have not gotten scrapy onto our build yet. As you can see from our latest deploy announcement, we have been working hard to get packages added- so don't worry we will get to scrapy at some point! :-p

However, just a heads up- to avoid breaking the exising code from our users that are using scrapy currently, we might lean towards installing the newest version of scrapy on python3.x but not updating scrapy on python 2. As you said, the older scrapy is very different from the newer version, which means that current users using scrapy will all find their webapps broken if we update.

Therefore I would strongly suggest installing scrapy yourself from within a virtualenv. Let me know if you have any problems with that.

Conrad

Staff conrad | 4232 posts | PythonAnywhere staff | Jan. 12, 2015, 12:52 p.m. | permalink