24

When I try to run "scrapy startproject" in vscode, below error comes up.

Traceback (most recent call last):
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/bin/scrapy", line 5, in <module>
    from scrapy.cmdline import execute
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/spiders/__init__.py", line 11, in <module>
    from scrapy.http import Request
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/request/form.py", line 10, in <module>
    import lxml.html
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/html/__init__.py", line 53, in <module>
    from .. import etree
ImportError: dlopen(/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so, 0x0002): tried: '/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/etree.cpython-39-darwin.so' (no such file), '/usr/lib/etree.cpython-39-darwin.so' (no such file)

And then I figure out that, the "mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')"is the main promblem. I tried with https://github.com/nmwsharp/robust-laplacians-py/issues/7 and I tried to uninstall the old packages and install it again using arch -arm64 pip install Scrapy. But it didn't work...

So...how to fix this problem...?

Ryan
  • 343
  • 1
  • 2
  • 5

7 Answers7

9

I use conda from miniforge for python packages etc. This works.

This means that I don't know first hand how pypi works, I just have looked at the documentation. I think by default it looks for prebuilt wheels and if it can't find one it builds the extension locally from source. This is no problem for pure python modules. However if the wheel should contain C code that has to be compiled then I think pypi has a large bug here in that it just looks for macos wheels and does not check the CPU architecture.

The error you have is that lxml has been built as an Intel binary. Looking at the downloads for lxml at pypi there are no wheels for Apple arm CPUs.

So if not conda miniforge you need to compile lxml yourself or force pip to do the build locally. pypi documentation suggests

You can override pip`s default behavior by e.g. using its –no-binary option.

I would also note that Macports provides arm versions of python and lxml and possibly scrapy

mmmmmm
  • 28,660
  • 17
  • 84
  • 140
  • I have been tried...but the same error comes back. It seems that lxml library does not working some reason... – Ryan Feb 12 '22 at 06:39
  • Heyyyyyyyy. I just figured out the solution!!!!!!!!!!!!! First I really appreciate @ mmmmmm help. I run `conda install -c conda-forge lxml`, then the scrapy startproject works!!!!! – Ryan Feb 12 '22 at 06:55
  • But Conda is not free for commercial usage. A paid license is needed, isn't it ? – Jeff T. Apr 08 '22 at 06:54
  • I use Macports and it works perfectly. Thanks @mmmmmm. – Jeff T. Apr 08 '22 at 10:31
  • 1
    @JeffT. Looking at https://docs.conda.io/projects/conda/en/latest/ It says it is open source. ALso https://github.com/conda/conda says it is BSD .Anaconda does have pricing – mmmmmm Apr 08 '22 at 10:43
8

What it worked for me was:

pip uninstall lxml
ARCHFLAGS="-arch arm64" pip install lxml --compile --no-cache-dir
Sandra Meneses
  • 81
  • 1
  • 1
5

I've faced with the same issue (and created this ticket with details of my research). But it ended with removing the installed package:

pip uninstall lxml

... and installing it building from sources:

pip install --no-binary lxml lxml

Note the useful recommendation about platform specifics (for MacOS, for example).

Alexandre Fedosov
  • 51
  • 1
  • 2
4

This is a case of some python libraries not being compiled for M1 Apple Silicon CPUs, but only for Intel.

Try running VSCode in Rosetta.

benwiggy
  • 28,223
  • 1
  • 36
  • 89
  • Thanks for your help. I really appreciated. But, I tried run scrapy startproject VSCode rosetta terminal...still it shows same error... – Ryan Feb 11 '22 at 11:56
  • I tried that without success. With uname in the terminal I get x86_64, but doing ´platform.uname()` in Python still get arm64. – Christian Jan 19 '23 at 17:25
-1

I got this when running an npm command that used compiled C packages. Solution was to run terminal in x86_64 mode and using node v14.x.x, as newer versions of node didn't work.

JTE
  • 99
  • 1
  • The OP is running in vscode so how does your answer improve on the previous https://apple.stackexchange.com/a/436826/237 ? – mmmmmm May 15 '22 at 11:29
-1

Running the commands below resolved the problem for me:

  • arch -arm64e gem install nokogiri -v '1.13.6' --platform=ruby -- --use-system-libraries
  • bundle
Glorfindel
  • 3,774
  • 7
  • 30
  • 46
Piyush Chaudhary
  • 1
-1

I got this same import error with uamqp library

ImportError: dlopen(/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so, 0x0002): tried: '/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

I resolved this import error by running below command:

  1. Uninstalling uamqp library:

    pip uninstall uamqp

  2. Setting ARCHFLAGS for arm64 architecture and using no cache so it will download respective library suits arm64 architecture.

    ARCHFLAGS="-arch arm64" pip install uamqp --compile --no-cache-dir

Hope this helps you as well !!

Mahesh
  • 1