Help installing PySAL on win10...

[FaultyMario]

I had dm'd TomServo with a question that i had. Tom, please ignore that DM as it seems that i can run cURL.

So I have Qgis installed and I need to import the PySAL packages.

However it seems the PySAL developers recommend using Conda (https://github.com/pysal/PySAL) over Python.

Which seems a little crazy considering that Qgis has a built-in interpreter for installing external Python packages, but not a Conda one.

So far, I've been unable to accomplish this: https://anitagraser.com/2014/07/13/installing-pysal-for-osgeo4w/

So I'm wondering if I should try to do this (https://gis.stackexchange.com/questions/119495/does-qgis-work-with-anaconda) instead. Any thoughts?

I'm just completely clueless about all these languages. But the guy answering is working on Linux, not Win.

There's also this suggestion (https://github.com/danioxoli/HotSpotAnalysis_Plugin/issues/2#issuecomment-343381545) from a dude working on a Windows environment.


On Windows there is a third possibility to install Pysal (not really suggeted but usually it works). It is described in the installation instruction within the readme file.

Basically, you have to open this folder:

C:\PROGRA1\QGIS21.18\apps\Python27\lib\site-packages

Remove (if exist) any folder of Pysal.

Download and decompress the folder you can download from here:

Please, help. I'm going crazy.

[FaultyMario]

I think the guy from the last link is correct. Maybe my OSGeoW and QGis paths are different and that's why installing something thru the OSGeoW shell doesn't show up in the Python terminal within QGis.

How could I check that?

[Tom Servo]

Just to make your life easier, any chance you can do all this on WSL? I'm giving it a quick shot to see if I can get it functioning in an Ubuntu install running in WSL2 on my Windows 11 box.

Conda appears to just be a different distro of Python that's specifically made to make things "easier". The definition of "easier", of course, is up for debate. I'm completely unfamiliar with conda and had not heard of it until now.

I was able to get pysal installed on windows but then ran into issues actually getting a python script to play nice with it. I was able to get it to work just fine on Ubuntu on WSL though.



swervo@DESKTOP-M700UUR:~$ sudo apt-get install -y python3 python3-pip
swervo@DESKTOP-M700UUR:~$ pip3 install pysal

swervo@DESKTOP-M700UUR:~$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysal
>>>

[Tom Servo]

Also, I have python3 installed on my windows box and was able to make that work as well from the standard command line. Same thing, "pip3 install pysal" and run it under "python3".

If you are running into situations where sometimes site-packages are causing issues, you might want to look into pyenv to isolate different apps to their own sets of libraries. Aside, the fact that things like pyenv have to exist is part of the reason I never want to work with Python again.

[Tom Servo]

Oooh, I wonder if that's part of the problem:



As of version 2.0.0 PySAL has shifted to Python 3 only.


The quote you showed is running Python 2.7, not sure if that's where you're also getting stuck?

[Tom Servo]

And in all of this, I think I overlooked that this is a problem getting Qgis to play nice. Okay, giving that a shot.

[Tom Servo]

Okay, I've got QGIS installed. I have no idea how to tell if it's recognizing my pysal installation or not - how would I verify that? I have no idea what I'm doing with this tool.

[FaultyMario]

ctrl + alt + p

opens the python terminal within qgis.

once there, check if you can import the numpy, scipy and pysal packages. the three are interdependent but only the first two come as default installs.

[FaultyMario]

i am running python 3.9 and i was able to paste the tar file with pysal 1.14 to the scripts folder, but it didn't seem to do anything.

[FaultyMario]

yes it's basically getting 3 "apps" to play nice with each other. Or rather just 2, qgis and python; osgeow is just there to thirdwheel; but as with all thirdwheelers, it's just a nuisance.

[Tom Servo]

Hrm, well, numpy and scipy are both fine, but pysal is saying it's not installed. Looks like the latest QGIS is running python 3.12 so that should be fine. Now that I've made it that far, let me see if I can figure out more of that. I'm assuming dropping a tarball into a folder won't quite do what we're hoping for, but lemme see what I can find out.

[Tom Servo]

Okay, so this just worked fine for me. I opened up OSGeo4W Shell and it was already defaulted to the directory that I installed QGIS into. I ran "pip install pysal", waited for a bit, then closed QGIS and restarted it. I was able to import pysal in the python interpreter window just fine.

Do you have any error messages or anything from OSGeo4W Shell?

In the shell, if you run "python -m site", it should give you a list of the directories it's looking for packages in. For mine, one of them is C:\Users\Swervo\AppData\Roaming\Python\Python312\s ite-packages and, in there, I see a pysal directory that was just updated a few minutes ago, so it does appear that's where it installs things.

EDIT - I have no idea why the board is inserting a space in "site" in the site packages bit, but that space isn't supposed to be there.

[FaultyMario]

You think uninstall and reinstalling OSGeo4W could help me solve my problem?

just making sure that it shares the packages folder with QGis?

[FaultyMario]

These are the contents of my Qgis folder:

https://pbs.twimg.com/media/GMccIdeWUAINTGZ.png

https://pbs.twimg.com/media/GMccI7yWAAAoqIv.png

And when I run pip install pysal in OSGeo4W this is what happens...

https://pbs.twimg.com/media/GMccJblW0AAgLjW.png

Like, dude, :? :? :?

[FaultyMario]

Why all the trouble?

I could, in theory, run the spatial weights analysis in R-Studio, but R is kind of a black box. And when you do things in QGis you are seeing what's happening.

PySAL would give me back a spatial matrix, a measure of how many of the objects in a map are adjacent to one another.

Like so:

https://www.researchgate.net/profile/Youyi-Bi/publication/327776659/figure/fig1/AS:672948855791617@1537454918147/A-typical-spatial-weights-matrix-Note-that-A-and-E-do-not-share-a-side-thus-their.png

The problem comes with an object matrix being N^2. Or, in the above example 5^2. My N=368. :rolleyes:

[FaultyMario]

Okay, so this just worked fine for me. I opened up OSGeo4W Shell and it was already defaulted to the directory that I installed QGIS into. I ran "pip install pysal", waited for a bit, then closed QGIS and restarted it. I was able to import pysal in the python interpreter window just fine.

Do you have any error messages or anything from OSGeo4W Shell?

In the shell, if you run "python -m site", it should give you a list of the directories it's looking for packages in. For mine, one of them is C:\Users\Swervo\AppData\Roaming\Python\Python312\s ite-packages and, in there, I see a pysal directory that was just updated a few minutes ago, so it does appear that's where it installs things.

EDIT - I have no idea why the board is inserting a space in "site" in the site packages bit, but that space isn't supposed to be there.

I think that's my problem.
https://pbs.twimg.com/media/GMcfhiJXcAAlIU5.png

[Tom Servo]

Oh yeah, weird, you don't have a site-packages folder. I also wonder if they did a shoddy job on a normally edge case part of the coding where you didn't install it under "Program Files," e.g. "C:\Program Files\OSGeo4W". However, do you also have a separate OSGeo4W install from your QGIS install? Mine came with both and OSGeo4W is installed in the same folder as QGIS. I'd imagine if it's failing to find the user-scoped site-packages, it'd use a local one, but if it's a separate installation it would be installing it its own local library rather than one that QGIS can see.

Your QGIS installation directory (which it looks like up above is in C:\Program Files\QGIS 3.28.4) should have a file in it called OSGeo4W.bat. Try double-clicking on that and seeing if it opens up a command prompt and, if it does, trying the "pip install" stuff from there. I think it is that you don't have a global site-packages location (likely because you didn't install python separately, which you wouldn't be expected to do) but you also have an installation of OSGeo4W that's entirely separate from QGIS and they're both using their own local site-packages directory for libraries.

[FaultyMario]

What you wrote is not making a lot of sense to me ATM, but I'll follow your instructions* fo sho.

Peace!


___

* Once I figure them out, give me a break, I just finished doing my taxes! I can do a couple of things I'm not familiar with, but not one after the other!

[FaultyMario]

However, do you also have a separate OSGeo4W install from your QGIS install?

I don't think so, I don't recall having installed OSGeo4W separately. I think I might have installed it from within QGis' GUI.

[Tom Servo]

What you wrote is not making a lot of sense to me ATM, but I'll follow your instructions* fo sho.

Peace!


___

* Once I figure them out, give me a break, I just finished doing my taxes! I can do a couple of things I'm not familiar with, but not one after the other!

Hahah, no worries. If you need me to go into more detail/step by step, I'm happy to do that. That said, it does look to me like you somehow have QGIS and OSGeo4W installed in different locations and I think they're each using their own repository location to store/retrieve pysal which is why OSGeo4W is happily storing it in its location ad then QGIS is looking in its and being like "I dunno where it is."

[FaultyMario]

That much, I understood.

The problem now is getting OSGeo4W to store its toys in QGis' chest so they both can play with them.

[Tom Servo]

Yeah - so I think that QGIS will have a batch file to run OSGeo4W in its installation directory. In "File Explorer" (or whatever Windows calls its "show me the filesystem" tool these days), go to C:\Archivos de Programa\QGIS 3.28.4\. When I go there, I see the directories "apps", "bin", "doc", "etc", "lib", "share", "var", and then two files, "OSGeo4W.bat" and "OSGeo4W.ico". If you double click on the .bat one, it should open up a command shell and the prompt should be "C:\Archivos de Programa\QGIS 3.28.4". (might be missing the trailing slash) At that point, run "pip install pysal" and it *should* install to the "site-packages" directory that QGIS is looking for.

It would not surprise me if you accidentally installed a separate OSGeo4W, as that kinda thing tends to happen when following a bunch of random articles on the internet that were written anywhere from 2 to 10 years ago. I do it constantly, and then you get yourself in a state that no article describes and have no idea how to get out of it. I'm *hoping* this will work, and then we can work on cleaning up the dodgy OSGeo4W. The fact that its directory is off your root C:\ directory makes me think that some bit of advice somewhere was less helpful than the author meant it to be.

[Tom Servo]

Otherwise, if that doesn't work, I think I'd recommend installing python3 from python's own website, then trying the "pip install pysal" thing again to see if it'll just install to a global packages directory and then you can just wipe your hands of all of it.

[JoshInKC]

I just wanted to pop in and say that while I can offer no actual help (& certainly not on the the scale that you're getting), I know and feel your pain, Mario. Just reading this thread made my anxiety kick into overdrive - if I could add up all the hours I've spent troubleshooting broken things in QGIS... I don't know what I'd do, but it equals a truly huge amount of time.

[Tom Servo]

Interpreted languages and versioning has been a massive pain point for at least the past 25 years. It's part of the reason I like go, you just compile the whole thing into a single binary and ship that.

[FaultyMario]

It would not surprise me if you accidentally installed a separate OSGeo4W, as that kinda thing tends to happen when following a bunch of random articles on the internet that were written anywhere from 2 to 10 years ago.

My intuition tells me this all started when I took an online course and us students all had different versions of QGIS, so the instructors went for the lowest common denominator when we had to install packages for the course.

[FaultyMario]

This ought to do it...

https://pbs.twimg.com/media/GMkuqFRWEAAZwhZ.png

https://pbs.twimg.com/media/GMkuqEyXYAArh13.png


right?

[FaultyMario]

So I followed the instructions to install the PySAL tarball from the command prompt, and this happened:

https://pbs.twimg.com/media/GMlBnkkWEAMJh8T.png

So I then made sure that that location was added to where QGIS looks for plugins.

Nothing. Happened.

[FaultyMario]

I repeated the tarball install but from the OSGeo4W prompt.

:cry:

[FaultyMario]

What do I need to do, FFS!

[Tom Servo]

If you open up the python prompt in QGIS and run:



import sys
sys.path


What does it say?

[Tom Servo]

If you open up the python prompt in QGIS and run:



import sys
sys.path


What does it say?

[FaultyMario]

This:


sys.path
['C:\\Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python/plugins\\spatialanalysistoolbox',
'C:/PROGRA~1/QGIS32~1.4/apps/qgis-ltr/./python',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python/plugins',
'C:/PROGRA~1/QGIS32~1.4/apps/qgis-ltr/./python/plugins',
'C:\\Program Files\\QGIS 3.28.4\\bin\\python39.zip',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\DLLs',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\lib',
'C:\\Program Files\\QGIS 3.28.4\\bin',
'C:\\Users\\ciber\\AppData\\Roaming\\Python\\Pytho n39\\site-packages',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\lib\\si te-packages',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\lib\\si te-packages\\win32',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\lib\\si te-packages\\win32\\lib',
'C:\\PROGRA~1\\QGIS32~1.4\\apps\\Python39\\lib\\si te-packages\\Pythonwin',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python']

(I've added some line breaks to make it easier on the eyes).

[FaultyMario]

What version QGis are you running Tom? I'm on 3.28 and I just checked the release notes for 3.3x and I think they repacked/rebuilt the python dependencies.

PySAL appears on the list of "updated" python extensions.

If it turns out that it was all a matter of updating to the latest stable release, I'm going postal.

[Tom Servo]

I'm on 3.36.2. One thing I'm noticing from your previous screenshots is the two paths I see, c:\users\ciber\appdata\local\programs\python\pytho n312\lib\site-packages and c:\program files\qgis 3.28.4\apps\python39\lib\site-packages, are not present in that sys.path list.

On the version I'm on, under Settings -> Options -> System, there's a list of environment variables down near the bottom of that screen. One of them is PYTHONPATH, it might work to try to add one (or both) of those directories to that value, separated by semicolons. I'd probably just focus on the second of those paths though, that's likely where you'd want to be pulling your packages from.

Oh, and to save you the effort of adding newlines next time, I should have had you run



import sys
print('\n'.join(sys.path))

[FaultyMario]

installed 3.34 (latest stable release) and then sys.path


'C:/PROGRA~1/QGIS33~1.6/apps/qgis-ltr/./python',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python/plugins',
'C:/PROGRA~1/QGIS33~1.6/apps/qgis-ltr/./python/plugins',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\grass\\grass83\\e tc\\python',
'C:\\Users\\ciber\\Documents', 'C:\\Program Files\\QGIS 3.34.6\\bin\\python312.zip',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\DLLs',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\Lib',
'C:\\Program Files\\QGIS 3.34.6\\bin',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\Lib\\s ite-packages',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\Lib\\s ite-packages\\win32',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\Lib\\s ite-packages\\win32\\lib',
'C:\\PROGRA~1\\QGIS33~1.6\\apps\\Python312\\Lib\\s ite-packages\\Pythonwin',
'C:/Users/ciber/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python'

[Tom Servo]

Okay. So if you go in file explorer to C:\Program Files\QGIS 3.34.6, there should be a file in there called OSGeo4W.bat. If you double click on that, it should open up a command prompt.

From there, run python and do the same thing as above. Here's the contents of my command prompt window when doing that:



run o-help for a list of available commands
I:\Program Files\QGIS>python
Python 3.12.3 (main, Apr 14 2024, 17:21:43) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print('\n'.join(sys.path))

I:\Program Files\QGIS\bin\python312.zip
I:\Program Files\QGIS\apps\Python312\DLLs
I:\Program Files\QGIS\apps\Python312\Lib
I:\Program Files\QGIS\bin
C:\Users\swervo\AppData\Roaming\Python\Python312\s ite-packages
I:\Program Files\QGIS\apps\Python312
I:\Program Files\QGIS\apps\Python312\Lib\site-packages
I:\Program Files\QGIS\apps\Python312\Lib\site-packages\win32
I:\Program Files\QGIS\apps\Python312\Lib\site-packages\win32\lib
I:\Program Files\QGIS\apps\Python312\Lib\site-packages\Pythonwin
>>>

[FaultyMario]

Gotcha.


run o-help for a list of available commands
C:\Program Files\QGIS 3.34.6>py
Python 3.12.3 (tags/v3.12.3:f6650f9, Apr 9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print('\n'.join(sys.path))

C:\Users\ciber\AppData\Local\Programs\Python\Pytho n312\python312.zip
C:\PROGRA~1\QGIS33~1.6\apps\Python312\DLLs
C:\PROGRA~1\QGIS33~1.6\apps\Python312\Lib
C:\Users\ciber\AppData\Local\Programs\Python\Pytho n312
C:\Users\ciber\AppData\Roaming\Python\Python312\si te-packages
C:\PROGRA~1\QGIS33~1.6\apps\Python312
C:\PROGRA~1\QGIS33~1.6\apps\Python312\Lib\site-packages
C:\PROGRA~1\QGIS33~1.6\apps\Python312\Lib\site-packages\win32
C:\PROGRA~1\QGIS33~1.6\apps\Python312\Lib\site-packages\win32\lib
C:\PROGRA~1\QGIS33~1.6\apps\Python312\Lib\site-packages\Pythonwin
>>>

[Tom Servo]

Okay, that seems reasonable I think. Running in that same OSGeo4W.bat command prompt, run "pip install pysal" and hopefully that'll get you there.

[FaultyMario]

OK. I did and when I called up the python prompt in QGIS and imported numpy, scipy and pysal I got no error messages.

So, if i understand this correctly, i should now be able to edit a script in the little python box with anything on this lib (https://pypi.org/project/pysal/#description), right?

[Tom Servo]

You should! If you didn't get an error importing pysal, that means that it's installed and findable!

[Tom Servo]

Everything good?

[FaultyMario]

not sure.

i ran a quick analysis with a package that depends on pysal and it went well and then i had to take a break. but i should give it a go later today.

one thing that worries me is that i can't seem to access [libpysal] from the process tools panel in qgis.

Yes, it's 4 AM my time as i write this. Yes, these are my working hours.

But yeah, thanks Brian. you will definitely hear more from me on this topic.