Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

It seems to me to be a very basic question, but I did not find a straightforward answer to it. I have two (in fact more) PCs where I work in conda environments. At some point I want to perform a major update, e.g. of Python version. Since conda indicates many conflicts, I did not know how to deal with, I could create a new environment with the desired Python version. But how should I know what packages shall I install in that new environment? In the 'old' one, I had:

So, when I create a new environment, how should I remember which packages I had installed in the old one, thus I should also install them in the new one? conda env export or conda list are not super helpful, as they display all packages (including their current versions and perhaps packages which are no longer valid with the new Python version) and I'd like to have filtered out only those that I myself had installed, so I would know they should also be present in the new environment (and the dependencies would be installed automagically). Same question applies to pip.

Conda has a specific term for environment constraints provided by user: explicit specifications . This would correspond to your first group of packages (requested by the user to be installed). One can use the --from-history flag to specify that the exported environment should only include explicit specs:

conda env export --from-history
Underneath it all, it processes the $CONDA_PREFIX/conda-meta/history file (which has the history of all env-mutating Conda commands) to distill out what constraints have been explicitly requested. It should also be noted that certain actions, such as using the --update-deps flag, can trigger Conda to add every package in the env to the explict specifications. Not sure of a clean way out of this; probably have to process the history file yourself to check for conda install commands.
None of this, however, will not capture things installed with pip.
                Thank you, that is helpful, although it does not solve the problem. E.g. the history file you mention does not contain, in my case, a single install nor create command. Greping it against the name of my environment does not find any hits either, thus I suspect this history file is not complete. I only know more less what is said here: docs.conda.io/projects/conda/en/latest/user-guide/tasks/… do you recommend any more exhaustive docs?
– Maciek
                Sep 16, 2020 at 7:39
                @Maciek each env has its own history file. Are you looking at the one in the env of interest? Or perhaps did you create the env with conda env create -f env.yaml? Did you try the export from history command? The history file is more of a last resort, i.e. looking at internals.
– merv
                Sep 16, 2020 at 21:03
                Lol, good point, no I went directly (and exactly) to $CONDA_PREFIX/conda-meta/history not going to a specific environment in the envs directory. There, indeed, there seems to be a valid history. Thanks!
– Maciek
                Sep 17, 2020 at 10:29
+++ The verbose pip show command should show a line with the Installer. For example:
$ python -m pip show --verbose setuptools
Name: setuptools
Version: 49.6.0
[...]
Installer: pip
[...]
+++ The verbose pip list command should list all distributions active in the current environment and their installer. For example:
$ python -m pip list --verbose
$ .venv/bin/python -m pip list --verbose
Package       Version Location                                  Installer
------------- ------- ----------------------------------------- ---------
pip           20.2.3  /path/to/venv/lib/python3.8/site-packages pip
setuptools    49.6.0  /path/to/venv/lib/python3.8/site-packages pip
+++ For every distribution installed there should be a INSTALLER file. For example:
$ cat /path/to/venv/lib/python3.8/site-packages/setuptools-49.6.0.dist-info/INSTALLER
+++ A short example code showing how to list all distributions in the current environment alongside their installer:
#!/usr/bin/env python3
import importlib.metadata
def main():
    for distribution in importlib.metadata.distributions():
        installer = None
        for file_path in distribution.files:
            if file_path.name == 'INSTALLER':
                installer = file_path.read_text().strip()
                break
        print(f"{distribution.metadata['Name']} -- {installer}")
if __name__ == '__main__':
    main()
                But that would show info about setuptools, how about listing all? It seems to me pip list --user  would do. How about a corresponding command in conda?
– Maciek
                Sep 16, 2020 at 7:30
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.