TDPyEnvManagerHelper

From Derivative
Jump to navigation Jump to search

These Extensions reference a specific Palette:tdPyEnvManager.


TDPyEnvManagerHelper[edit]

A class to manage the installation and setup of Python and Conda environments for TouchDesigner.

This class does not require TouchDesigner to be running and can be used as a standalone Python script.

It gets imported as a module in TouchDesigner in the context of the TDPyEnvManager to be used as a helper class within the TDPyEnvManager.

It provides methods to download and install Miniconda, create Conda environments, and manage Python virtual environments. It also includes methods for logging, downloading files, creating .gitignore files, creating .condarc files, and verifying installations as well as other tools and utilities.

The file can be found in your TouchDesigner installation folder, %TD_INSTALL_PATH%\\bin\\Lib\\tdutils\\TDPyEnvManagerHelper.py where %TD_INSTALL_PATH% is the path to your TouchDesigner root install folder. (or $YOUR_TOUCHDESIGNER_INSTALL_PATH/Contents/Frameworks/Python.framework/Versions/$TD_PYENVMANAGER_PYTHONVERSION/lib/python$TD_PYENVMANAGER_PYTHONVERSION/site-packages/tdutils/TDPyEnvManagerHelper.py on MacOS, where $YOUR_TOUCHDESIGNER_INSTALL_PATH is the path to your TouchDesigner.app and $TD_PYENVMANAGER_PYTHONVERSION is the current TouchDesigner build Python version, such as 3.11.)

The TDPyEnvManagerHelper is initialized during TouchDesigner startup. If a TDPyEnvManagerContext.yaml file is found and the environment registered in the context file is valid, then the environment will be added to the Python search path before any TouchDesigner COMP cooks and any custom extensions initialized. Legacy TDPyEnvManagerContext.json files are converted to YAML automatically unless pyproject.toml already contains [tool.touchdesigner.TDPyEnvManagerContext]. As of TouchDesigner build 2025.32280, context data can also be stored in that pyproject.toml section, which is then used as the preferred context store for both reads and writes.

It's exposed as the Helper attribute of the TDPyEnvManager, or on app.pyEnvHelper.

When debugging, using the environment variable TOUCH_APP_LOG_LEVEL and setting a log level lower than WARNING can be useful (INFO, DEBUG...). (log level table for reference)

Using the TDPyEnvManagerHelper as a CLI[edit]

The TDPyEnvManagerHelper can be used as a standalone CLI tool. Below is a guide on how to use it, including the arguments that can be passed, their example values, and what they do.

Run the script from the command line: python TDPyEnvManagerHelper.py [OPTIONS]

Where python is an alias pointing to your TouchDesigner's Python:

  • at YOUR_TOUCHDESIGNER_INSTALL_PATH\bin\python.exe on Windows
  • at YOUR_TOUCHDESIGNER_INSTALL_PATH/Contents/Frameworks/Python.framework/bin/python3.11 on MacOS
Caption text
Argument Description Example Value Default Value
--mode Specifies the mode of environment setup. Can be Conda Env or Python vEnv. --mode "Conda Env" Python vEnv
--installPath The path where Miniconda or Python virtual environment should be created. --installPath "C:\Envs" Current working directory
--envName The name of the Conda or Python virtual environment to be created. --envName "MyEnv" TDPyEnv
--pythonVersion The version of Python to be installed in the environment. --pythonVersion "3.11" Current Python version
--useReqs Python vEnv mode only. Install requirements after creating or validating the vEnv. --useReqs False
--requirementsFiles Python vEnv mode only. Optional list of requirements files to use with --useReqs. --requirementsFiles base.txt dev.txt requirements.txt
--useEnvYml Conda mode only. Create the environment from environment.yml when present. --useEnvYml False
--clean Cleans the install directory before installing. --clean False
--keepInstaller Keeps the installer after installation or if installation fails. --keepInstaller False

Argument Details[edit]

--mode[edit]

Description: Determines whether to create a Conda environment (Conda Env) or a Python virtual environment (Python vEnv).

Example:python TDPyEnvManagerHelper.py --mode "Conda Env"

Default: Python vEnv

--installPath[edit]

Description: Specifies the directory where the environment will be installed.

Example:python TDPyEnvManagerHelper.py --installPath "C:\MyEnvironments"

Default: Current working directory.

--envName[edit]

Description: The name of the environment to be created.

Example:python TDPyEnvManagerHelper.py --envName "MyCustomEnv"

Default: TDPyEnv

--pythonVersion[edit]

Description: Specifies the Python version to be installed in the environment. It should always match TouchDesigner's own Python version.

Example:python TDPyEnvManagerHelper.py --pythonVersion "3.11"

Default: The current Python version of the system.

--useReqs[edit]

Description: In Python vEnv mode, explicitly install requirements after the vEnv is created or when an existing vEnv is reused.

Example:python TDPyEnvManagerHelper.py --mode "Python vEnv" --useReqs

Default: False

--requirementsFiles[edit]

Description: In Python vEnv mode, provide one or more custom requirements files to install when --useReqs is passed. If omitted, the helper falls back to requirements.txt in the current working directory.

Example:python TDPyEnvManagerHelper.py --mode "Python vEnv" --useReqs --requirementsFiles requirements/base.txt requirements/dev.txt

Default: requirements.txt

--useEnvYml[edit]

Description: In Conda mode, create the environment from environment.yml instead of creating a bare environment and then adding packages manually.

Example:python TDPyEnvManagerHelper.py --mode "Conda Env" --useEnvYml

Default: False

--clean[edit]

Description: If specified, the install directory will be cleaned before installation. Example:python TDPyEnvManagerHelper.py --clean Default: False

--keepInstaller[edit]

Description: If specified, the installer will not be deleted after installation.

Example:python TDPyEnvManagerHelper.py --keepInstaller

Default: False

Examples[edit]

1. Create a Python Virtual Environment[edit]

python TDPyEnvManagerHelper.py --mode "Python vEnv" --installPath "C:\Envs" --envName "MyPythonEnv" --pythonVersion "3.11"

Creates a Python virtual environment named MyPythonEnv in the C:\Envs directory with Python version 3.11.

2. Create a Conda Environment[edit]

python TDPyEnvManagerHelper.py --mode "Conda Env" --installPath "C:\Conda" --envName "MyCondaEnv" --pythonVersion "3.11" --clean

Creates a Conda environment named MyCondaEnv in the C:\Conda directory with Python version 3.11. The C:\Conda directory will be cleaned before installation.

3. Keep the Installer After Installation[edit]

python TDPyEnvManagerHelper.py --mode "Conda Env" --keepInstaller

Creates a Conda environment and keeps the installer file after installation.

4. Create or Reuse a Python vEnv and Install Multiple Requirements Files[edit]

python TDPyEnvManagerHelper.py --mode "Python vEnv" --installPath "C:\Envs" --envName "MyPythonEnv" --useReqs --requirementsFiles requirements\base.txt requirements\gpu.txt

Creates the vEnv if needed, then installs packages from the listed requirements files in order.

5. Create a Conda Environment From environment.yml[edit]

python TDPyEnvManagerHelper.py --mode "Conda Env" --installPath "C:\Conda" --useEnvYml

Creates the Conda environment using the current folder's environment.yml.

Notes[edit]

  • The script no longer prompts for clean or keepInstaller confirmations. Those behaviors are controlled explicitly through CLI flags.
  • Ensure that the installPath directory is writable and has sufficient space for the installation.
  • For Conda environments, the script will automatically download and install Miniconda if it is not already installed.
  • In Python vEnv mode, requirements installation is refused when the resolved Python executable is not actually inside a valid vEnv. This protects against accidentally installing into TouchDesigner or a system interpreter.
  • This guide should help you get started with using TDPyEnvManagerHelper as a CLI tool.

Example CLI usages can be found in Samples/TDPyEnvManager folder. You will find the script and both .sh and .bat files to run the script using arguments to create a Conda or Python virtual environment.

Members[edit]

modestr or None :

The mode used to install an environment. Can be `Conda Env` or `Python vEnv`. Defaults to None.

envNamestr or None :

The name of the environment to be created for a Python vEnv or a Conda Env. Defaults to None.

installPathpathlib.Path or None :

The path to the Miniconda installation, or the Python Virtual Environment. Defaults to None.

pythonVersionstr :

Python version to be installed. Only useful for Miniconda environments. Python vEnv are created with TouchDesigner's own Python interpreter and the version is matching.

loggerlogging.Logger or None :

Python logger instance used by the helper.

taskQueuequeue.Queue :

Queue for tasks to run in the worker thread.

errorQueuequeue.Queue :

Queue storing worker errors.

onErrorcallable or None :

Optional callback invoked when worker tasks fail.

runningProcesssubprocess.Popen or None :

The currently running subprocess, if any, doing work in the helper.

envPathpathlib.Path or None :

Path to the Conda or Python virtual environment.

executablePathpathlib.Path or None :

Path to the Python executable inside the environment.

osPathlist[str] :

List of PATH entries managed by the helper.

sysPathlist[str] :

List of sys.path entries managed by the helper.

extraPathslist[str] :

Extra paths to add to sys.path after environment linking.

loadedFromContextbool :

Flag indicating whether a context file was loaded.

initLoadedFromContextbool :

Flag tracking first-load-from-context state.

Readybool :

Flag indicating the environment is ready to use.

autoSetupbool :

When True, environment setup happens automatically from context data.

autoSetupReqslist[str] :

List of requirements files to install during auto-setup.

autoSetupSyncReqsbool :

When True, auto-setup installs requirements even when the target environment already exists.

contextVersionint :

Serialized context schema version written into YAML or pyproject.toml context data.

contextCreatedByVersionstr :

Version metadata indicating which TDPyEnvManager build last wrote the context.

threadthreading.Thread :

Worker thread that executes queued tasks.

Methods[edit]

TDPyEnvManagerHelper.__init__(mode: str or None = None, envName: str or None = None, installPath: pathlib.Path or None = None)None:

Initialize the TDPyEnvManagerHelper class.

Args:

  • mode (str or None): The mode used to install an environment. Can be `Conda Env` or `Python vEnv`.
  • envName (str or None): The environment name.
  • installPath (pathlib.Path or None): The path to the Miniconda installation or Python virtual environment.

TDPyEnvManagerHelper.postInit()None:

Initialize logger, start the worker thread, and load context data from pyproject.toml or legacy context files if present.

TDPyEnvManagerHelper.__del__()None:

Stop the worker thread when the instance is deleted.

TDPyEnvManagerHelper.setupLogger()logging.Logger: A logger instance to be used within the helper.:

Setup the logger for the helper.

TDPyEnvManagerHelper.worker()None:

Worker thread that processes queued tasks and subprocess calls.

TDPyEnvManagerHelper.reportWorkerError(task, exc)None:

Log and enqueue worker errors without stopping the worker loop.

TDPyEnvManagerHelper.checkForContextFile(configFile: pathlib.Path)bool: True if the configuration file exists, False otherwise.:

Check if the specified configuration file exists.

TDPyEnvManagerHelper.downloadFile(url: str, destPath: pathlib.Path, timeout: int = 30, skip_if_exists: bool = False)None:

Download a file from a URL to the specified destination path.

TDPyEnvManagerHelper.createGitIgnore(path: pathlib.Path, skip_if_exists: bool = True)None:

Create a .gitignore file in the specified path.

TDPyEnvManagerHelper.createCondaRc(path: pathlib.Path, skip_if_exists: bool = True)None:

Create a .condarc file in the specified path.

TDPyEnvManagerHelper.verifyConda(installPath: pathlib.Path)bool: True if Conda is installed, False otherwise.:

Verify if Conda is installed in the specified path.

TDPyEnvManagerHelper.verifyCondaLib(installPath: pathlib.Path)bool: True if the Conda library is accessible, False otherwise.:

Verify if the Conda library is accessible in the specified path.

TDPyEnvManagerHelper.downloadConda()pathlib.Path: The path to the downloaded Miniconda installer.:

Download the Miniconda installer for the current platform.

TDPyEnvManagerHelper.cleanDirectory(path: pathlib.Path)None:

Clean the specified directory by removing it and its contents.

TDPyEnvManagerHelper.installConda(installPath: pathlib.Path, keepInstaller: bool = False)None:

Install Miniconda using the downloaded installer.

TDPyEnvManagerHelper.cleanupFile(filePath: pathlib.Path)None:

Remove the specified file.

TDPyEnvManagerHelper.createCondaEnv(installPath: pathlib.Path, envName: str, pythonVersion: str, useEnv: bool = False)None:

Create a Conda environment using the specified parameters.

TDPyEnvManagerHelper.verifyCondaEnv(condaInstallPath: str, envName: str)bool: True if the Conda environment exists, False otherwise.:

Verify if the Conda environment exists in the specified path.

TDPyEnvManagerHelper.activateCondaEnv(installPath: pathlib.Path, envName: str)None:

Open a CLI and activate the specified Conda environment.

TDPyEnvManagerHelper.setCondaRoot(installPath: pathlib.Path)pathlib.Path: The path where Conda is installed.:

Set the root directory of the Conda installation.

TDPyEnvManagerHelper.getCondaRoot()pathlib.Path: The path where Conda is installed.:

Get the root directory of the Conda installation.

TDPyEnvManagerHelper.getCondaEnvPath(envName: str)pathlib.Path: The path to the Conda environment.:

Get the path to the Conda environment.

TDPyEnvManagerHelper.exportCondaEnvYaml(envPath: pathlib.Path)None:

Export the Conda environment to a YAML file.

TDPyEnvManagerHelper.verifyPython(installPath: pathlib.Path)bool: True if Python is installed, False otherwise.:

Verify if Python is installed at the specified path.

TDPyEnvManagerHelper.createPythonEnv(installPath: str, envName: str, useReq: bool = False)None:

Create a Python virtual environment using the specified name.

TDPyEnvManagerHelper.activatePythonEnv(envName: str)None:

Open a CLI and activate the specified Python virtual environment.

TDPyEnvManagerHelper.setPythonRoot(rootPath: pathlib.Path)pathlib.Path: The path where Python is installed.:

Set the root directory of the Python installation.

TDPyEnvManagerHelper.getPythonEnvPath(envName: str)pathlib.Path: The path to the Python virtual environment.:

Get the path to the Python virtual environment.

TDPyEnvManagerHelper.getPythonSitePackagesPath(envName: str)pathlib.Path: The path to the site-packages directory of the Python virtual environment.:

Get the path to the site-packages directory of the Python virtual environment.

TDPyEnvManagerHelper.exportPyEnvRequirements(envPath: pathlib.Path)None:

Export the requirements of the Python virtual environment to a requirements.txt file.

TDPyEnvManagerHelper.addToOsPath(path: pathlib.Path)None:

Add a path to os.environ['PATH'] if it is not already present.

TDPyEnvManagerHelper.addToSysPath(path: pathlib.Path)None:

Add a path to sys.path if it is not already present.

TDPyEnvManagerHelper.addPathsToEnvironment(condaEnvPath: pathlib.Path, libFolder: str, pythonFolder: str = None)None:

Add Conda environment paths to os.environ['PATH'] and sys.path.

TDPyEnvManagerHelper.resolveExtraPath(rawPath: str) -> pathlib.Path or Nonepathlib.Path or None:

Resolve extra paths relative to the current working directory and expand environment variables.

TDPyEnvManagerHelper.applyExtraPaths()None:

Apply extra paths to sys.path after environment linking.

TDPyEnvManagerHelper.linkEnv(envPath: pathlib.Path)bool: True if successful, False otherwise.:

Link the current session to the specified environment based on the active mode.

TDPyEnvManagerHelper.LinkPyVenv(envPath: pathlib.Path)bool: True if successful, False otherwise.:

Link a Python virtual environment to the current session.

TDPyEnvManagerHelper.LinkConda(installPath: pathlib.Path)bool: True if successful, False otherwise.:

Link a Conda installation to the current session.

TDPyEnvManagerHelper.LinkCondaEnv(envPath: pathlib.Path = None)bool: True if successful, False otherwise.:

Link a Conda environment to the current session.

TDPyEnvManagerHelper.unlinkEnv(mode: str = None)bool: True if successful, False otherwise.:

Unlink the current environment based on the given or active mode.

TDPyEnvManagerHelper.UnlinkCondaEnv()bool: True if successful, False otherwise.:

Unlink the current Conda environment from the session.

TDPyEnvManagerHelper.UnlinkConda()bool: True if successful, False otherwise.:

Unlink the current Conda installation from the session.

TDPyEnvManagerHelper.UnlinkPyVenv()bool: True if successful, False otherwise.:

Unlink the current Python virtual environment from the session.

TDPyEnvManagerHelper.isCondaInstallPathInOSPath(condaInstallPath: pathlib.Path)bool:

Check whether the conda install path is already in os.environ['PATH'].

TDPyEnvManagerHelper.isCondaEnvPathInSysPath(condaEnvPath: pathlib.Path)bool:

Check whether the conda environment path is already in sys.path.

TDPyEnvManagerHelper.validatePath(path: pathlib.Path)bool: True if the path is valid, False otherwise.:

Validate if the given path exists and is writable.

TDPyEnvManagerHelper.validateEnvName(envName: str)str: Cleaned and valid environment name.:

Validate and sanitize an environment name.

TDPyEnvManagerHelper.normalizeReqList(value: str or list[str] or tuple[str, ...] or None)list[str]:

Normalize a requirements list input to a list of strings. On non-Windows platforms, Windows-style separators are normalized so shared context files remain portable.

TDPyEnvManagerHelper.AsContext()dict:

Return a dictionary representation of the current helper state, including contextVersion, createdByVersion, and autoSetupSyncReqs metadata.

TDPyEnvManagerHelper.applyContextDict(context: dict or None)None:

Apply a context dictionary to the helper instance, including schema/version metadata and auto-setup sync settings.

TDPyEnvManagerHelper.loadContextFromPyproject(pyprojectPath: pathlib.Path)dict or None:

Load context data from [tool.touchdesigner.TDPyEnvManagerContext] in pyproject.toml if present.

TDPyEnvManagerHelper.pyprojectHasTouchDesignerContextSection(pyprojectPath: pathlib.Path)bool:

Check whether pyproject.toml contains a [tool.touchdesigner.TDPyEnvManagerContext] section.

TDPyEnvManagerHelper.toTomlScalar(value: object)str:

Serialize a scalar Python value for TOML output.

TDPyEnvManagerHelper.toTomlValue(value: object)str:

Serialize a scalar or list value for TOML output.

TDPyEnvManagerHelper.WriteContextToPyproject(pyprojectPath: pathlib.Path)bool:

Write context data into an existing [tool.touchdesigner.TDPyEnvManagerContext] section in pyproject.toml.

TDPyEnvManagerHelper.WriteContextToFile(filePath: pathlib.Path)None:

Write the current context to YAML, or update pyproject.toml instead when a TouchDesigner context section already exists there.

TDPyEnvManagerHelper.migrateContextJsonToYaml(jsonPath: pathlib.Path, yamlPath: pathlib.Path)None:

Convert a legacy JSON context file to YAML.

TDPyEnvManagerHelper.ReadContextFromFile(filePath: pathlib.Path)None:

Read context data from YAML or legacy JSON and apply it. Legacy JSON may be migrated to YAML first.

TDPyEnvManagerHelper.stopWorker()None:

Stop the worker thread by sending a shutdown signal.

TDPyEnvManagerHelper.enqueueTask(task, blocking: bool = False)None:

Enqueue a task for the worker thread and optionally wait for completion.

TDPyEnvManagerHelper.appendVEnvSuffix(envName: str)str:

Append '_vEnv' to the environment name if it is not already present.

TDPyEnvManagerHelper.getHighestPyVer(condaPath: str)str or None:

Get the highest Python version folder name in the specified conda path.

TDPyEnvManagerHelper.getAutoSetupReqFiles()list[pathlib.Path]:

Return a normalized list of requirements files for auto-setup.

TDPyEnvManagerHelper.installRequirementsForPython(pythonPath: pathlib.Path, reqFiles: list[pathlib.Path])None:

Install requirements files using the specified Python executable. In Python vEnv mode, the helper refuses to run if that interpreter is not part of a valid vEnv.

TDPyEnvManagerHelper.autoSetupEnv()None:

Auto-create an environment and install requirements when autoSetup is enabled. If autoSetupSyncReqs is True, requirements also run when the target environment already exists.

TouchDesigner Build: