Remote Publishing

Advanced

Package Manager supports adding packages to local Python or R repositories remotely. This feature allows easier integration with separate CI/CD package build pipelines or adding packages directly from a development environment.

Remote Token Authentication

Local sources support remote token authentication. You can use remote authentication to upload packages via the rspm add command in the CLI or, additionally, Python packages also support twine.

An API token needs to be generated to allow a user to upload packages to the source. To create the token, run the command on the server:

Terminal
$ rspm create token --description="Local Python token" --sources=local-python-src --expires=30d --scope=sources:write
Generated an access token. Be sure to record this token immediately since you will not be able to retrieve it later.
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk

Uploading Packages Remotely with the CLI

On the remote machine, export the correct environment variables and download the CLI:

Terminal
$ export PACKAGEMANAGER_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk
$ export PACKAGEMANAGER_ADDRESS=<https://my-rspm-server.org:4443>

# Download the CLI
$ curl -fLOJH "Authorization: Bearer ${PACKAGEMANAGER_TOKEN}" "${PACKAGEMANAGER_ADDRESS}/__api__/download"
$ chmod +x rspm
Terminal
$ export PACKAGEMANAGER_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk
$ export PACKAGEMANAGER_ADDRESS=<https://my-rspm-server.org:4443>

# Download the CLI
$ curl -fLOJH "Authorization: Bearer ${PACKAGEMANAGER_TOKEN}" "${PACKAGEMANAGER_ADDRESS}/__api__/download?os=darwin"
$ chmod +x rspm
Powershell
$Env:PACKAGEMANAGER_TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk"
$Env:PACKAGEMANAGER_ADDRESS = "https://my-rspm-server.org:4443"

# Download the CLI
$urlCLI = "$Env:PACKAGEMANAGER_ADDRESS/__api__/download?os=windows"
$outputCLI = "rspm.exe"
$wc = New-Object System.Net.WebClient
$wc.Headers['Authorization'] = "Bearer $Env:PACKAGEMANAGER_TOKEN"
$wc.DownloadFile($urlCLI, $outputCLI)

# Alternatively, on Windows 10 (v1803+), you can download the CLI with curl
curl -OutFile rspm.exe -Headers @{ Authorization = "Bearer $Env:PACKAGEMANAGER_TOKEN"} "$Env:PACKAGEMANAGER_ADDRESS/__api__/download?os=windows"

After the environment variables are set, the remote machine can use CLI commands that support remote use:

Terminal
./rspm add --source=local-python-src --path=/path/to/packages
Powershell
.\rspm.exe add --source=local-python-src --path=C:\path\to\packages

Uploading Python Packages Remotely with Twine

Twine is a tool that can be used to upload Python packages that have been built locally. To learn more about its full feature set, review the Twine documentation.

Note

Other Python upload tools like Poetry and Flit should work since the Package Manager API attempts to maintain compatibility with the PyPI Warehouse upload endpoint. However, Twine is the officially recommended and documented method of uploading Python packages to Package Manager.

To start, install Twine on the system that will be uploading packages:

Terminal
pip install twine

Twine Configuration

There are several ways that Twine can be used to remotely authenticate with Package Manager:

  • Using the TWINE_* environment variables.
  • Configuring a .pypirc file.
  • Authenticating with keyring.

Using the TWINE_* environment variables

The three environment variables Twine uses are:

  • TWINE_REPOSITORY_URL
  • TWINE_USERNAME
  • TWINE_PASSWORD

The TWINE_REPOSITORY_URL environment variable is the address that is used to upload to Package Manager remotely, ending with the endpoint /upload/pypi/{source-name}. Since we are using token authentication, TWINE_USERNAME will always be set to __token__. Finally, TWINE_PASSWORD is set to the token that was generated above. To put this all together, the following is an example of what the variables could be set to:

Terminal
$ export TWINE_REPOSITORY_URL=https://packagemanager.posit.co/upload/pypi/local-python-src
$ export TWINE_USERNAME=__token__
$ export TWINE_PASSWORD=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk

With the environment variables set, you can easily upload with Twine:

Terminal
$ twine upload dist/*

Configuring a .pypirc file

An alternative approach to using the Twine environment variables is to configure a .pypirc file. Twine looks for this file in the ~/.pypirc location by default. If instead a user wants to pass in a .pypirc file that is not in that location, they can use Twine’s --config-file flag to point to that file in a different directory.

The .pypirc looks like the following:

~/.pypirc
[distutils]
index-servers =
   package-manager

[package-manager]
repository = https://packagemanager.posit.co/upload/pypi/local-python-src
username = __token__
password = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImV4cCI6MTY1ODU4MjA3OCwianRpIjoiYmM5ZTg1NGYtNGNlNy00Zjc4LTlhMmMtZDliYzRlYTQ0NGVkIiwiaWF0IjoxNjU1OTkwMDc4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjQyNDIvIiwic2NvcGVzIjp7InNvdXJjZXMiOiIzYTI4NjFhYi0xNWYwLTRjM2MtODZlMy0xNjNkMTY0ZDE0ZDYifX0.BWJXLworo44Nvfrh5a2hm_NIqgUoXTLjQlxyy7uaSWk

Then to use this file to upload with Twine:

Terminal
twine upload -r package-manager dist/*

The .pypirc file makes it easy to upload to various sources quickly. When you create another repository definition in the .pypirc file, it is made available to use with Twine’s -r flag.

Authenticating with keyring

An alternative approach to putting the username/password directly into a .pypirc file or environment variables is to store them securely in a keyring. Keyring is automatically installed with Twine, so it is available to use out-of-the-box.

Note

From keyring: macOS keychain support for macOS 11 (Big Sur) and later requires Python 3.8.7 or later with the “universal2” binary.

If a user is using the .pypirc method, their file would now look like:

~/.pypirc
[distutils]
index-servers =
   package-manager

[package-manager]
repository = https://packagemanager.posit.co/upload/pypi/local-python-src

If a user is instead using the Twine environment variables, now all they need to set is:

Terminal
export TWINE_REPOSITORY_URL=https://packagemanager.posit.co/upload/pypi/local-python-src

Then, to store the username and password for the repository, a user can save them with keyring:

Terminal
$ keyring set https://packagemanager.posit.co/upload/pypi/local-python-src __token__
Password for '__token__' in 'https://packagemanager.posit.co/upload/pypi/local-python-src':

The password has been saved securely within the keyring and can be used by Twine. When uploading with Twine, it will now prompt for the username and check keyring to see if it has a password. If it does, it will successfully upload the distributions:

Terminal
$ cd directory/of/internal/example_package
$ twine upload -r package-manager dist/*
Enter your username: __token__
Uploading distributions to https://packagemanager.posit.co/upload/pypi/local-python-src
Uploading example_package.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 MB • 00:02 • 3.5 MB/s

Twine Package Signing

Twine and Package Manager also support uploading signed packages. Package Manager detects when a signed file is uploaded and stores the file with the Python package. A user can see this by navigating to the package in the UI and clicking on the specific distribution. If the distribution has a green SIGNED label, then it successfully uploaded the .asc file with the Python distribution. To download this file for future use, a user can append .asc to the URL for that specific distribution to download the signed file.

There are two methods for uploading signed files:

  1. Manually uploading pre-signed files.
  2. Automatically signing with Twine.

Manually uploading pre-signed files

If you already have a package pre-signed, the Twine command is:

Terminal
twine upload example_package.whl example_package.whl.asc
Note

Simply uploading with twine upload dist/* will automatically upload any .asc files that exist for the packages in that directory.

Automatically signing with Twine

There is also the --sign flag that instructs Twine to sign the files before uploading:

Terminal
$ twine upload --sign example_package.whl
Uploading distributions to https://packagemanager.posit.co/upload/pypi/local-python-src
Signing example_package.whl
Uploading example_package.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 MB • 00:02 • 3.5 MB/s
Back to top