Repositories and Sources#

Repositories are the primary vehicle for organizing and distributing packages. These are views of the data that your client (R, BiocManager, pip, etc.) uses to find and install packages. For more information on packages and repositories see the Package Ecosystem section.

In RStudio Package Manager, repositories include one or more sources. A source is a collection of package files from one place. For example, CRAN and PyPI are places where a source may draw files from.

The documentation in this chapter outlines the types and structure of repositories and sources.

Repository types#

RSPM supports repositories of three types:

• R - A CRAN-like repository for R packages. This is the default repository type.
• Bioconductor - A repository for Bioconductor R packages that supports use by BiocManager.
• Python - A repository that provides the Simple Repository API specified by PEP 503. Pip and other tools that draw from Python Package Index (PyPI) use this interface.

Repository types may only include specific types of sources. For example, an R repository cannot include a PyPI source.

Repository Structure#

A repository appears to the end user as just a set of files served from disk. RStudio Package Manager does not create repositories on disk. Instead, RStudio Package Manager maintains a single copy of each source and binary package, and uses a database and specialized web server to handle HTTP requests from R and Python. Although a repository may include many sources, it appears as one collection of packages to the client. Some example requests that can be served by the RStudio Package Manager are in the appendix.

Most importantly, these request types provide the interfaces that client software expects for installing packages. A CRAN-like repository means users can access and install packages using their regular R functions: install.packages, available.packages, renv::install, and devtools::install. A Bioconductor repository allows installing packages with BiocManager::install. A Python repository allows using pip, pipenv, poetry, or other clients for the PEP 503 interface that PyPI provides.

Repository Versioning#

Repository versioning provides a way to reference several sources in one repository at particular snapshot points for each source. This is a useful feature for reproducibility: you can obtain a view of the packages that were available on some past date and ensure that you have the same packages that you used when you ran a particular analysis. Software updates may change interfaces or other behavior, so controlling your environment can be critically important to keeping you code working. Versioning is available for all repository types except Bioconductor, since Bioconductor includes its own versioning scheme.

For repositories that support it, the web UI provides a simple interface to choose a particular date to freeze to. A guide to this UI is in the Getting Started section.

Sources#

RStudio Package Manager repositories include one or more sources. Sources are collections of files from a single location (e.g. CRAN or PyPI). RStudio Package Manager tracks every change to files in a source and groups each set of changed files as a snapshot. Together, these snapshots create a full versioned history of each source. There are currently seven types of sources:

• cran source - A single cran source is automatically created. This source contains metadata and packages from RStudio's CRAN service. The source can be used directly in a repository to give users access to all CRAN packages, or it can used indirectly by curated-cran sources.
• pypi source - A single PyPI source is automatically created. This source contains metadata and packages from RStudio's PyPI service. The source can be used directly in a Python repository to give users access to all PyPI packages.
• bioconductor source - Bioconductor sources combine one or more internal sources (such as "bioc", "data/annotation", etc.) into a "meta-source" to represent a given bioconductor version. These meta-sources behave similarly to cran sources. See the Bioconductor Source section for more information.
• curated-cran source - A curated CRAN source exposes an approved set of CRAN packages. Administrators can add packages to the set and update the set. See the Curated CRAN Source section for more information.
• cran-snapshot source - A CRAN source that is pinned to a specific CRAN snapshot. Administrators can periodically update the snapshot to which the cran-snapshot source is pinned. See the CRAN Snapshot Source section for more information.
• local source - A local source is used as a mechanism to distribute locally developed packages or other packages without native support in RStudio Package Manager. Administrators add packages to local sources by specifying a path to a package's tar file.
• git source - A git source allows RStudio Package Manager to automatically make packages in Git available to R users through install.packages (without requiring devtools). Git sources work for internal packages as well as external sites such as GitHub. Packages can be automatically updated on each commit or when a new Git tag is pushed.

Snapshot Identifiers#

RSPM has two ways to specify a snapshot associated with a repository:

• Transaction ID: the internal numeric identifier for the action of importing the snapshot information. All sources share the same table, so transaction IDs are never duplicated between sources.
• Date alias: A date in YYYY-MM-DD form, which obtains the snapshot that occurred latest on the specified date

All source types support specifying snapshots using transaction IDs, but not all sources support date aliases. To date, the CRAN and PyPI sources are the only ones that currently support date aliases.

Other source types do not currently support date aliases because of the difficulty in making date information relevant across multiple RSPM installations. For example, the date associated with curated CRAN and CRAN snapshot sources is the date when the source was created or updated using the rspm update command. For Git sources, the date is when a Git package was built and updated, not the date of the commit or release that was built. For local sources, that will be a date when a package was added, updated, or removed. All of these are dates/times that are highly dependent on each local installation.

These identifiers are very important when considering reproducibility of referenced content. A transaction ID is highly unlikely to mean the same thing across two different RSPM installations (say staging and production). A date alias will generally be robust, and facilitates handling snapshot references across different RSPM installations. The date aliases are the default behavior where possible, but you can choose to freeze to transaction IDs instead. Please see the user interface overview for more information.

Repositories with Multiple Sources#

A repository can have more than one source. If you wish to serve both local packages and CRAN packages from a single repository, you can create a single repository that subscribes to multiple sources. For example:

• all (a repository)
• internal (local source)
• cran (CRAN source)

The "all" repository above gives users access to both local and CRAN packages, and its PACKAGES list could be accessed, for example, at https://packagemanager.rstudio.com/all/latest/src/contrib/PACKAGES. A repository subscribes to sources, which means that changes to a source will be reflected in the repository. For example, if an administrator adds a new package to the internal source, users will automatically be able to access the new package via the all repository.

Package Conflicts Between Sources#

If a repository has multiple sources and a package with the same name exists in both sources, RStudio Package Manager eliminates duplicates, giving preference in the order the sources are subscribed. In the example repository above, if a package named "plumber" exists in both the "cran" and "internal" sources, the "plumber" package from the "internal" source would be served and listed since it is the first source for the repository. The same conflict resolution occurs as sources change. For example, in the sample above, even if a new package is added to CRAN with the same name as an internal package, the internal package will continue to be served. The precedence is also maintained during updates. In the example above, the internal version of plumber will continue to be served even if the CRAN version of plumber is updated. The order of sources within a repository can be re-arranged using the reorder command.

What is RStudio's Package Service?#

RStudio Package Manager doesn't download packages directly from CRAN, Bioconductor, or PyPI. Instead, RStudio maintains a curated S3 bucket that contains metadata about CRAN, Bioconductor, and PyPI, in addition to package tar files. The metadata is used to track day-to-day changes.

See the Air-Gapped RStudio Package Manager section if your environment does not have access to the RStudio Package Service

During a sync, the metadata is downloaded to RStudio Package Manager. The metadata is compared against the RStudio Package Manager database to determine what changes need to be applied. Package tarballs are then downloaded to the cache on demand.

See the Package Security section for details about the security measures that are in place for the RStudio Package Service.

Publishing Snapshots to the RStudio Package Service#

The RSPM team evaluates CRAN, Bioconductor, and PyPI each business day and publish new snapshots when updates are available. Any RSPM installations then sync these snapshots based on their configured schedules. For example, suppose a CRAN package gets updated on Saturday. We will publish a new snapshot to the RStudio Package Service sometime on Monday, usually Monday afternoon (roughly America Central Time). Then, the user's default CRAN sync will pick up the change on Tuesday at 12:00am UTC. Occasionally there are very important updates, and we generate an extra snapshot to make the updates available sooner. The timing of each snapshot varies based on the number of changes and the number of dependencies involved.

To be sure RStudio Package Manager synchronizes new snapshots as soon as possible, consider setting your sync schedule to occur more than once per day.

Package Fetching#

RStudio Package Manager fetches packages on-demand as they are requested by end users. Package Manager will still download the CRAN, Bioconductor, and PyPI metadata on the sync schedule to keep the RStudio Package Manager database updated. The database serves as the source of truth for package availability. The benefit of on-demand fetching is a smaller footprint in terms of network bandwidth and disk space.

Package Caching#

RStudio Package Manager downloads each version of a package only once, and always checks the local cache to see if the required tar file is already available.

Synchronizing with the RStudio Package Service#

The CRAN, Bioconductor, and PyPI sources are synchronized according to a schedule set using the SyncSchedule properties in the RStudio Package Manager configuration file. These properties accept a string in crontab format. See the appropriate appendix section below.

• CRAN Configuration
• Bioconductor Configuration
• PyPI Configuration

By default, the configuration file includes crontabs that will cause RStudio Package Manager to sync once a day (early morning in the server's timezone), if any of the following conditions have been met:

• For CRAN:
  • Any repository subscribes to the cran source.
  • A "curated-cran" source is used by any repo.
  • A manual sync has been run with the sync --type=cran command.
• For Bioconductor:
  • Any repository subscribes to a Bioconductor source.
  • A Bioconductor repo has been created.
• For PyPI:
  • Any repository subscribes to the pypi source.
  • A manual sync has been run with the sync --type=pypi command.

A sync schedule will not be applied if the above conditions are not met. If you only want manual syncs, change the configuration file to have a blank value for SyncSchedule:

;/etc/rstudio-pm/rstudio-pm.gcfg
[CRAN]
SyncSchedule = ""

[Bioconductor]
SyncSchedule = ""

[PyPI]
SyncSchedule = ""

The SyncSchedule property does not necessarily determine when a repository will make updated packages available to users. For example, if a repository subscribes directly to the cran source, users will see updates according to the sync schedule. In contrast, if the repository subscribes to a curated CRAN source, an administrator must explicitly update the source in order for updates to become available.

In addition, updating the repository does not automatically push updated packages to R clients. A repository specifies what packages are available, but the R user is in control of when and how to update the packages used by a project.

See the section on Managing Change Control for more information.

RStudio Package Manager keeps track of old versions of packages as well. Old versions of packages are available in the repository's archive, and are listed in the RStudio Package Manager web UI. This allows users to roll back updates if necessary or install packages as they existed at a prior time.