packamon contains tools that can be used for package
quality control.
The name is a portmanteau of package and
monitoring: meaning that if there are problems with your
package, packamon will help you catch ’em all.
Most of the included tools center around setting up a continuous integration pipeline on Jenkins which will run automated checks on every change to your package repositories (hence the monitoring aspect). For portability and reproducibility, everything is run and built in docker containers.
After following this vignette you will have a pipeline that, on every commit:
R CMD checkThe Docker- and Jenkinsfile are the bread and butter of modern CI
with Jenkins. This section will explain how to automatically generate
them with packamon.
Note: the Jenkinsfile for use with the OA Jenkins and
scm.openanalytics.eu is actually called JenkinsfileSCM, but
the syntax is still that of a regular Jenkinsfile. These two terms will
be used interchangebly, just remember that the file should always be
called JenkinsfileSCM.
Typically, you will start with your own local clone of a git repository. But for this example, we will use an artificial test directory created from some packages pulled from CRAN:
Now you can use the init() function. For more
flexibility, you can also use the functions
writeJenkinsfile() and writeDockerfile
instead.
## RDepot package submission has been generated with a default branch policy.
## Modify it by editing the Jenkinsfile.## Warning in findTestPkg(file.path(repositoryPath, path), name): No testthat.R
## file and no inst/tinytest folder found for package data.table. Stage 'Test and
## coverage' will be skipped## Jenkinsfile has been written to : /tmp/RtmpNPYBSM/sourceDirbb59e05a4a/JenkinsfileSCM## Dockerfile template has been written to: /tmp/RtmpNPYBSM/sourceDirbb59e05a4a/template.Dockerfile## Using latest versions of the dependent packages## Dockerfile has been written to :/tmp/RtmpNPYBSM/sourceDirbb59e05a4a/DockerfileNow you have a Dockerfile and a Jenkinsfile that can be used on Jenkins to create a continuous integration pipeline. You can find example content of these files at the end of this section.
Make sure to commit all generated files and push. The first build should start automatically soon™ (now may be a good time to get some coffee).
You can use the search bar in the Jenkins interface to enter the name of your repository to find builds. From now on, every time you push commits a new build should start. Refer to the OA Jenkins and Docker docs to read more about this subject and the function of the Jenkinsfile and Dockerfile.
If the build was successful first time: great! (This is highly unusual.)
If something went wrong keep reading: you may find some clues in the next sections.
# Generated by packamon: do not edit by hand
# Instead of modifying this file, you can modify a template. See ?init for details.
FROM registry.openanalytics.eu/openanalytics/r-ver-tools:4.4.3
# System libraries (incl. system requirements for R packages)
RUN apt-get update && apt-get install --no-install-recommends -y \
libcurl4-openssl-dev \
libicu-dev \
libssl-dev \
libxml2-dev \
make \
pandoc \
&& rm -rf /var/lib/apt/lists/*
RUN echo \
'options(repos = c(CRAN = gsub("/([0-9-]*)$", "/latest", Sys.getenv("CRAN", "https://cloud.r-project.org")), CRAN.source = "https://cloud.r-project.org"))' \
>> $(R RHOME)/etc/Rprofile.site
# install dependencies
RUN R -q -e "options(warn = 2); \
install.packages('remotes'); \
remotes::install_version('bit', version = '4.6.0', upgrade = FALSE); \
remotes::install_version('brio', version = '1.1.5', upgrade = FALSE); \
remotes::install_version('cli', version = '3.6.5', upgrade = FALSE); \
remotes::install_version('commonmark', version = '1.9.5', upgrade = FALSE); \
remotes::install_version('crayon', version = '1.5.3', upgrade = FALSE); \
remotes::install_version('digest', version = '0.6.37', upgrade = FALSE); \
remotes::install_version('evaluate', version = '1.0.3', upgrade = FALSE); \
remotes::install_version('fs', version = '1.6.6', upgrade = FALSE); \
remotes::install_version('glue', version = '1.8.0', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('jsonlite', version = '2.0.0', upgrade = FALSE); \
remotes::install_version('lattice', version = '0.22-7', upgrade = FALSE); \
remotes::install_version('magrittr', version = '2.0.3', upgrade = FALSE); \
remotes::install_version('ps', version = '1.9.1', upgrade = FALSE); \
remotes::install_version('R.methodsS3', version = '1.8.2', upgrade = FALSE); \
remotes::install_version('R6', version = '2.6.1', upgrade = FALSE); \
remotes::install_version('rlang', version = '1.1.6', upgrade = FALSE); \
remotes::install_version('rprojroot', version = '2.0.4', upgrade = FALSE); \
remotes::install_version('withr', version = '3.0.2', upgrade = FALSE); \
remotes::install_version('xfun', version = '0.52', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('yaml', version = '2.3.10', upgrade = FALSE); \
remotes::install_version('bit64', version = '4.6.0-1', upgrade = FALSE); \
remotes::install_version('desc', version = '1.4.3', upgrade = FALSE); \
remotes::install_version('diffobj', version = '0.3.6', upgrade = FALSE); \
remotes::install_version('highr', version = '0.11', upgrade = FALSE); \
remotes::install_version('lifecycle', version = '1.0.4', upgrade = FALSE); \
remotes::install_version('litedown', version = '0.7', upgrade = FALSE); \
remotes::install_version('processx', version = '3.8.6', upgrade = FALSE); \
remotes::install_version('R.oo', version = '1.27.1', upgrade = FALSE); \
remotes::install_version('zoo', version = '1.8-14', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('callr', version = '3.7.6', upgrade = FALSE); \
remotes::install_version('knitr', version = '1.50', upgrade = FALSE); \
remotes::install_version('markdown', version = '2.0', upgrade = FALSE); \
remotes::install_version('R.utils', version = '2.13.0', upgrade = FALSE); \
remotes::install_version('waldo', version = '0.6.1', upgrade = FALSE); \
remotes::install_version('xts', version = '0.14.1', upgrade = FALSE); \
remotes::install_version('pkgbuild', version = '1.4.7', upgrade = FALSE); \
remotes::install_version('pkgload', version = '1.4.0', upgrade = FALSE); \
remotes::install_version('testthat', version = '3.2.3', upgrade = FALSE); \
if (!requireNamespace('roxygen2', quietly = TRUE)) install.packages('roxygen2')"
RUN R -q -e "options(warn = 2); \
if (!requireNamespace('covr', quietly = TRUE)) install.packages('covr')"
# install tar gz package(s)
RUN mkdir -p /tmp
COPY praise_1.0.0.tar.gz /tmp/praise_1.0.0.tar.gz
RUN R -q -e "install.packages('/tmp/praise_1.0.0.tar.gz', repos = NULL, dependencies = FALSE)"pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: '3'))
}
environment {
IMAGE = 'sourcedirbb59e05a4a'
NS = 'shared'
REGISTRY = 'registry.openanalytics.eu'
TAG = env.BRANCH_NAME.toLowerCase().replaceAll(/[^a-z0-9._-]/, '.').trim()
REGION = 'eu-west-1'
NOT_CRAN = 'true'
_R_CHECK_TESTS_NLINES_ = 0
}
stages {
stage('Build Image') {
agent {
kubernetes {
inheritFrom 'kaniko'
}
}
steps {
container('kaniko') {
sh """/kaniko/executor \
-v info \
--log-timestamp=true \
--context ${env.WORKSPACE} \
--cache=true \
--cache-ttl=8760h0m0s \
--cache-repo ${env.REGISTRY}/${env.NS}/${env.IMAGE} \
--cleanup \
--destination ${env.REGISTRY}/${env.NS}/${env.IMAGE}:${env.TAG} \
--registry-mirror ${env.REGISTRY}"""
}
}
post {
always {
sh "cp /kaniko/jenkins/mem*.log ${env.WORKSPACE}"
archiveArtifacts artifacts: 'mem*.log', fingerprint: true
}
}
}
stage('Packages') {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: r
image: ${env.REGISTRY}/${env.NS}/${env.IMAGE}:${env.TAG}
resources:
limits:
cpu: 250m
memory: 1Gi
requests:
cpu: 250m
memory: 250Mi"""
inheritFrom 'r'
yamlMergeStrategy merge()
defaultContainer 'r'
}
}
stages {
stage('data.table') {
stages {
stage('Rcpp Compile Attributes') {
steps {
sh 'R -q -e \'if (requireNamespace("Rcpp", quietly = TRUE)) Rcpp::compileAttributes("data.table")\''
}
}
stage('Roxygen') {
steps {
sh 'R -q -e \'roxygen2::roxygenize("data.table")\''
}
}
stage('Build') {
steps {
sh 'R CMD build data.table'
}
}
stage('Check') {
steps {
script() {
switch(sh(script: 'ls data.table_*.tar.gz && R CMD check data.table_*.tar.gz --no-manual', returnStatus: true)) {
case 0: currentBuild.result = 'SUCCESS'
default: currentBuild.result = 'FAILURE'; error('script exited with failure status')
}
}
}
}
stage('Install') {
steps {
sh 'R -q -e \'install.packages(list.files(".", "data.table_.*.tar.gz"), repos = NULL)\''
}
}
}
}
stage('Archive artifacts') {
steps {
archiveArtifacts artifacts: '*.tar.gz, *.pdf, **/test-results.txt', fingerprint: true
}
}
stage('RDepot') {
when {
//packamon info: specify when you want your package to be submitted
//see https://www.jenkins.io/doc/book/pipeline/syntax/#built-in-conditions
anyOf {
branch 'main'
branch 'master'
}
}
environment {
RDEPOT_TOKEN = credentials('jenkins-rdepot-token')
RDEPOT_HOST = 'https://rdepot.openanalytics.eu/server'
}
steps {
sh 'rm -f pseudo.package.*tar.gz'
container('rdepot-cli') {
sh '''rdepot packages submit \
-f *.tar.gz \
--replace=false \
--repo internal'''
}
}
}
}
post {
always {
sh "cp /mem_r.log ${env.WORKSPACE}"
archiveArtifacts artifacts: '**/00check.log, mem*.log', fingerprint: true, allowEmptyArchive: true
}
}
}
}
}You may have noticed that init() also created an extra
file template.Dockerfile (unless you specified
writeTemplate = FALSE in which case you can use
writeDefaultDockerfileTemplate to still create it).
As the name suggests, this is a template which packamon
uses to construct the Dockerfile. Let us inspect the
content:
#include packamon.disclaimer
#include packamon.from
#include packamon.system-dependencies
#include packamon.r-repos
#include packamon.r-dependencies
#include packamon.local-r-dependencies
#include packamon.runtime-settingsThis may look a bit arcane, but the good news is that this template is itself a valid Dockerfile. The lines of the form
are comments which will be replaced in the final Dockerfile by
packamon::writeDockerfile().
Since this is just a normal Dockerfile you can do anything which can normally do. For example if you wanted to replace the base image you could delete the following line:
and replace it with:
In general you can add any docker command or comments at any line and it will be copied over to the generated Dockerfile.
The template is perhaps most useful to install extra
dependencies which packamon does not detect
automatically. The big advantage of editing the template
instead of the Dockerfile directly (and why it tells you not to at the
top) is that you can simply run writeDockerfile() when your
package or local environment has changed while keeping the inputs for
the extra dependencies stored safely in a template.
These are the available #include directives:
#include packamon.disclaimer: inserts a note that
the generated file should not be adapted directly
#include packamon.from: inserts a base
image
#include packamon.system-dependencies: inserts an
apt command to install system dependencies that could be inferred from
the R package dependencies
#include packamon.r-repos: insert a command to add
extra repos to the Rprofile.site file
#include packamon.r-dependencies: insert R install
commands for the remote R package dependencies
#include packamon.local-r-dependencies: insert R
install commands for the R package dependencies that are included in the
repository as a source folder or as a package tar.gz archive.
#include packamon.runtime-settings: insert commands
to prepare the container image for runtime execution of R
OpenAnalytics offers private and public repositories for serving packages that can be treated as a local CRAN. These repositories are managed by RDepot. You can read more about the use of RDepot at OpenAnalytics here.
To have your package submitted to RDepot, call
writeJenkinsfile with
submitPackagesToRDepot = c("develop", "master").
This will generate a stage that uses the rdepot cli to
submit the package.
By default, packages will be stored in the private repository. You can change this by editing the Jenkinsfile.