packamon contains tools that can be used for package quality control.
The name is a portmanteau of package and monitoring: meaning that if there are problems with your package, packamon will help you catch ’em all.
Most of the included tools center around setting up a continuous integration pipeline on Jenkins which will run automated checks on every change to your package repositories (hence the monitoring aspect). For portability and reproducibility, everything is run and built in docker containers.
After following this vignette you will have a pipeline that, on every commit:
R CMD check# install.packages("packamon", repos = c(rdepot = "https://repos.openanalytics.eu/repo/public"))
# install.packages("~/git/packamon/packamon/", repos=NULL)
library(packamon)The Docker- and Jenkinsfile are the bread and butter of modern CI with Jenkins. This section will explain how to automatically generate them with packamon.
Typically, you will start with your own local clone of a git repository. But for this example, we will use an artificial test directory created from some packages pulled from CRAN:
testDir <- createTestDirectory()Now you can use the init() function. For more flexibility, you can also use the functions writeJenkinsfile() and writeDockerfile instead.
init(testDir, overwrite = TRUE)## RDepot package submission has been generated with a default branch policy.
## Modify it by editing the Jenkinsfile.## Warning in findTestPkg(file.path(repositoryPath, path), name): No testthat.R
## file and no inst/tinytest folder found for package data.table. Stage 'Test and
## coverage' will be skipped## Jenkinsfile has been written to : /tmp/RtmpSew29Q/sourceDir98436c1320/Jenkinsfile## Dockerfile template has been written to: /tmp/RtmpSew29Q/sourceDir98436c1320/template.Dockerfile## Using latest versions of the dependent packages## Dockerfile has been written to :/tmp/RtmpSew29Q/sourceDir98436c1320/DockerfileNow you have a Dockerfile and a Jenkinsfile that can be used on Jenkins to create a continuous integration pipeline. You can find example content of these files at the end of this section.
Make sure to commit all generated files and push. The first build should start automatically soon™ (now may be a good time to get some coffee).
You can use the search bar in the Jenkins interface to enter the name of your repository to find builds. From now on, every time you push commits a new build should start. Refer to the OA Jenkins and Docker docs to read more about this subject and the function of the Jenkinsfile and Dockerfile.
If the build was successful first time: great! (This is highly unusual.)
If something went wrong keep reading: you may find some clues in the next sections.
Dockerfile
# Generated by packamon: do not edit by hand
# Instead of modifying this file, you can modify a template. See ?init for details.
FROM registry.openanalytics.eu/openanalytics/r-ver:4.3.3
# System libraries (incl. system requirements for R packages)
RUN apt-get update && apt-get install --no-install-recommends -y \
libcurl4-openssl-dev \
libicu-dev \
libssl-dev \
libxml2-dev \
make \
pandoc \
&& rm -rf /var/lib/apt/lists/*
RUN echo \
'options(repos = c(CRAN = gsub("/([0-9-]*)$", "/latest", Sys.getenv("CRAN", "https://cloud.r-project.org")), CRAN.source = "https://cloud.r-project.org"))' \
>> $(R RHOME)/etc/Rprofile.site
# install dependencies
RUN R -q -e "options(warn = 2); \
install.packages('remotes'); \
remotes::install_version('bit', version = '4.0.5', upgrade = FALSE); \
remotes::install_version('brew', version = '1.0-10', upgrade = FALSE); \
remotes::install_version('brio', version = '1.1.5', upgrade = FALSE); \
remotes::install_version('cli', version = '3.6.2', upgrade = FALSE); \
remotes::install_version('commonmark', version = '1.9.1', upgrade = FALSE); \
remotes::install_version('cpp11', version = '0.4.7', upgrade = FALSE); \
remotes::install_version('crayon', version = '1.5.2', upgrade = FALSE); \
remotes::install_version('curl', version = '5.2.1', upgrade = FALSE); \
remotes::install_version('digest', version = '0.6.35', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('evaluate', version = '0.23', upgrade = FALSE); \
remotes::install_version('fansi', version = '1.0.6', upgrade = FALSE); \
remotes::install_version('fs', version = '1.6.4', upgrade = FALSE); \
remotes::install_version('glue', version = '1.7.0', upgrade = FALSE); \
remotes::install_version('jsonlite', version = '1.8.8', upgrade = FALSE); \
remotes::install_version('lattice', version = '0.22-6', upgrade = FALSE); \
remotes::install_version('lazyeval', version = '0.2.2', upgrade = FALSE); \
remotes::install_version('magrittr', version = '2.0.3', upgrade = FALSE); \
remotes::install_version('mime', version = '0.12', upgrade = FALSE); \
remotes::install_version('pkgconfig', version = '2.0.3', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('ps', version = '1.7.6', upgrade = FALSE); \
remotes::install_version('R.methodsS3', version = '1.8.2', upgrade = FALSE); \
remotes::install_version('R6', version = '2.5.1', upgrade = FALSE); \
remotes::install_version('rlang', version = '1.1.3', upgrade = FALSE); \
remotes::install_version('rprojroot', version = '2.0.4', upgrade = FALSE); \
remotes::install_version('stringi', version = '1.8.4', upgrade = FALSE); \
remotes::install_version('sys', version = '3.4.2', upgrade = FALSE); \
remotes::install_version('utf8', version = '1.2.4', upgrade = FALSE); \
remotes::install_version('withr', version = '3.0.0', upgrade = FALSE); \
remotes::install_version('xfun', version = '0.44', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('yaml', version = '2.3.8', upgrade = FALSE); \
remotes::install_version('askpass', version = '1.2.0', upgrade = FALSE); \
remotes::install_version('bit64', version = '4.0.5', upgrade = FALSE); \
remotes::install_version('desc', version = '1.4.3', upgrade = FALSE); \
remotes::install_version('diffobj', version = '0.3.5', upgrade = FALSE); \
remotes::install_version('highr', version = '0.10', upgrade = FALSE); \
remotes::install_version('lifecycle', version = '1.0.4', upgrade = FALSE); \
remotes::install_version('markdown', version = '1.12', upgrade = FALSE); \
remotes::install_version('processx', version = '3.8.4', upgrade = FALSE); \
remotes::install_version('R.oo', version = '1.26.0', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('rex', version = '1.2.1', upgrade = FALSE); \
remotes::install_version('xml2', version = '1.3.6', upgrade = FALSE); \
remotes::install_version('zoo', version = '1.8-12', upgrade = FALSE); \
remotes::install_version('callr', version = '3.7.6', upgrade = FALSE); \
remotes::install_version('knitr', version = '1.46', upgrade = FALSE); \
remotes::install_version('openssl', version = '2.2.0', upgrade = FALSE); \
remotes::install_version('R.utils', version = '2.12.3', upgrade = FALSE); \
remotes::install_version('vctrs', version = '0.6.5', upgrade = FALSE); \
remotes::install_version('xts', version = '0.13.2', upgrade = FALSE); \
remotes::install_version('httr', version = '1.4.7', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('pillar', version = '1.9.0', upgrade = FALSE); \
remotes::install_version('pkgbuild', version = '1.4.4', upgrade = FALSE); \
remotes::install_version('purrr', version = '1.0.2', upgrade = FALSE); \
remotes::install_version('stringr', version = '1.5.1', upgrade = FALSE); \
remotes::install_version('covr', version = '3.6.4', upgrade = FALSE); \
remotes::install_version('pkgload', version = '1.3.4', upgrade = FALSE); \
remotes::install_version('tibble', version = '3.2.1', upgrade = FALSE); \
remotes::install_version('rematch2', version = '2.1.2', upgrade = FALSE); \
remotes::install_version('roxygen2', version = '7.3.1', upgrade = FALSE); \
remotes::install_version('waldo', version = '0.5.2', upgrade = FALSE)"
RUN R -q -e "options(warn = 2); \
remotes::install_version('testthat', version = '3.2.1.1', upgrade = FALSE)"
# install tar gz package(s)
RUN mkdir -p /tmp
WORKDIR /tmp
COPY praise_1.0.0.tar.gz /tmp/praise_1.0.0.tar.gz
RUN R -q -e "install.packages('/tmp/praise_1.0.0.tar.gz', repos = NULL, dependencies = FALSE)"Jenkinsfile
pipeline {
agent any
options {
buildDiscarder(logRotator(numToKeepStr: '3'))
}
environment {
IMAGE = 'sourcedir98436c1320'
NS = 'shared'
REGISTRY = 'registry.openanalytics.eu'
TAG = sh(returnStdout: true, script: "echo $BRANCH_NAME | sed -e 's/[A-Z]/\\L&/g' -e 's/[^a-z0-9._-]/./g'").trim()
REGION = 'eu-west-1'
NOT_CRAN = 'true'
_R_CHECK_TESTS_NLINES_ = 0
}
stages {
stage('Build Image') {
agent {
kubernetes {
yaml '''
apiVersion: v1
kind: Pod
spec:
imagePullSecrets:
- name: registry-robot
volumes:
- name: kaniko-dockerconfig
secret:
secretName: registry-robot
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:v1.21.1-debug
env:
- name: AWS_SDK_LOAD_CONFIG
value: "true"
command:
- /kaniko/docker-credential-ecr-login
- get
tty: true
resources:
requests:
memory: "1024Mi"
limits:
memory: "4096Mi"
ephemeral-storage: "4Gi"
imagePullPolicy: IfNotPresent
volumeMounts:
- name: kaniko-dockerconfig
mountPath: /kaniko/.docker/config.json
subPath: .dockerconfigjson
'''
defaultContainer 'kaniko'
}
}
steps {
container('kaniko') {
sh """/kaniko/executor \
-v info \
--context ${env.WORKSPACE} \
--cache=true \
--cache-ttl=8760h0m0s \
--cache-repo ${env.REGISTRY}/${env.NS}/${env.IMAGE} \
--cleanup \
--destination ${env.REGISTRY}/${env.NS}/${env.IMAGE}:${env.TAG} \
--registry-mirror ${env.REGISTRY}"""
}
}
}
stage('Packages') {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
spec:
imagePullSecrets:
- name: registry-robot
containers:
- name: r
image: ${env.REGISTRY}/${env.NS}/${env.IMAGE}:${env.TAG}
command:
- cat
tty: true
imagePullPolicy: Always
- name: rdepot-cli
command:
- cat
tty: yes
image: ${env.REGISTRY}/openanalytics/rdepot-cli:latest"""
defaultContainer 'r'
}
}
stages {
stage('data.table') {
stages {
stage('Rcpp Compile Attributes') {
steps {
sh 'R -q -e \'if (requireNamespace("Rcpp", quietly = TRUE)) Rcpp::compileAttributes("data.table")\''
}
}
stage('Roxygen') {
steps {
sh 'R -q -e \'roxygen2::roxygenize("data.table")\''
}
}
stage('Build') {
steps {
sh 'R CMD build data.table'
}
}
stage('Check') {
steps {
script() {
switch(sh(script: 'ls data.table_*.tar.gz && R CMD check data.table_*.tar.gz --no-manual', returnStatus: true)) {
case 0: currentBuild.result = 'SUCCESS'
default: currentBuild.result = 'FAILURE'; error('script exited with failure status')
}
}
}
}
stage('Install') {
steps {
sh 'R -q -e \'install.packages(list.files(".", "data.table_.*.tar.gz"), repos = NULL)\''
}
}
}
}
stage('Archive artifacts') {
steps {
archiveArtifacts artifacts: '*.tar.gz, *.pdf, **/00check.log, test-results.txt', fingerprint: true
}
}
stage('RDepot') {
when {
//packamon info: specify when you want your package to be submitted
//see https://www.jenkins.io/doc/book/pipeline/syntax/#built-in-conditions
anyOf {
branch 'develop'
branch 'master'
}
}
environment {
RDEPOT_TOKEN = credentials('jenkins-rdepot-token')
RDEPOT_HOST = 'https://rdepot.openanalytics.eu'
}
steps {
container('rdepot-cli') {
sh '''rdepot packages submit \
-f *.tar.gz \
--replace false \
--repo internal'''
}
}
}
}
}
}
}
You may have noticed that init() also created an extra file template.Dockerfile (unless you specified writeTemplate = FALSE in which case you can use writeDefaultDockerfileTemplate to still create it).
As the name suggests, this is a template which packamon uses to construct the Dockerfile. Let us inspect the content:
#include packamon.disclaimer
#include packamon.from
#include packamon.system-dependencies
#include packamon.r-repos
#include packamon.r-dependencies
#include packamon.local-r-dependencies
#include packamon.runtime-settingsThis may look a bit arcane, but the good news is that this template is itself a valid Dockerfile. The lines of the form
#include packamon.<step>are comments which will be replaced in the final Dockerfile by packamon::writeDockerfile().
Since this is just a normal Dockerfile you can do anything which can normally do. For example if you wanted to replace the base image you could delete the following line:
#include packamon.fromand replace it with:
FROM openanalytics/my-super-cool-image:latestIn general you can add any docker command or comments at any line and it will be copied over to the generated Dockerfile.
The template is perhaps most useful to install extra dependencies which packamon does not detect automatically. The big advantage of editing the template instead of the Dockerfile directly (and why it tells you not to at the top) is that you can simply run writeDockerfile() when your package or local environment has changed while keeping the inputs for the extra dependencies stored safely in a template.
These are the available #include directives:
#include packamon.disclaimer: inserts a note that the generated file should not be adapted directly
#include packamon.from: inserts a base image
#include packamon.system-dependencies: inserts an apt command to install system dependencies that could be inferred from the R package dependencies
#include packamon.r-repos: insert a command to add extra repos to the Rprofile.site file
#include packamon.r-dependencies: insert R install commands for the remote R package dependencies
#include packamon.local-r-dependencies: insert R install commands for the R package dependencies that are included in the repository as a source folder or as a package tar.gz archive.
#include packamon.runtime-settings: insert commands to prepare the container image for runtime execution of R
OpenAnalytics offers private and public repositories for serving packages that can be treated as a local CRAN. These repositories are managed by RDepot. You can read more about the use of RDepot at OpenAnalytics here.
To have your package submitted to RDepot, call writeJenkinsfile with submitPackagesToRDepot = c("develop", "master").
This will generate a stage that uses the rdepot cli to submit the package.
By default, packages will be stored in the private repository. You can change this by editing the Jenkinsfile.