Publish Modules
Nextflow module that fires once per dataset after both the NSForest and scsilhouette branches have completed.
It clones NIH-NLM/cell-kn, copies outputs into the
data/prod/{organ}/ tree, commits, pushes a new branch, and
opens a pull request against main.
Warning
params.github_token must be a GitHub personal access token
with repo write access. Never hardcode it — pass via
-params-file params.json or an environment variable.
Generate S3 Manifest Process
generate_s3_manifest_process
Source: modules/publish/generate_s3_manifest.nf
Generate S3 Manifest
Runs once across all datasets. Collects all published output files and writes master_s3_manifest.csv listing each filename and its permanent S3 path in the CloudOS results bucket.
Input:
Output:
Params referenced:
params.outdirparams.publish_mode
Publish Results Process
publish_results_process
Source: modules/publish/publish_results.nf
Publish Results to cell-kn GitHub Repository
Fires once per dataset. Reads files from S3 results path derived from workflow.workDir (works on CloudOS and other platforms).
Branch naming:
{YYYY}-{mon}-{DD}}-{session id [-6..-1]}-${organSlug}-sc_nsforest_qc_nf one branch per organ e.g. 2026-mar-06-abcde-kidney-sc_nsforest_qc_nf
Platform notes:
- CloudOS: publishDir files land at s3://…/jobs/{id}/results/{label}/
derived by replacing /work with /results in workflow.workDir
Params referenced:
params.github_tokenparams.organparams.publish_dest_dirparams.publish_repo