diff --git a/_episodes/01-introduction.md b/_episodes/01-introduction.md index 94e9a1f..de1b81e 100644 --- a/_episodes/01-introduction.md +++ b/_episodes/01-introduction.md @@ -12,6 +12,8 @@ keypoints: - This tutorial is brought to you by the DUNE Computing Consortium. - The goals are to give you the computing basis to work on DUNE. --- + +{% include 01-introduction.toc.md %} ## DUNE Computing Consortium The DUNE Computing Consortium works to establish a global computing network that will handle the massive data streams produced by distributing these across the computing grid. diff --git a/_episodes/02-storage-spaces.md b/_episodes/02-storage-spaces.md index a2b4b25..29a252d 100644 --- a/_episodes/02-storage-spaces.md +++ b/_episodes/02-storage-spaces.md @@ -17,6 +17,7 @@ keypoints: - The tool suites idfh and XRootD allow for accessing data with appropriate transfer method and in a scalable way. --- +{% include 02-storage-spaces.toc.md %} ## This is an updated version of the 2023 training -**Locally mounted volumes** are physical disks, mounted directly on the computer +### Locally mounted volumes +local volumes are physical disks, mounted directly on the computer * physically inside the computer node you are remotely accessing * mounted on the machine through the motherboard (not over network) * used as temporary storage for infrastructure services (e.g. /var, /tmp,) @@ -90,17 +98,24 @@ Each has its own advantages and limitations, and knowing which one to use when i -**Network Attached Storage (NAS)** element behaves similar to a locally mounted volume. +### Network Attached Storage (NAS) +NAS elements behaves similar to a locally mounted volume. * functions similar to services such as Dropbox or OneDrive * fast and stable POSIX access to these volumes * volumes available only on a limited number of computers or servers * not available on grid computing (FermiGrid, Open Science Grid, WLCG, HPC, etc.) +#### At Fermilab * /exp/dune/app/users/.... has periodic snapshots in /exp/dune/app/..../.snap, but /exp/dune/data does NOT * easy to share files with colleagues using /exp/dune/data and /exp/dune/app +* See the [Ceph](https://fifewiki.fnal.gov/wiki/Ceph) documentation for details on those systems. +#### At CERN +At CERN the analog is EOS +See [EOS](https://cern.service-now.com/service-portal?id=kb_article&n=KB0001998) for information about using EOS - -## Grid-accessible storage volumes +### Grid-accessible storage volumes + +The following areas are grid accessible via methods such as `xrdcp/xrootd` and `ifdh`. You can read files in dCache across DUNE if you have the appropriate authorization. Writing files may require special permissions. - At Fermilab, an instance of dCache+CTA is used for large-scale, distributed storage with capacity for more than 100 PB of storage and O(10000) connections. - At CERN, the analog is EOS+CASTOR @@ -111,29 +126,83 @@ DUNE also maintains disk copies of most recent files across many sites worldwide Whenever possible, these storage elements should be accessed over xrootd (see next section) as the mount points on interactive nodes are slow, unstable, and can cause the node to become unusable. Here are the different dCache volumes: -**Persistent dCache**: the data in the file is actively available for reads at any time and will not be removed until manually deleted by user. The persistent dCache contains 3 logical areas: (1) /pnfs/dune/persistent/users in which every user has a quota up to 5TB total (2) /pnfs/dune/persistent/physicsgroups. This is dedicated for DUNE Physics groups and managed by the respective physics conveners of those physics groups. +#### Persistent dCache + `/pnfs/dune/persistent/` is "persistent" storage. If a file is in persistent dCache, the data in the file is actively available for reads at any time and will not be removed until manually deleted by user. The persistent dCache contains 3 logical areas: (1) /pnfs/dune/persistent/users in which every user has a quota up to 5TB total (2) /pnfs/dune/persistent/physicsgroups. This is dedicated for DUNE Physics groups and managed by the respective physics conveners of those physics groups. + https://wiki.dunescience.org/wiki/DUNE_Computing/Using_the_Physics_Groups_Persistent_Space_at_Fermilab gives more details on how to get access to these groups. In general, if you need to store more than 5TB in persistent dCache you should be working with the Physics Groups areas. (3) the "staging" area /pnfs/dune/persistent/staging which is not accessible by regular users but is by far the largest of the three. It is used for official datasets. + + -**Scratch dCache**: large volume shared across all experiments. When a new file is written to scratch space, old files are removed in order to make room for the newer file. Removal is based on Least Recently Utilized (LRU) policy, and performed by an automated daemon. -**Tape-backed dCache**: disk based storage areas that have their contents mirrored to permanent storage on CTA tape. +#### Scratch dCache +`/pfns/dune/scratch` is a large volume shared across all experiments. When a new file is written to scratch space, old files are removed in order to make room for the newer file. Removal is based on Least Recently Utilized (LRU) policy, and performed by an automated daemon. + + +#### Tape-backed dCache +Tape-backed disk based storage areas that have their contents mirrored to permanent storage on CTA tape. Files are not available for immediate read on disk, but needs to be 'staged' from tape first ([see video of a tape storage robot](https://www.youtube.com/watch?v=kiNWOhl00Ao)). -**Rucio Storage Elements**: Rucio Storage Elements (or RSEs) are storage elements provided by collaborating institution for official DUNE datasets. Data stored in DUNE RSE's must be fully cataloged in the [metacat][metacat] catalog and is managed by the DUNE data management team. This is where you find the official data samples. +#### Rucio Storage Elements + Rucio Storage Elements (or RSEs) are storage elements provided by collaborating institution for official DUNE datasets. Data stored in DUNE RSE's must be fully cataloged in the [metacat][metacat] catalog and is managed by the DUNE data management team. This is where you find the official data samples. + + See the [data management]({{ site.baseurl }}/03-data-management) lesson for much more information about using the `rucio` system to find official data. -**CVMFS**: CERN Virtual Machine File System is a centrally managed storage area that is distributed over the network, and utilized to distribute common software and a limited set of reference files. CVMFS is mounted over the network, and can be utilized on grid nodes, interactive nodes, and personal desktops/laptops. It is read only, and the most common source for centrally maintained versions of experiment software libraries/executables. CVMFS is mounted at `/cvmfs/` and access is POSIX-like, but read only. +### CVMFS +CVMFS is the CERN Virtual Machine File System is a centrally managed storage area that is distributed over the network, and utilized to distribute common software and a limited set of reference files. CVMFS is mounted over the network, and can be utilized on grid nodes, interactive nodes, and personal desktops/laptops. It is read only, and the most common source for centrally maintained versions of experiment software libraries/executables. CVMFS is mounted at `/cvmfs/` and access is POSIX-like, but read only. See [CVMFS]({{ site.baseurl }}/02.3-cvmfs) for more information. +### What is my quota? + +We use multiple systems so there are multiple ways for checking your disk quota. + +#### Your home area at FNAL + +~~~ +quota -u -m -s +~~~ +{: ..language-bash} + +#### Your home area at CERN +~~~ +fs listquota +~~~ +{: ..language-bash} + +#### The /app/ and /data/ areas at FNAL + +These use the Ceph file system which has directory quotas instead of user quotas. +See the quota section of: +[https://fifewiki.fnal.gov/wiki/Ceph#Quotas](https://fifewiki.fnal.gov/wiki/Ceph#Quotas) + +The most useful commands for general users are +~~~ +getfattr -n ceph.quota.max_bytes /exp/dune/app/users/$USER +getfattr -n ceph.quota.max_bytes /exp/dune/data/users/$USER +~~~ +{: ..language-bash} + +#### EOS at CERN + +~~~ +export EOS_MGM_URL=root://eosuser.cern.ch +eos quota +~~~ +{: ..language-bash} + +#### Fermilab dCache + +Go to [https://fndca.fnal.gov/cgi-bin/quota.py](https://fndca.fnal.gov/cgi-bin/quota.py) - you need to be on the Fermilab VPN - otherwise it sits there not loading. + > ## Note - When reading from dcache always use the root: syntax, not direct /pnfs > The Fermilab dcache areas have NFS mounts. These are for your convenience, they allow you to look at the directory structure and, for example, remove files. However, NFS access is slow, inconsistent, and can hang the machine if I/O heavy processes use it. Always use the `xroot root://` ... when reading/accessing files instead of `/pnfs/` directly. Once you have your [dune environment set up](software_setup) the `pnfs2xrootd` command can do the conversion to `root:` format for you (only for files at FNAL for now). {: .callout} @@ -187,7 +256,7 @@ This section will teach you the main tools and commands to display storage infor Another useful data handling command you will soon come across is ifdh. This stands for Intensity Frontier Data Handling. It is a tool suite that facilitates selecting the appropriate data transfer method from many possibilities while protecting shared resources from overload. You may see *ifdhc*, where *c* refers to *client*. > ## Note -> ifdh is much more efficient than NFS file access. Please use it and/or xrdcp when accessing remote files. +> `ifdh` is much more efficient than NFS file access. Please use it and/or `xrdcp/xrootd` when accessing remote files. {: .challenge} Here is an example to copy a file. Refer to the [Mission Setup]({{ site.baseurl }}/setup.html) for the setting up the `DUNELAR_VERSION`. @@ -196,42 +265,41 @@ Here is an example to copy a file. Refer to the [Mission Setup]({{ site.baseurl > For now do this in the Apptainer {: .challenge} -~~~ -/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \ --B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid \ -/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest -~~~ -{: .language-bash} +Do the standard [sl7 setup]({{ site.baseurl }}/setup_sl7) + -once in the Apptainer +once you are set up ~~~ -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -setup ifdhc -export IFDH_TOKEN_ENABLE=1 +export IFDH_TOKEN_ENABLE=1 # only need to do this once ifdh cp root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro/physics/full-reconstructed/2023/mc/out1/MC_Winter2023_RITM1592444_reReco/54/05/35/65/NNBarAtm_hA_BR_dune10kt_1x2x6_54053565_607_20220331T192335Z_gen_g4_detsim_reco_65751406_0_20230125T150414Z_reReco.root /dev/null ~~~ {: .language-bash} +This should go quickly as you are not actually writing the file. + Note, if the destination for an ifdh cp command is a directory instead of filename with full path, you have to add the "-D" option to the command line. -Prior to attempting the first exercise, please take a look at the full list of IFDH commands, to be able to complete the exercise. In particular, mkdir, cp, rmdir, +Prior to attempting the first exercise, please take a look at the full list of IFDH commands, to be able to complete the exercise. In particular, cp, rmdir, **Resource:** [ifdh commands](https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands) + > ## Exercise 1 -> Using the ifdh command, complete the following tasks: -> * create a directory in your dCache scratch area (/pnfs/dune/scratch/users/${USER}/) called "DUNE_tutorial_2025" +> use normal `mkdir` to create a directory in your dCache scratch area (/pnfs/dune/scratch/users/${USER}/) called "DUNE_tutorial_2025" +> Using the `ifdh command, complete the following tasks: > * copy /exp/dune/app/users/${USER}/my_first_login.txt file to that directory > * copy the my_first_login.txt file from your dCache scratch directory (i.e. DUNE_tutorial_2024) to /dev/null > * remove the directory DUNE_tutorial_2025 > * create the directory DUNE_tutorial_2025_data_file > Note, if the destination for an ifdh cp command is a directory instead of filename with full path, you have to add the "-D" option to the command line. Also, for a directory to be deleted, it must be empty. > +> Note `ifdh` no longer has a `mkdir` command as it auto-creates directories. In this example, we use the NFS command `mkdir` directly for clarity. +> > > ## Answer > > ~~~ -> > ifdh mkdir /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_2025 +> > mkdir /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_2025 > > ifdh cp -D /exp/dune/app/users/${USER}/my_first_login.txt /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_2025 > > ifdh cp /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_2025/my_first_login.txt /dev/null > > ifdh rm /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_2025/my_first_login.txt @@ -296,6 +364,10 @@ root://fndca1.fnal.gov:1094//pnfs/fnal.gov/usr/dune/tape_backed/dunepro/protodun ~~~ {: .output} +> ## Note - if you don't have pfns2xrootd on your system +> Copy [this]({{ side.baseurl}}/pnfs2xrootd) to your local area, make it executable and use it instead. +{: .callout} + you can then ~~~ @@ -314,6 +386,9 @@ export DUNELAR_QUALIFIER=e26:prof export UPS_OVERRIDE="-H Linux64bit+3.10-2.17" source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh setup dunesw $DUNELAR_VERSION -q $DUNELAR_QUALIFIER +setup justin # use justin to get appropriate tokens +justin time # this will ask you to authenticate via web browser +justin get-token # this actually gets you a token ~~~ {: .language-bash} diff --git a/_episodes/03-data-management.md b/_episodes/03-data-management.md index 7eafab6..be0533d 100644 --- a/_episodes/03-data-management.md +++ b/_episodes/03-data-management.md @@ -13,7 +13,9 @@ keypoints: - Xrootd allows user to stream data files. --- -#### Session Video +{% include 03-data-management.toc.md %} + +## Session Video @@ -77,7 +79,11 @@ If you want to process data using the full power of DUNE computing, you should t ## How to find and access official data -### What is metacat? +{% include OfficialDatasets_include.md %} + +You can also query the catalogs yourself using [metacat][metacat] and [rucio][rucio] catalogs. Metacat contains information about file content and official datasets, rucio stores the physical location of those files. Files should have entries in both catalogs. Generally you ask metacat first to find the files you want and then ask rucio for their location. + +## What is metacat? Metacat is a file and dataset catalog - it allows you to search for files and datasets that have particular attributes and understand their provenance, including details on all of their processing steps. It also allows for querying jointly the file catalog and the DUNE conditions database. @@ -94,9 +100,9 @@ DUNE runs multiple experiments (far detectors, protodune-sp, protodune-dp hd-pro To find your data you need to specify at the minimum -- `core.run_type` (the experiment) +- `core.run_type` (the experiment: fardet-vd, hd-protodune ...) - `core.file_type` (mc or detector) -- `core.data_tier` (the level of processing raw, full-reconstructed, root-tuple) +- `core.data_tier` (the level of processing raw, full-reconstructed, root-tuple ...) and when searching for specific types of data @@ -145,7 +151,8 @@ First get metacat if you have not already done so token authentication. {: .callout} --> -### then do queries to find particular sets of files +### then do queries to find particular groups of files + ~~~ metacat query "files from dune:all where core.file_type=detector and core.run_type=hd-protodune and core.data_tier=raw and core.runs[any]=27331 limit 1" @@ -240,10 +247,9 @@ Total size: 17553648200600 (17.554 TB) {: .output} - -## Official datasets + + +### find out how much data there is in a dataset + +Do a query using the `-s` or `--summary` option + +~~~ +metacat query -s "files from fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official" +~~~ +{: .language-bash} + +~~~ +Files: 20648 +Total size: 34550167782531 (34.550 TB) +~~~ +{: .output} +this may take a while as that is a big dataset. + ### What describes a dataset? -Let's look at the metadata describing that anti-neutrino dataset: the -j means json output +Let's look at the metadata describing an anti-neutrino dataset: the -j means json output ~~~ metacat dataset show -j fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official @@ -386,7 +410,7 @@ You can use any of those keys to refine dataset searches as we did above. You pr ### What files are in that dataset and how do I use them? -You can either click on a dataset in the web data catalog or: +You can either locate and click on a dataset in the [web data catalog](https://dune-tech.rice.edu/dunecatalog/) or use the[metacat web interface](https://metacat.fnal.gov:9443/dune_meta_prod/app/gui) or use the command line: ~~~ metacat query "files from fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official limit 10" @@ -398,7 +422,7 @@ will list the first 10 files in that dataset (you probably don't want to list al You can also use a similar query in your batch job to get the files you want. -### Finding those files on disk +## Finding those files on disk To find your files, you need to use [Rucio](#Rucio) directly or give the [justIN](https://dunejustin.fnal.gov/docs/tutorials.dune.md) batch system your query and it will locate them for you. @@ -417,7 +441,8 @@ export SAM_EXPERIMENT=dune --> ## Getting file locations using Rucio -### What is Rucio? +### What is Rucio? + Rucio is the next-generation Data Replica service and is part of DUNE's new Distributed Data Management (DDM) system that is currently in deployment. Rucio has two functions: 1. A rule-based system to get files to Rucio Storage Elements around the world and keep them there. @@ -427,7 +452,7 @@ As of the date of the 2025 tutorial: - The Rucio client is available in CVMFS and Spack - Most DUNE users are now enabled to use it. New users may not automatically be added. -### You will need to authenticate to use read files +### You will need to authenticate to read files > #### For SL7 use justin to get a token {:.callout} @@ -498,7 +523,7 @@ which the locations of the file on disk and tape. We can use this to copy the f > Try to access the file at manchester using the command: > ~~~ > root -l root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/fd/a6/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250217T033222Z_gen_004122_supernova_g4stage1_g4stage2_detsim_reco.root -> _file0->ls +> _file0->ls() > ~~~ > {: .language-bash} {: .challenge} diff --git a/_extras/OfficialDatasets.md b/_extras/OfficialDatasets.md new file mode 100644 index 0000000..558bc54 --- /dev/null +++ b/_extras/OfficialDatasets.md @@ -0,0 +1,6 @@ +--- +title: Official Datasets +permalink: OfficialDatasets +--- + +{% include OfficialDatasets_include.md %} \ No newline at end of file diff --git a/_extras/Tokens.md b/_extras/Tokens.md index d5bb0ae..59b8848 100644 --- a/_extras/Tokens.md +++ b/_extras/Tokens.md @@ -8,4 +8,6 @@ title: Tokens ## AL9 Tokens -{% include al9_token.md %} \ No newline at end of file +{% include al9_token.md %} + +Check your token `httogendecode` \ No newline at end of file diff --git a/_extras/pnfs2xrootd.md b/_extras/pnfs2xrootd.md index 436011c..61225de 100644 --- a/_extras/pnfs2xrootd.md +++ b/_extras/pnfs2xrootd.md @@ -12,7 +12,7 @@ permalink: pnfs2xrootd while true do -echo -n `readlink -f $1` | sed -e 's%/pnfs%root://fndca1.fnal.gov:1094//pnfs/fna +echo -n `readlink -f $1` | sed -e 's%/pnfs%root://fndcadoor.fnal.gov:1094//pnfs/fna l.gov/usr%' shift if [ x$1 == x ]; then break; fi diff --git a/_extras/sl7_setup.md b/_extras/sl7_setup.md index 5488b83..07ca2a0 100644 --- a/_extras/sl7_setup.md +++ b/_extras/sl7_setup.md @@ -30,10 +30,16 @@ Start the Apptainer {: .challenge} -## then do the following +## then do the following to set up DUNE code you can store this as `mysl7.sh` and run it every time you log in. {% include sl7_setup_2025.md %} + +## then do the following to get authentication to remote data and batch systems + +If you want to do data access or submit batch jobs, you also need to do some authentication + +{% include sl7_token.md %} diff --git a/_includes/01-introduction.md.toc.md b/_includes/01-introduction.md.toc.md new file mode 100644 index 0000000..14f5f96 --- /dev/null +++ b/_includes/01-introduction.md.toc.md @@ -0,0 +1,7 @@ +## Table of Contents +- [DUNE Computing Consortium](#dune computing consortium) +- [Schedule](#schedule) + - [Workshop Introduction Video from December 2024](#workshop introduction video from december 2024) +- [Basic setup reminder](#basic setup reminder) +- [Instructional Crew](#instructional crew) +- [Support](#support) \ No newline at end of file diff --git a/_includes/01-introduction.toc.md b/_includes/01-introduction.toc.md new file mode 100644 index 0000000..79dde35 --- /dev/null +++ b/_includes/01-introduction.toc.md @@ -0,0 +1,7 @@ +## Table of Contents +- [DUNE Computing Consortium](#dune-computing-consortium) +- [Schedule](#schedule) + - [Workshop Introduction Video from December 2024](#workshop-introduction-video-from-december-2024) +- [Basic setup reminder](#basic-setup-reminder) +- [Instructional Crew](#instructional-crew) +- [Support](#support) \ No newline at end of file diff --git a/_includes/01.5-documentation.toc.md b/_includes/01.5-documentation.toc.md new file mode 100644 index 0000000..4ed8252 --- /dev/null +++ b/_includes/01.5-documentation.toc.md @@ -0,0 +1,9 @@ +## Table of Contents +- [Documentation access](#documentation-access) +- [DUNE tools list](#dune-tools-list) +- [Docdb (Requires FNAL SSO)](#docdb-(requires-fnal-sso)) +- [DUNE wiki (Requires FNAL SSO)](#dune-wiki-(requires-fnal-sso)) + - [Tutorials list on the wiki](#tutorials-list-on-the-wiki) +- [CERN EDMS](#cern-edms) +- [Github repositories](#github-repositories) +- [DUNE Computing FAQ](#dune-computing-faq) \ No newline at end of file diff --git a/_includes/02-storage-spaces.toc.md b/_includes/02-storage-spaces.toc.md new file mode 100644 index 0000000..e80b274 --- /dev/null +++ b/_includes/02-storage-spaces.toc.md @@ -0,0 +1,35 @@ +## Table of Contents +- [This is an updated version of the 2023 training](#this-is-an-updated-version-of-the-2023-training) + - [Live Notes](#live-notes) + - [Workshop Storage Spaces Video from December 2024](#workshop-storage-spaces-video-from-december-2024) +- [Introduction](#introduction) +- [Vocabulary](#vocabulary) +- [Interactive storage volumes (mounted on dunegpvmXX.fnal.gov or lxplus.cern.ch)](#interactive-storage-volumes-(mounted-on-dunegpvmxx.fnal.gov-or-lxplus.cern.ch)) + - [Your home area](#your-home-area) + - [at Fermilab](#at-fermilab) + - [at CERN](#at-cern) + - [Locally mounted volumes](#locally-mounted-volumes) + - [Network Attached Storage (NAS)](#network-attached-storage-(nas)) + - [At Fermilab](#at-fermilab) + - [At CERN](#at-cern) + - [Grid-accessible storage volumes](#grid-accessible-storage-volumes) + - [Persistent dCache](#persistent-dcache) + - [Scratch dCache](#scratch-dcache) + - [Tape-backed dCache](#tape-backed-dcache) + - [Rucio Storage Elements](#rucio-storage-elements) + - [CVMFS](#cvmfs) + - [What is my quota?](#what-is-my-quota) + - [Your home area at FNAL](#your-home-area-at-fnal) + - [Your home area at CERN](#your-home-area-at-cern) + - [The /app/ and /data/ areas at FNAL](#the-/app/-and-/data/-areas-at-fnal) + - [EOS at CERN](#eos-at-cern) + - [Fermilab dCache](#fermilab-dcache) +- [Summary on storage spaces](#summary-on-storage-spaces) +- [Monitoring and Usage](#monitoring-and-usage) +- [Commands and tools](#commands-and-tools) + - [ifdh](#ifdh) + - [xrootd](#xrootd) + - [What is the right xroot path for a file.](#what-is-the-right-xroot-path-for-a-file.) + - [The df command](#the-df-command) +- [Quiz](#quiz) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/02.3-cvmfs.toc.md b/_includes/02.3-cvmfs.toc.md new file mode 100644 index 0000000..52c36e9 --- /dev/null +++ b/_includes/02.3-cvmfs.toc.md @@ -0,0 +1,4 @@ +## Table of Contents +- [CVMFS](#cvmfs) +- [Restrictions](#restrictions) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/03-data-management.toc.md b/_includes/03-data-management.toc.md new file mode 100644 index 0000000..b3c5c90 --- /dev/null +++ b/_includes/03-data-management.toc.md @@ -0,0 +1,39 @@ +## Table of Contents +- [Session Video](#session-video) + - [Live Notes](#live-notes) +- [Introduction](#introduction) + - [What we need to do to produce accurate physics results](#what-we-need-to-do-to-produce-accurate-physics-results) + - [How we do it ?](#how-we-do-it-) + - [How do I use this.](#how-do-i-use-this.) +- [How to find and access official data](#how-to-find-and-access-official-data) +- [Official datasets ](#Official_Datasets) + - [Fast web catalog queries](#fast-web-catalog-queries) + - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries) + - [metacat web interface](#metacat-web-interface) + - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo) + - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches) +- [What is metacat?](#what-is-metacat) + - [Find a file in metacat](#find-a-file-in-metacat) + - [Example of doing a metacat search](#example-of-doing-a-metacat-search) + - [then do queries to find particular groups of files](#then-do-queries-to-find-particular-groups-of-files) + - [What do those fields mean?](#what-do-those-fields-mean) + - [find out how much raw data there is in a run using the summary option](#find-out-how-much-raw-data-there-is-in-a-run-using-the-summary-option) + - [Fast web catalog queries](#fast-web-catalog-queries) + - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries) + - [metacat web interface](#metacat-web-interface) + - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo) + - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches) + - [find out how much data there is in a dataset](#find-out-how-much-data-there-is-in-a-dataset) + - [What describes a dataset?](#what-describes-a-dataset) + - [What files are in that dataset and how do I use them?](#what-files-are-in-that-dataset-and-how-do-i-use-them) +- [Finding those files on disk](#finding-those-files-on-disk) +- [Getting file locations using Rucio](#getting-file-locations-using-rucio) + - [What is Rucio?](#what-is-rucio) + - [You will need to authenticate to read files](#you-will-need-to-authenticate-to-read-files) +- [Accessing `rucio` and `justin` require a bit more](#accessing-`rucio`-and-`justin`-require-a-bit-more) + - [getting a token for xroot access in AL9](#getting-a-token-for-xroot-access-in-al9) + - [finding a file](#finding-a-file) +- [More finding files by characteristics using metacat](#more-finding-files-by-characteristics-using-metacat) +- [Accessing data for use in your analysis](#accessing-data-for-use-in-your-analysis) +- [Quiz](#quiz) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/03.2-UPS.toc.md b/_includes/03.2-UPS.toc.md new file mode 100644 index 0000000..e6ca5d9 --- /dev/null +++ b/_includes/03.2-UPS.toc.md @@ -0,0 +1,3 @@ +## Table of Contents +- [What is UPS and why do we need it?](#what-is-ups-and-why-do-we-need-it) + - [UPS basic commands](#ups-basic-commands) \ No newline at end of file diff --git a/_includes/04-Spack.toc.md b/_includes/04-Spack.toc.md new file mode 100644 index 0000000..7066fcf --- /dev/null +++ b/_includes/04-Spack.toc.md @@ -0,0 +1,5 @@ +## Table of Contents +- [What is Spack and why do we need it?](#what-is-spack-and-why-do-we-need-it) +- [Minimal spack for root analysis and file access](#minimal-spack-for-root-analysis-and-file-access) +- [A more flexible environment with more packages but you have to make choices of versions](#a-more-flexible-environment-with-more-packages-but-you-have-to-make-choices-of-versions) + - [Spack basic commands](#spack-basic-commands) \ No newline at end of file diff --git a/_includes/05-end-of-basics.toc.md b/_includes/05-end-of-basics.toc.md new file mode 100644 index 0000000..f1e3b06 --- /dev/null +++ b/_includes/05-end-of-basics.toc.md @@ -0,0 +1,3 @@ +## Table of Contents +- [You can ask questions here:](#you-can-ask-questions-here:) +- [You can continue on with these additional modules.](#you-can-continue-on-with-these-additional-modules.) \ No newline at end of file diff --git a/_includes/05.1-improve-code-efficiency.toc.md b/_includes/05.1-improve-code-efficiency.toc.md new file mode 100644 index 0000000..5bedb17 --- /dev/null +++ b/_includes/05.1-improve-code-efficiency.toc.md @@ -0,0 +1,11 @@ +## Table of Contents + - [Session Video](#session-video) + - [Live Notes](#live-notes) + - [Code Make-over](#code-make-over) + - [CPU optimization:](#cpu-optimization:) +- [Memory optimization:](#memory-optimization:) +- [I/O optimization:](#i/o-optimization:) +- [Build time optimization:](#build-time-optimization:) +- [Workflow optimization:](#workflow-optimization:) +- [Software readability and maintainability:](#software-readability-and-maintainability:) +- [Coding for Thread Safety](#coding-for-thread-safety) \ No newline at end of file diff --git a/_includes/10-closing-remarks.toc.md b/_includes/10-closing-remarks.toc.md new file mode 100644 index 0000000..ea1d5ab --- /dev/null +++ b/_includes/10-closing-remarks.toc.md @@ -0,0 +1,6 @@ +## Table of Contents +- [Video Session](#video-session) +- [Two Days of Training](#two-days-of-training) +- [Survey time!](#survey-time!) +- [Next Steps](#next-steps) +- [Long Term Support](#long-term-support) \ No newline at end of file diff --git a/_includes/OfficialDatasets_include.md b/_includes/OfficialDatasets_include.md new file mode 100644 index 0000000..cc0d064 --- /dev/null +++ b/_includes/OfficialDatasets_include.md @@ -0,0 +1,94 @@ + +## Official datasets + +The production group make official datasets which are sets of files which share important characteristics such as experiment, data_tier, data_stream, processing version and processing configuration. Often all you need is an official dataset. + +See [DUNE Physics Datasets](https://docs.dunescience.org/cgi-bin/sso/RetrieveFile?docid=29787&filename=DUNEdataset_v1.pdf) for a detailed description. + +### Fast web catalog queries + +You can do fast string queries based on keywords embedded in the dataset name. + +Go to [dunecatalog](https://dune-tech.rice.edu/dunecatalog/) and log in with your services password. + +Choose your apparatus (Far Detector for example), use the category key to further refine your search and then type in keywords. Here I chose the `Far Detectors` tab and the `FD-VD` category from the pulldown menu. + +![Fast keyword search]({{ site.baseurl }}/fig/keywordquery.png){: .image-with-shadow } + +If you click on a dataset you can see a sample of the files inside it. + + +You can find a more detailed tutorial for the dunecatalog site at: +[Dune Catalog Tutorial](https://docs.dunescience.org/cgi-bin/sso/RetrieveFile?docid=33738&filename=DUNE%20Catalog%20Presentation.pdf&version=2) + + + +### Command line tools and advanced queries + +You can also explore and find the right dataset on the command line by using metacat dataset keys: + +First you need to know your namespace and then explore within it. + +~~~ +metacat namespace list # find likely namespaces +~~~ +{: .language-bash} + +There are official looking ones like `hd-protodune-det-reco` and ones for users doing production testing like `schellma`. The default for general use is `usertests` + +Creation of namespaces by non-privileged users is currently disabled. A tool is in progress which will automatically make one namespace for each user + +### metacat web interface + +Metacat also has a web interface that is useful in exploring file parentage [metacat gui](https://metacat.fnal.gov:9443/dune_meta_prod/app/gui) + +### Example of finding reconstructed Monte Carlo + +Let's look for some reconstructed Monte Carlo from the VD far detector. + +~~~ +metacat query "datasets matching fardet-vd:*official having core.data_tier=full-reconstructed" +~~~ +{: .language-bash} + +Lots of output ... looks like there are 2 types of official ones - let's get "v2" + +~~~ +metacat query "datasets matching fardet-vd:*v2_official having core.data_tier=full-reconstructed" +~~~ +{: .language-bash} + +and there are then several different generators. Let's explore reconstructed simulation of the vertical drift far detector. + +~~~ +metacat query "datasets matching fardet-vd:*v2_official having core.data_tier=full-reconstructed and dune_mc.gen_fcl_filename=prodgenie_nu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg.fcl" +~~~ +{: .language-bash} + +Ok, found the official neutrino beam dataset: + +~~~ +fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3__prodgenie_nu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official +~~~ +{: .output} + + +~~~ +metacat query "datasets matching fardet-vd:*v2_official having core.data_tier=full-reconstructed and dune_mc.gen_fcl_filename=prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg.fcl" +~~~ + +And the anti-neutrino dataset: + +~~~ +fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official +~~~ +{: .output} + + + +### you can use the web data catalog to do advanced searches + +You can also do keyword/value queries like the ones above using the Other tab on the web-based Data Catalog. + +![Full query search]({{ site.baseurl }}/fig/otherquery.png){: .image-with-shadow } + diff --git a/_includes/al9_token.md b/_includes/al9_token.md index 5f1ed5b..f851532 100644 --- a/_includes/al9_token.md +++ b/_includes/al9_token.md @@ -22,7 +22,7 @@ export RUCIO_ACCOUNT=justinreadonly Then use htgettoken to get a token so you can read the files you find. ~~~ -htgettoken -i dune --vaultserver htvaultprod.fnal.gov #:8200 +htgettoken -i dune --vaultserver htvaultprod.fnal.gov -r interactive export BEARER_TOKEN_FILE=/run/user/`id -u`/bt_u`id -u` ~~~ {: .language-bash} diff --git a/_includes/setup.toc.md b/_includes/setup.toc.md new file mode 100644 index 0000000..310d2aa --- /dev/null +++ b/_includes/setup.toc.md @@ -0,0 +1,33 @@ +## Table of Contents +- [Objectives](#objectives) +- [Requirements](#requirements) +- [Step 1: DUNE membership](#step 1: dune membership) +- [Step 2: Getting accounts](#step 2: getting accounts) + - [With FNAL](#with fnal) + - [With CERN](#with cern) +- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page)) +- [0. Basic setup on your computer.](#0. basic setup on your computer.) +- [1. Kerberos business](#1. kerberos business) +- [2. ssh-in](#2. ssh-in) +- [3. Get a clean shell](#3. get a clean shell) +- [Software setup ](#software setup ) + - [4.1 Setting up DUNE software - Scientific Linux 7 version ](#4.1 setting up dune software - scientific linux 7 version ) + - [Caveats for later](#caveats for later) + - [4.2 Setting up DUNE software - Alma9 version ](#4.2 setting up dune software - alma9 version ) + - [Caveats for later](#caveats for later) +- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version) + - [Caveats](#caveats) +- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy)) +- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy)) +- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access) + - [Tokens method ](#tokens method ) + - [1. Get and store your token](#1. get and store your token) + - [2. Tell the system where your token is](#2. tell the system where your token is) +- [Set up on CERN machines ](#set up on cern machines ) + - [1. Setup in Alma9](#1. setup in alma9) + - [2. For SL7](#2. for sl7) + - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script) + - [3. Getting authentication for data access](#3. getting authentication for data access) + - [4. Access tutorial datasets](#4. access tutorial datasets) + - [5. Notify us](#5. notify us) + - [Useful Links](#useful links) \ No newline at end of file diff --git a/_includes/sl7_setup_2025.md b/_includes/sl7_setup_2025.md index 17e93d7..a3543f1 100644 --- a/_includes/sl7_setup_2025.md +++ b/_includes/sl7_setup_2025.md @@ -21,9 +21,5 @@ export DUNELAR_QUALIFIER=e26:prof # you want to update this setup -B dunesw ${DUNELAR_VERSION} -q ${DUNELAR_QUALIFIER} -setup metacat -setup rucio -export RUCIO_ACCOUNT=justinreadonly -setup justin ~~~ {: .language-bash} \ No newline at end of file diff --git a/_includes/sl7_token.md b/_includes/sl7_token.md index b74881f..1781140 100644 --- a/_includes/sl7_token.md +++ b/_includes/sl7_token.md @@ -1,34 +1,76 @@ -To get a token that allows you to access files (and rucio) in SL7 +To get a token that allows you to access files interactively in SL7 ~~~ +htgettoken -i dune --vaultserver htvaultprod.fnal.gov -r interactive +export BEARER_TOKEN_FILE=/run/user/`id -u`/bt_u`id -u` +~~~ +{: ..language-bash} + +put this in a file called `dune_token.sh` + +The first time you do this you will see: +~~~ +Attempting kerberos auth with https://htvaultprod.fnal.gov:8200 ... succeeded +Attempting to get token from https://htvaultprod.fnal.gov:8200 ... failed +Attempting OIDC authentication with https://htvaultprod.fnal.gov:8200 + +Complete the authentication at: + https://cilogon.org/device/?user_code=XXXX +No web open command defined, please copy/paste the above to any web browser +Waiting for response in web browser +~~~ +{: ..output} + +Go to that web site and authenticate + +~~~ +Storing vault token in /tmp/vt_uXXX +Saving credkey to /nashome/s/USER/.config/htgettoken/credkey-dune-interactive +Saving refresh token ... done +Attempting to get token from https://htvaultprod.fnal.gov:8200 ... succeeded +Storing bearer token in /run/user/XXXX/bt_XXXX +~~~ +{: ..output} + + + +## Accessing `rucio` and `justin` require a bit more + +in SL7 - put this in a file called `dune_data_sl7.sh` so you can use it again. + +~~~ +setup metacat +setup rucio +export RUCIO_ACCOUNT=justinreadonly setup justin -justin time # this just tells justin you want to authenticate +justin time # this just tells justin that you exist and want to authenticate +justin get-token # this actually gets a token and associated proxy for access to rucio and the batch system ~~~ -{: .language-bash} +{: ..language-bash} -The first time it will ask you to open a web browser, authenticate and enter the long string it delivers to you. +The first time you do this you will get asked (after the `justin time` command) ~~~ To authorize this computer to run the justin command, visit this page with your usual web browser and follow the instructions within the next 10 minutes: -https://dunejustin.fnal.gov/authorize/_W_azUJcLhYmAOqClYz9RAsnKbDgzQ6lNA +https://dunejustin.fnal.gov/authorize/XXXXX -Check that the Session ID displayed on that page is -cprbbe +Check that the Session ID displayed on that page is BfhVBmQ -Once you've followed the instructions on that web page, you can run the justin -command without needing to authorize this computer again for 7 days. +Once you've followed the instructions on that web page, please run the command +you tried again. You won't need to authorize this computer again for 7 days. ~~~ -{: .output} +{: ..output} -That gave you authorization to use justin. Now do the command again to get an actual token. +Once again go to the website that appears and authenticate. After the first authentication to justIn you need to do a second justin call ~~~ justin get-token ~~~ -{: .language-bash} +{: ..language-bash} -You will have to do this sequence weekly as your justin access expires. +You will need to do this sequence weekly as your justin access expires. > ## Note: > Despite the name of this command it gets you both a token and a special X.509 proxy and it is the latter you are actually using to talk to rucio in these SL7 examples \ No newline at end of file diff --git a/_includes/sl7_token_justin.md b/_includes/sl7_token_justin.md new file mode 100644 index 0000000..b74881f --- /dev/null +++ b/_includes/sl7_token_justin.md @@ -0,0 +1,34 @@ + +To get a token that allows you to access files (and rucio) in SL7 + +~~~ +setup justin +justin time # this just tells justin you want to authenticate +~~~ +{: .language-bash} + +The first time it will ask you to open a web browser, authenticate and enter the long string it delivers to you. + +~~~ +To authorize this computer to run the justin command, visit this page with your +usual web browser and follow the instructions within the next 10 minutes: +https://dunejustin.fnal.gov/authorize/_W_azUJcLhYmAOqClYz9RAsnKbDgzQ6lNA + +Check that the Session ID displayed on that page is -cprbbe + +Once you've followed the instructions on that web page, you can run the justin +command without needing to authorize this computer again for 7 days. +~~~ +{: .output} + +That gave you authorization to use justin. Now do the command again to get an actual token. + +~~~ +justin get-token +~~~ +{: .language-bash} + +You will have to do this sequence weekly as your justin access expires. + +> ## Note: +> Despite the name of this command it gets you both a token and a special X.509 proxy and it is the latter you are actually using to talk to rucio in these SL7 examples \ No newline at end of file diff --git a/code/tocgen.py b/code/tocgen.py new file mode 100644 index 0000000..7d4831d --- /dev/null +++ b/code/tocgen.py @@ -0,0 +1,52 @@ +import os,sys,string + +def dealwithline(input,toclines): + for line in input: + if line.startswith('{% include'): + print ("dealing with include",line) + includename = line.split('%')[1].strip().split(' ')[1] + newfile = os.path.join("_includes/",includename) + newlines = open(newfile,'r').readlines() + dealwithline(newlines,toclines) + continue + # remove any labels + # if " ## If you run into problems now or later, check out the [Common Error Messages]({{ site.baseurl }}/ErrorMessages) page and the [FAQ page](https://github.com/orgs/DUNE/projects/19/) > if that doesn't help, use [DUNE Slack](https://dunescience.slack.com/archives/C02TJDHUQPR) channel `#computing-training-basics` to ask us about the problem - there is always a new one cropping up. {: .challenge} +{%include setup.toc.md%} + ## Requirements @@ -750,7 +751,7 @@ If you are experiencing issues, please contact us as soon as possible. Be sure t > If you have issues here, please go to the [#computing-training-basics](https://dunescience.slack.com/archives/C02TJDHUQPR)Slack channel to get support. Please note that you are on a CERN machine in your message. Thanks! {: .discussion} -### Useful Links +## Useful Links The [DUNE FAQ][DUNE FAQ] on GitHub. diff --git a/setup.toc.md b/setup.toc.md new file mode 100644 index 0000000..74d5e5f --- /dev/null +++ b/setup.toc.md @@ -0,0 +1,32 @@ +- [Objectives](#objectives) +- [Requirements](#requirements) +- [Step 1: DUNE membership](#step 1: dune membership) +- [Step 2: Getting accounts](#step 2: getting accounts) + - [With FNAL](#with fnal) + - [With CERN](#with cern) +- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page)) +- [0. Basic setup on your computer.](#0. basic setup on your computer.) +- [1. Kerberos business](#1. kerberos business) +- [2. ssh-in](#2. ssh-in) +- [3. Get a clean shell](#3. get a clean shell) +- [Software setup ](#software setup ) + - [4.1 Setting up DUNE software - Scientific Linux 7 version ](#4.1 setting up dune software - scientific linux 7 version ) + - [Caveats for later](#caveats for later) + - [4.2 Setting up DUNE software - Alma9 version ](#4.2 setting up dune software - alma9 version ) + - [Caveats for later](#caveats for later) +- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version) + - [Caveats](#caveats) +- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy)) +- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy)) +- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access) + - [Tokens method ](#tokens method ) + - [1. Get and store your token](#1. get and store your token) + - [2. Tell the system where your token is](#2. tell the system where your token is) +- [Set up on CERN machines ](#set up on cern machines ) + - [1. Setup in Alma9](#1. setup in alma9) + - [2. For SL7](#2. for sl7) + - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script) + - [3. Getting authentication for data access](#3. getting authentication for data access) + - [4. Access tutorial datasets](#4. access tutorial datasets) + - [5. Notify us](#5. notify us) + - [Useful Links](#useful links) \ No newline at end of file