Skip to content

Switch all remote access from XRootD to WebDAV #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
djw8605 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Switch all remote access from XRootD to WebDAV #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
377 changes: 206 additions & 171 deletions bin/cvmfs_sync
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions config/cvmfs-sync.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ URL: https://github.com/bbockelm/cvmfs-sync
Source0: %{name}-%{version}.tar.gz

Requires: cvmfs-server
Requires: xrootd-python
Requires: python3-webdav4
Requires(pre): shadow-utils

%{?systemd_requires}
Expand Down Expand Up @@ -58,7 +58,7 @@ install -d $RPM_BUILD_ROOT/%{_localstatedir}/cache/cvmfs-sync
getent group cvmfs-sync >/dev/null || groupadd -r cvmfs-sync
getent passwd cvmfs-sync >/dev/null || \
useradd -r -g cvmfs-sync -d /var/lib/cvmfs-sync -s /sbin/nologin \
-c "User to synchronize CVMFS with a remote XRootD server namespace" cvmfs-sync
-c "User to synchronize CVMFS with a remote WebDAV server namespace" cvmfs-sync
exit 0


Expand Down
4 changes: 2 additions & 2 deletions config/des.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
repo = des.osgstorage.org

[Sync DES]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/des/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/des/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/des/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/des/persistent/stash/
destination = /cvmfs/des.osgstorage.org/pnfs/fnal.gov/usr/des/persistent/stash/
6 changes: 3 additions & 3 deletions config/dune.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = dune.osgstorage.org

[Sync mu2e]
#source = root://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/dune/persistent/stash/
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/dune/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/dune/persistent/stash/
#source = https://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/dune/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/dune/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/dune/persistent/stash/
destination = /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/
2 changes: 1 addition & 1 deletion config/et-gw.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
repo = et-gw.osgstorage.org

[Sync et-gw MDC1]
source = root://et-origin.cism.ucl.ac.be:1094//et-gw/PUBLIC/MDC1
source = https://et-origin.cism.ucl.ac.be:1094//et-gw/PUBLIC/MDC1
destination = /cvmfs/et-gw.osgstorage.org/et-gw/PUBLIC/MDC1
4 changes: 2 additions & 2 deletions config/gluex.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
repo = gluex.osgstorage.org

[Sync Gluex-uconn0]
source = root://nod25.phys.uconn.edu:1094//gluex/uconn0
source = https://nod25.phys.uconn.edu:1094//gluex/uconn0
destination = /cvmfs/gluex.osgstorage.org/gluex/uconn0
include =
concurrency = 10
max_time = 600
ignore = *.tmp, *.swp

[Sync Gluex-uconn1]
source = root://cn445.storrs.hpc.uconn.edu:1094//gluex/uconn1
source = https://cn445.storrs.hpc.uconn.edu:1094//gluex/uconn1
destination = /cvmfs/gluex.osgstorage.org/gluex/uconn1
include =
concurrency = 10
Expand Down
2 changes: 1 addition & 1 deletion config/gwosc.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
repo = gwosc.osgstorage.org

[Sync gwosc]
source = root://origin.ligo.caltech.edu:1094/
source = https://origin.ligo.caltech.edu:1094/
destination = /cvmfs/gwosc.osgstorage.org/
concurrency = 50
4 changes: 2 additions & 2 deletions config/icarus.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
repo = icarus.osgstorage.org

[Sync icarus]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/icarus/persistent/stash
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/icarus/persistent/stash
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/icarus/persistent/stash
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/icarus/persistent/stash
destination = /cvmfs/icarus.osgstorage.org/pnfs/fnal.gov/usr/icarus/persistent/stash
2 changes: 1 addition & 1 deletion config/icecube.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
repo = icecube.osgstorage.org

[Sync icecube]
source = root://stash-origin.icecube.wisc.edu//icecube/PUBLIC
source = https://stash-origin.icecube.wisc.edu//icecube/PUBLIC
destination = /cvmfs/icecube.osgstorage.org/icecube/PUBLIC
4 changes: 2 additions & 2 deletions config/jlab.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
repo = jlab.osgstorage.org

[Sync jlab]
source = root://sci-xrootd.jlab.org:2094//jlab
source = https://sci-xrootd.jlab.org:2094//jlab
destination = /cvmfs/jlab.osgstorage.org/jlab
include =
concurrency = 10
#max_time = 600
max_time = 30

[Sync eic]
source = root://sci-xrootd.jlab.org:2094//eic
source = https://sci-xrootd.jlab.org:2094//eic
destination = /cvmfs/jlab.osgstorage.org/eic
include =
#concurrency = 10
Expand Down
2 changes: 1 addition & 1 deletion config/kagra.storage.igwn.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ x509_credential = /tmp/x509up_u1221
command = /usr/libexec/cvmfs-sync/ligo-auth-gen {authz_output} -p ZZZ_PASSWORD_ZZZ -w /usr/share/cvmfs-sync/kagra_authz

[Sync kagra]
source = root://kagra-dsr-b2.icrr.u-tokyo.ac.jp:1095//igwn/kagra/
source = https://kagra-dsr-b2.icrr.u-tokyo.ac.jp:1095//igwn/kagra/
destination = /cvmfs/kagra.storage.igwn.org/igwn/kagra/
2 changes: 1 addition & 1 deletion config/ligo-test.storage.igwn.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ x509_credential = /tmp/x509up_u1221
command = /usr/libexec/cvmfs-sync/ligo-auth-gen {authz_output} -p ZZZ_PASSWORD_ZZZ -w /usr/share/cvmfs-sync/ligo_storage_authz

[Sync ligo]
source = root://origin-readtest.ligo.caltech.edu:1095//igwn/test/ligo
source = https://origin-readtest.ligo.caltech.edu:1095//igwn/test/ligo
destination = /cvmfs/ligo-test.storage.igwn.org/igwn/test/ligo
concurrency = 20
2 changes: 1 addition & 1 deletion config/ligo.storage.igwn.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ x509_credential = /tmp/x509up_u1221
command = /usr/libexec/cvmfs-sync/ligo-auth-gen {authz_output} -p ZZZ_PASSWORD_ZZZ -w /usr/share/cvmfs-sync/ligo_storage_authz

[Sync ligo]
source = root://origin-ifo.ligo.caltech.edu:1095//igwn/ligo/
source = https://origin-ifo.ligo.caltech.edu:1095//igwn/ligo/
destination = /cvmfs/ligo.storage.igwn.org/igwn/ligo/
concurrency = 20
6 changes: 3 additions & 3 deletions config/minerva.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = minerva.osgstorage.org

[Sync mu2e]
#source = root://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/minerva/persistent/stash/
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/minerva/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/minerva/persistent/stash/
#source = https://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/minerva/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/minerva/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/minerva/persistent/stash/
destination = /cvmfs/minerva.osgstorage.org/pnfs/fnal.gov/usr/minerva/persistent/stash/
6 changes: 3 additions & 3 deletions config/mu2e.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = mu2e.osgstorage.org

[Sync mu2e]
#source = root://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/mu2e/persistent/stash/
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/mu2e/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/mu2e/persistent/stash/
#source = https://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/mu2e/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/mu2e/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/mu2e/persistent/stash/
destination = /cvmfs/mu2e.osgstorage.org/pnfs/fnal.gov/usr/mu2e/persistent/stash/
8 changes: 4 additions & 4 deletions config/nova.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
repo = nova.osgstorage.org

#[Sync flux]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/nova/data/flux/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/nova/data/flux/
#destination = /cvmfs/nova.osgstorage.org/pnfs/fnal.gov/usr/nova/data/flux/

[Sync analysis]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/analysis/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/analysis/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/analysis/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/analysis/
destination = /cvmfs/nova.osgstorage.org/pnfs/fnal.gov/usr/nova/persistent/analysis/
ignore = /pnfs/fnal.gov/usr/nova/persistent/analysis/nux/nus_cmf

[Sync stash]
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/nova/persistent/stash/
destination = /cvmfs/nova.osgstorage.org/pnfs/fnal.gov/usr/nova/persistent/stash/
2 changes: 1 addition & 1 deletion config/public-uc.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
repo = public-uc.osgstorage.org

[Sync public-uc]
source = root://pelican-osdf-public.tempest.uchicago.edu:8443//ospool/uc-shared/public
source = https://pelican-osdf-public.tempest.uchicago.edu:8443//ospool/uc-shared/public
destination = /cvmfs/public-uc.osgstorage.org/ospool/uc-shared/public
concurrency = 50
4 changes: 2 additions & 2 deletions config/sbn.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = sbn.osgstorage.org

[Sync sbn]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/sbn/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/sbn/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/sbn/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/sbn/persistent/stash/
destination = /cvmfs/sbn.osgstorage.org/pnfs/fnal.gov/usr/sbn/persistent/stash/

4 changes: 2 additions & 2 deletions config/sbnd.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = sbnd.osgstorage.org

[Sync sbnd]
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/sbnd/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/sbnd/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/sbnd/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/sbnd/persistent/stash/
destination = /cvmfs/sbnd.osgstorage.org/pnfs/fnal.gov/usr/sbnd/persistent/stash/

2 changes: 1 addition & 1 deletion config/scitokens.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ file = /etc/cvmfs-sync/scitokens.authz

# Skipping, as recommended by Tom Downes
#[Sync postO2]
#source = root://xrootd-local.unl.edu//user/ligo/frames/postO2/
#source = https://xrootd-local.unl.edu//user/ligo/frames/postO2/
#destination = /cvmfs/ligo.osgstorage.org/frames/postO2/


2 changes: 1 addition & 1 deletion config/sdsc-nrp-osdf-origin.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
repo = sdsc-nrp-osdf-origin.osgstorage.org

[Sync nrp]
source = root://sdsc-origin.nationalresearchplatform.org:1094//nrp
source = https://sdsc-origin.nationalresearchplatform.org:1094//nrp
destination = /cvmfs/sdsc-nrp-osdf-origin.osgstorage.org/nrp
concurrency = 10
max_time = 600
2 changes: 1 addition & 1 deletion config/shared.storage.igwn.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ x509_credential = /tmp/x509up_u1221
command = /usr/libexec/cvmfs-sync/ligo-auth-gen {authz_output} -p ZZZ_PASSWORD_ZZZ -w /usr/share/cvmfs-sync/shared_igwn_authz

[Sync ligo]
source = root://origin-shared.ligo.caltech.edu:1095//igwn/shared/
source = https://origin-shared.ligo.caltech.edu:1095//igwn/shared/
destination = /cvmfs/shared.storage.igwn.org/igwn/shared/
#concurrency = 20
8 changes: 4 additions & 4 deletions config/snomass21.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,31 @@
repo = snomass21.osgstorage.org

#[Sync Merra2]
#source = root://stashcache-origin-merra2.nautilus.optiputer.net:31094//merra2
#source = https://stashcache-origin-merra2.nautilus.optiputer.net:31094//merra2
#destination = /cvmfs/stash.osgstorage.org/merra2
#include =
#concurrency = 50
#max_time = 600
#ignore = *.tmp, *.swp

#[Sync Gluex]
#source = root://nod25.phys.uconn.edu//Gluex
#source = https://nod25.phys.uconn.edu//Gluex
#destination = /cvmfs/stash.osgstorage.org/Gluex
#include =
#concurrency = 10
#max_time = 600
#ignore = *.tmp, *.swp

#[Sync chtc]
#source = root://sc-origin.chtc.wisc.edu//chtc/PUBLIC
#source = https://sc-origin.chtc.wisc.edu//chtc/PUBLIC
#destination = /cvmfs/stash.osgstorage.org/chtc/PUBLIC
#include =
#concurrency = 10
#max_time = 600
#ignore = *.tmp, *.swp

#[Sync Stash]
#source = root://stash.osgconnect.net//osgconnect/public/
#source = https://stash.osgconnect.net//osgconnect/public/
#destination = /cvmfs/stash.osgstorage.org/osgconnect/public/
#include = /user/*/public/*
#ignore = *.tmp, *.swp, /user/*/public/public, /user/krieger, /user/nepomuk, /user/toth
Expand Down
6 changes: 3 additions & 3 deletions config/uboone.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
repo = uboone.osgstorage.org

[Sync uboone]
#source = root://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/uboone/persistent/stash/
#source = root://stashcache.fnal.gov//pnfs/fnal.gov/usr/uboone/persistent/stash/
source = root://stashredirector.fnal.gov//pnfs/fnal.gov/usr/uboone/persistent/stash/
#source = https://fndca1.fnal.gov:1095/pnfs/fnal.gov/usr/uboone/persistent/stash/
#source = https://stashcache.fnal.gov//pnfs/fnal.gov/usr/uboone/persistent/stash/
source = https://stashredirector.fnal.gov//pnfs/fnal.gov/usr/uboone/persistent/stash/
destination = /cvmfs/uboone.osgstorage.org/pnfs/fnal.gov/usr/uboone/persistent/stash/
4 changes: 2 additions & 2 deletions config/uhawaii.osgstorage.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ repo = uhawaii.osgstorage.org


[Sync me2022]
source = root://prp01.ifa.hawaii.edu//uhawaii/manoa/ifa/me2022
source = https://prp01.ifa.hawaii.edu//uhawaii/manoa/ifa/me2022
destination = /cvmfs/uhawaii.osgstorage.org/uhawaii/manoa/ifa/me2022
#concurrency = 10
#ignore = sent2/*, sent1/*, c3/*, tfr/*

[Sync hinode]
source = root://prp01.ifa.hawaii.edu//uhawaii/manoa/ifa/hinode
source = https://prp01.ifa.hawaii.edu//uhawaii/manoa/ifa/hinode
destination = /cvmfs/uhawaii.osgstorage.org/uhawaii/manoa/ifa/hinode
2 changes: 1 addition & 1 deletion config/virgo.storage.igwn.org.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ x509_credential = /tmp/x509up_u1221
command = /usr/libexec/cvmfs-sync/ligo-auth-gen {authz_output} -p ZZZ_PASSWORD_ZZZ -w /usr/share/cvmfs-sync/virgo_authz

[Sync virgo]
source = root://ingrid-se09.cism.ucl.ac.be:1095//igwn/virgo/
source = https://ingrid-se09.cism.ucl.ac.be:1095//igwn/virgo/
destination = /cvmfs/virgo.storage.igwn.org/igwn/virgo/
8 changes: 1 addition & 7 deletions update-scripts/cvmfs-sync-driver
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,6 @@ class CvmfsSyncDriver(object):
if config.has_option(section, "include"):
self.include = re.split(r",?\s*", config.get(section, "include"))
self.source = config.get(section, "source")
self.gridftp_source = None
if config.has_option(section, "gridftp_source"):
self.gridftp_source = config.get(section, "gridftp_source")
self.destination = config.get(section, "destination")

def run(self):
Expand All @@ -74,10 +71,7 @@ class CvmfsSyncDriver(object):
if self.include:
command += ["--include", ",".join(self.include)]
command += ["--verbose"]
source = self.source
if self.gridftp_source:
source += "," + self.gridftp_source
command += [source, self.destination]
command += [self.source, self.destination]
process = subprocess.Popen(command)
process.wait()
return process.returncode
Expand Down