William & Mary Research Computing: File Systems Tutorial

3.3. William & Mary Research Computing: File Systems Tutorial#

3.3.1. Overview#

William & Mary Research Computing offers multiple file systems for different purposes. This tutorial will guide you through each file system, its use case, and how to interact with it. Most of the content in this page has been taken from the lecture given at W&M by W&M Research Computing

3.3.2. File Systems#

3.3.2.1. 1. Home Directory (/sciclone/home)#

  • Purpose: Long-term storage for personal files and software installations.

  • Backup: Yes.

  • Usage:

    # Navigate to home directory
cd /sciclone/home/<username>

3.3.2.2. 2. Data Directory (/sciclone/data10)#

  • Purpose: Long-term storage for project data.

  • Backup: Yes.

  • Usage:

    # Navigate to data directory
cd /sciclone/data10/<username>

3.3.2.3. 3. Scratch Directory (/sciclone/scr10)#

  • Purpose: Temporary storage for running jobs. Files are purged after 90 days.

  • Backup: No.

  • Usage:

    # Navigate to scratch directory
cd /sciclone/scr10/<username>

3.3.2.4. 4. Project Scratch Directory (/sciclone/pscr)#

  • Purpose: Temporary storage for project-related jobs. Files are purged after 90 days.

  • Backup: No.

  • Usage:

    # Navigate to project scratch directory
cd /sciclone/pscr/<username>

3.3.3. Best Practices#

  • Use Scratch Space for Jobs: Always use scratch directories for running jobs to avoid overloading the home and data directories.

  • Backup Important Data: Regularly backup important data stored in scratch directories to avoid data loss.

  • Responsible Usage: Be mindful of disk space usage to ensure fair resource distribution among users.

3.3.4. Transferring Files#

3.3.4.1. Using Globus#

  • Purpose: Efficiently transfer files between local systems and W&M clusters. Check out globus.org

  • Usage:

    # Example command to transfer files using Globus
globus transfer <source_endpoint> <destination_endpoint>

3.3.5. Permissions and Sharing#

3.3.5.1. Changing Permissions#

  • Commands:

    # Change group ownership
chgrp <group> <file/directory>

# Change permissions
chmod <permissions> <file/directory>

3.3.6. Conclusion#

Understanding and effectively using the different file systems at William & Mary Research Computing is crucial for efficient and responsible resource usage. For more detailed information, refer to the Research Computing documentation.

Try this

Can you perform the git exercise here in at one of the HPC systems? Try to create a directory at sciclone/home/<username>/data10/

Contents