GIT workshop

Manuela Salvucci

manuelasalvucci@rcsi.ie

2019-11-06

Outline

• What is version control?
• Why bother with “formal” version control?
• How to install and get started with GIT
• Use GIT core features
• Review files history, revert/amend changes
• Collaborate online with others with BitBucket, GitHub or GitLab
• Hands-on examples

What is version control?

“Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later.”

(Pro Git, Scott Chacon and Ben Straub, 2014)

Without version control… a way too familiar picture

Without version control… “informal” versioning

• None
• Named files:
  • OK:
    • manuscript_my_draft.docx
    • manuscript_my_draft_with_coauthor_comments.docx
    • …
  • Better:
    • manuscript_draft_v01.docx
    • manuscript_draft_v02.docx
    • …
• Named zip-files:
  • manuscript_drafts.zip
  • manuscript_cell_submission.zip
  • manuscript_pnas_submission.zip
  • manuscript_pnas_revisions.zip
  • manuscript_pnas_proofs.zip
• Sync online services (Microsoft/Dropbox/Google/Overleaf/Sharelatex)

Without version control… challenges

• Time consuming
• Error prone
• Requires self-discipline (save everything, good file names, sticking to a routine, …)
• Relationship between changes in multiple files is lost
• Information about what, when and why something changed is lost?
  • How would you go about finding out when the p-value for Figure 2.A got set to the (wrong) value?
• Non-linear history (parallel versions)
• Disk space

Why bother with “formal” version control?

• We are too busy to use inefficient, manual, error-prone versioning
• Research is increasingly collaborative:
  • we need a better way to document the rational behind data cleaning, analysis steps, generation of figures, write-ups…
  • we need a better way to “merge” inputs and feedback to the project from collaborators
  • often your future self is the collaborator (and you don’t reply to emails…)
• We do research anywhere:
  • on our workstation at work
  • on the laptop at home/bus/conference
  • on a dedicated facility workstation
  • …
• Projects are always evolving and never “really” finished

What can Version Control Systems do for my research?

• Version Control Systems are software that keep track of your files and their full history
• Project files and “history” in the form of “snapshots”/“checkpoints” are organized in a folder
• Explicitly indicate what file(s) and what change(s) to store with a named snapshot (include why the changes were made)
• Can “go back in time” and see/use files how they look at a specific snapshot
• Can see what changed between snapshots, and in what snapshot content was first introduced
• Can “experiment” by having “organized parallel versions” of files
• Synchronize different copies of the project between different computers/collaborators

Version Control Systems - Vocabulary

• Version Control Systems are software that keep track of your files and their full history
• Project files and “history” in the form of “snapshots”/“checkpoints” are organized in a folder -> repository
• Explicitly indicate what file(s) and what change(s) to store with a named snapshot (include why the changes were made) -> commit or revision
• Can “go back in time” or “jump forward” and see/use files how they look at a specific snapshot -> checkout or revert
• Can see what changed between snapshots, and in what snapshot content was first introduced -> diff, annotate and blame
• Can “experiment” by having “organized parallel versions” of files -> branch
• Synchronize different copies of the project between different computers/collaborators -> push & pull

What type of files can I track with version control?

• All types of files can be tracked with version control (but big files may require special care)
• Version control is most useful for plain “text”-files (txt, md, tex, csv, .py, .R, .m, html, ….) where differences between versions can be “easily” visualized and multiple changes can be merged/combined automatically
• Version control works also for binary files (docx, xlsx, etc.), but it would only tell us if there is a change, but not visualize the change and the version control system will not be able to merge changes automatically

What version control systems are available?

• GIT, PerForce, Mercurial, Subversion (SVN), Bazaar, Concurrent Versions System (CVS), Monotone, ….
• We will focus on GIT in this workshop

Centralised vs. distributed version control system

• In the centralized setup, there is a single (central) copy of the project and each user will apply changes to the central copy

• In the distributed setup, each user has their own (full) copy of the project (a clone)

• SVN, PerForce, CVS are examples of centralised version control system

• GIT and Mercurial are examples of distributed version control system

GIT

• Popular version control software:
  • Distributed system
  • Free and Open Source
  • Available for Windows, Linux and Mac
  • A lot of support, infrastructure and tools available to interface with GIT:
    • graphical user interfaces (GUIs)
    • seamless integration with Integrated Development Environments (IDEs) for R, MATLAB, Python, …
    • cloud services (BitBucket, GitHub, GitLab)
• Developed by the Linus Torsvalds in 2005 to manage the development of Linux and maintained by Junio Hamano

Tracking large files with GIT

“Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise”

https://git-lfs.github.com/

• Drop-in replacement for “normal” GIT -> git lfs add vs. git add
• Files are stored “externally”, so that GIT operations can run seamlessly and fast
• Good solution for “large” files (100 MB - 2 GB)

• Alternatives:
  • git-annex
  • Do not version control large file
    • set permission to Read Only
    • version control metadata instead
  • …

Exclude files from tracking

• .gitignore: “special” file to list files and folders to intentionally not track

• Prevents files/folders from showing when running git status -> less clutter

• Can also be added by running git add -f (short for force)

• Rationale: not all files need to be version controlled

  • figures, tables, manuscript pdf generated by running code -> version control the raw data and the code to generate outputs instead
  • temporary files, compiled outputs, …

• Checkout https://www.gitignore.io/ to help identify files to ignore

Now you install GIT on your computer (5-10 min)

1. Go to https://git-scm.com/
2. Download executable in suggested directory
3. Install by following step-by-step instructions and accepts default settings

Demo

To demonstrate we are going to go through an example for writing a manuscript.

• We will track the history of our manuscript and accompanying files in git
• We will use git to see the history of our files and to undo a mistake
• We will use git to synchronize the files between multiple computers and to collaborate with other authors

Writing papers with Markdown

“Markdown is a lightweight markup language with plain text formatting syntax”

(Wikipedia)

• Markdown text (.md extension) can be converted to other formats (.docx, .pdf, .html) with Pandoc
• References can also be stored in plain text files (.bib)
• Learn more about markdown here
• Try it online

Writing papers with Markdown - example

---
title: 'HCP: A Matlab package to create beautiful heatmaps with richly annotated covariates'
authors:
 - name: Manuela Salvucci
   orcid: 0000-0001-9941-4307
   affiliation: 1
 - name: Jochen H. M. Prehn
   orcid: 0000-0003-3479-7794
   affiliation: 1
affiliations:
 - name: Centre for Systems Medicine, Department of Physiology and Medical Physics, Royal College of Surgeons in Ireland, Dublin, Ireland
   index: 1
date: 20 January 2019
bibliography: paper.bib
---

# Summary
A heatmap is a graphical technique that maps 2-dimensional matrices of numerical values to colors to provide an immediate
and intuitive visualization of the underlying patterns [@Eisen1998]. Heatmaps are often used in conjunction with cluster
analysis to re-order observations and/or features by similarity and thus, rendering common and distinct patterns more apparent.
When generating these visualizations, it is often of interest to interpret the underlying patterns in the context of other
data sources. In the field of bioinformatics, heatmaps are frequently used to visualize high-throughput and high-dimensional
datasets, such as those derived from profiling biological samples with *-omic* technologies (whole genome sequencing,
transcriptomics and proteomics). Often, biological samples (for example, patient tumour samples) are characterized at
multiple *-omic* level and it is of interest to contrast and compare patterns captured at the different molecular layers
along with their associations with other observable features (covariates). The concurrent display of continuous or
categorical covariates enriches the visualization with additional information such as group membership.

Getting started

Two main approaches to get a git repository:

• start a repository from scratch -> git init
• start by cloning an existing repository -> git clone

Practical example

Anatomy of a commit

• Includes:
  • what changed compared to the previous commit (snapshot)
    • which files are affected by changes and how
  • rationale for the change (commit message)
  • timestamp
  • “name”: unique identifier represented by SHA-1 hashes
    • for example: 4fc82ba7bb3f3a3de8ac57f16b6a926a7e60a21e
    • first 6 digits are typically sufficient to describe a commit -> shorthand version 4fc82ba
  • “parent” commit (reference to previous snapshot)
    • first commit is special (has no parent)
    • last commit is special (it is called HEAD)
• The full series of commits makes up the whole project

Guidelines on commits… size matter

• Commit small units of changes and commit often
• A good unit of change is a small, self-contained, working change
  • GOOD: data.csv, process_data_figure1.py, make_figure1.py
  • BAD: 1 commit with a day worth of work (on multiple fronts)
• Rule of thumb: commit together what you would need to undo if you later want to disregard this change

Guidelines on commits… message

Other useful GIT commands

• git rm FILENAME: delete tracked file
• git mv FILENAME1 FILENAME2: rename file from FILENAME1 to FILENAME2
• git log –follow: inspec t log (even with renaming)

I am working on it…

• git add -p FILENAME: add portions of changes you made to a file
  • preserve other changes, but they will not be captured in this commit
  • useful when you set out to make some changes, but you could not help fixing (other) unrelated stuff
• git squash: pool related commits in a meta-commit
• git stash: stash away work in progress which is in a state that is too preliminary to be committed and get back to it later
  • git stash list
  • git stash pop
  • git stash drop

Ops, I did not mean to do that…. Let’s pretend it never happened

• git commit –amend: by far the most used command
  • useful when you forgot to add a file before committing or you would like to change commit message
• git revert SHA: revert changes applied by SHA by creating a new commit
• git checkout FILENAME: undo (uncommited) changes to FILENAME
• git checkout SHA: checkout a snapshot where all was good
• git reset: undo changes, degree of annihilation depends on flags (–soft vs. –hard), be careful

Ops, something went terribly terribly wrong, at some point in the past

• git show : inspect (suspicious) commit
• git blame: when and who changed/broke this?
• git bisect: run binary search to identify when problem was introduced
  • extremelly useful command, a life-saver
  • requires knowing what right vs. wrong means (unit tests, ground truth, …)

GIT SubModules… russian dolls repositories

“It often happens that while working on one project, you need to use another project from within it. Perhaps it’s a library that a third party developed or that you’re developing separately and using in multiple parent projects. A common issue arises in these scenarios: you want to be able to treat the two projects as separate yet still be able to use one from within the other.”

https://git-scm.com/book/en/v2/Git-Tools-Submodules

Exercise 1 (20 min)

1. Create a folder named “christmas_repo”
2. Open GIT bash, verify the installation and configure GIT
3. Initialize a GIT repository in the folder
4. Create a text file (“wish_list.md”) with 3 gifts you wish to receive for Christmas
5. Add and commit the wish list file to GIT
6. Edit the wish list file, and add 2 more presents
7. Use GIT to check the difference between the current and previous version
8. Commit the updated file
9. Create a file (“recipients.md”) with a list of people you plan to buy gifts for
10. Check the status of the GIT repository
11. Add and commit the new file to GIT
12. Create a file (“past_gifts.md”) with a list of what gifts you gave last year
13. Maybe you remembered a few more people you would like to give gifts to. Add them to “recipients.md”
14. Add the new file and commit “past_gifts.md” and “recipients.md” to GIT
15. Look at the GIT history
16. Revert the change that added more presents to the wish list in step 5
17. Play around with doing more changes and commits

GIT support tools

• Graphical user interface (GUIs)
• Integration with software Integrated Development Environments (IDEs) for R, MATLAB, Python, …
• Cloud services (BitBucket, GitHub, GitLab)

GIT graphical user interface (GUI)

GIT graphical user interface (GUI)

GIT graphical user interface (GUI)

GIT integration with software Integrated Development Environments (IDE)

Cloud services that support GIT… “social” coding

• Servers that can host a copy of your repository
• Useful as a backup
• Can make synchronization and collaboration easier
• Free plans available
• Most popular alternatives:

• Other alternatives include Crucible, AWS CodeCommit, CodeCommit, ….

Cloud services that support GIT… “social” coding

Comparison of key features in free plans from GitHub, BitBucket and GitLab

• Similar products, select the one that suits best your needs

Using branches and tags

• Implicit branches:
  • The collaborator has their own temporary unnamed branch when working on the files simultaneous to other work. The branches get merged when they pull and push.
  • Similarly you can get temporary unnamed branches when doing work on two computers
• Explicit branches:
  • You can create a named branch for changes that you want to keep separate from your main work (master branch)
  • The branches optionally be can be be merged later
• Tags:
  • You can name a specific revision (snapshot), like a milestone, to keep track of it
  • For example to have a record of what specific revision was used for:
    • a paper submission
    • conference talk

Exercise 2 (20 min)

1. Create a BitBucket account
2. Push your “christmas_repo” from exercise 1 to BitBucket
3. Add more past_gifts and push to BitBucket
4. Update recipients through the BitBucket web-interface, and pull to your local machine
5. Collaborate in pairs

Take home messages

• Version control with GIT helps keep your file history organized
• Light weight: minimum effort required
• If you are not comfortable with using the command line, download a GUI or use GIT from your IDE
• Commit early and often
• Write good commit messages - future you will appreciate it
• Useful for backups and for collaboration

Thanks & Questions

• Get int touch: manuelasalvucci@rcsi.ie

• Presentation, CheatSheet, Handout and Solution: https://bitbucket.org/manuela_s/git_workshop/downloads/

• Workshop repo: https://bitbucket.org/manuela_s/git_workshop

• Useful resources

  • https://git-scm.com/
  • https://git-scm.com/book/en/v2
  • https://try.github.io/
  • https://stackoverflow.com/questions/tagged/git
  • https://sethrobertson.github.io/GitBestPractices/