Best resources

We are going to make a repo for a wiki. Along the way we are going to watch what Git is doing.

Why is Git so hard?

In my opinion, the reason Git is so hard is that the commands (git add, git commit, git checkout, etc.), have names whose semantics seem to be mapped onto the history of a variety of versioning tools, especially Subversion. The actual git commands, however, have behaviors with very precise meanings regarding specific git entities. For instance, git add means: Compress the files and add them to a git-managed directory, and add references to those files to the binary "index" file. Even when the official Git docs use the language of the implementation (for instance, we are told by git help add that git-add means "Add file contents to the index") it isn't very clear what "index" means in this context. A lot would be gained by banning words such as "staging area" and "cache" and just sticking with "index."

To get started

Create a couple of extra terminal tabs, and in the second tab,

watch-do $HOME/src/wiki $HOME/src/wiki "tree -t -r -C -a ."

This lets us observe the directory hierarchy Git creates to manage the state of our repository. [To set up watch-do see the end of this document.]

Create the wiki

In the first tab,

cd $HOME
cd src
rm -fR wiki
mkdir wiki
cd wiki

Create a file and initialize the repository

echo "### Welcome to the wiki" >home.md
git init

What are these directories and files?

home.md (and everything else except for .git/) is your content in what Git calls the "working directory."

The .git/ directory holds the entire history of the repository. Within that,

Name What
index (You won't see this until something has been added.) A binary file with a sorted list, each item being a hash of a blob object and its permission. The index becomes the basis for the next "tree" object that will be committed.
refs/tags/
refs/heads/
objects/
objects/pack/
objects/info/
config A text file with configuration information for the repository.
HEAD An ordinary text file that tracks a reference to whatever branch is the current head.
description (You may not see this) If present, contains a name for the repository. Used by the GitWeb program.
hooks/ (You may not see this) Contains files that define behaviors that are triggered at various times during Git operations.
info/ (You may not see this) Contains a global exclude file for patterns you don’t want to track in a .gitignore file

Add a file and watch the index

git add home.md

In a third tab:

watch-do $HOME/src/wiki $HOME/src/wiki/.git/index "git ls-files --stage"

The concept: We are adding home.md to the "index." The index is a place where changes accumulate that you will eventually commit to your repo. Think of the index, then, as "the next commit." (Ryan Tomayko has a nice post on this, though he calls the index "the next patch" -- see http://2ndscale.com/rtomayko/2008/the-thing-about-git.) This is the place where you set up your changes properly before making them official.

A directory has been added in .git/objects/ called 81 and within that, a file called d4aaeff611587c378d25a90e1d2f178b484727. Additionally, .git/index has changed. The idea is that objects/ is effectively a database of compressed files, and the file names are based on the hashes, which are based on the contents. By this means, git is a content-addressable database. Notice that the contents of objects/ has no information regarding filenames; it's all about content.

Since objects/ just contained compressed objects, there needs to be a structure that manages the current state: That's what the index is for: A list of hashes pointing into the objects/ store, with the names used when you added.

Indeed, if you added a second identical file with a different name with echo "### Welcome to the wiki" >home2.md you will find that what's in objects/ doesn't change. If you then add that file with git add home2.md the only change you will find is in the index, which will reference the name object, but with a different name.

Again, the exact sequence:

  1. A directory 81/ is created. Within 81/ a file d4aaeff611587c378d25a90e1d2f178b484727 is created. [The path 81/d4aaeff611587c378d25a90e1d2f178b484727 is a hash of the content of the home.md file, which you can get with git hash-object home.md (try it). git hash-object is pretty simple; here's a Ruby version of how to get a git-compatible hash for a blob: https://gist.github.com/jgn/b3dddf091db54b2d719fa02e996dc5a3

  2. The compressed contents of the file home.md are put the file 81/d4aaeff611587c378d25a90e1d2f178b484727 -- to see this, type git show 81d4aa (81d4aa is the opening part of the hash). Or you can manually uncompress it with

    ruby -rzlib -e 'print Zlib::Inflate.new.inflate(STDIN.read)' < .git/objects/81/d4aaeff611587c378d25a90e1d2f178b484727

    The git way to see the content (-p means "pretty print"):

    git cat-file -p 81d4aaeff611587c378d25a90e1d2f178b484727

    And to see the type . . .

    git cat-file -t 81d4aaeff611587c378d25a90e1d2f178b484727

  3. A binary file .git/index is added. An entry is added to .git/index with the mode of the file, the hash, and the filename. To see this, type git ls-files --stage (which is what is being shown in your third terminal tab, if you've been following along. Also, git ls-files --cached does the same thing as --stage but just lists the filenames only -- --cached is the default.)

Make the first commit to the repo

Commit everything that has been added (everything in the index)

git commit -m "Initial commit"

Observations

  1. The index stays the same

  2. There are now a total of three objects; two are new. One is a tree.

    $ git cat-file -t 8dfcc6
tree
$ git cat-file -p 8dfcc6
100644 blob 81d4aaeff611587c378d25a90e1d2f178b484727	home.md

    It contains a pointer to the blob, and also includes a name, so that when the tree is applied, the blob will get a name.

    All of the paths are relative to the root of the project.

  3. The other new object is a commit. Note that the hash will be different from what I'm using, because the hash is based on data that is different on your system, namely your name/email and the date/time of the commit, which are included in the commit messages and change the hash.

    $ git cat-file -t 0e12bc
commit
$  git cat-file -p 0e12bc
tree 8dfcc6e23dec849d71d7428290cf66aab644a22f
author John Norman <john@iorahealth.com> 1475667520 -0500
committer John Norman <john@iorahealth.com> 1475667520 -0500

Initial commit

    The commit points to the tree. The tree points to the blob.

Now add a second file

In the first tab,

echo "Software we like: git, atom" > software.md
git add software.md

You should see a new item ed/91a30546d771a1c22ab2d1e846bdea6b52a624 in the objects/ directory, and a new line added to the index.

Let's check our status

In the first tab,

git status

Notice that the heading before the files in the index says: "Changes to be committed" and that we are told that the command to "unstage" a file is to type git rm --cached <file> .... Thus already we have a proliferation of terms regarding the files that are compressed in .git/objects and listed in .git/index -- the terms are (1) changes to be committed; (2) "staged"; (3) "cached." These all mean the same thing. Nowhere is the word "index" mentioned, even though it is in the docs if you type git help add. But we know what all this stuff means: git status is telling us what is in .git/objects and .git/index.

Let's add a directory and another file and see what happens

Let's create an images/ directory and put something in there:

mkdir images
curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg
git add images

Notice that we have added the compressed image file d2/dd63f0a1e8f1c07f4c33e5af782a60fda3546f to .git/objects as well as an entry in .git/index. The newly added directory (images/) is only indicated as the directory for the wat.jpg file (there is no separate addition for the index for images/ itself. Also, if we type

git rm --cached images/wat.jpg

the lines is taken out of the index, but the compressed file remains in objects/. [NOTE: The fact that the compressed file remains in objects/ provides you some safety should you remove files from the index and from your working copy. But if you have not "pushed" your repo to a remote, deleting stuff from the objects/ hierarchy can result in loss-of-data. In short, leave objects/ alone, except for manipulations via git commands.]

Do this:

rm images/wat.jpg
git add images

are not files there, and . . . nothing happens to the index. This is because Git cannot represent empty directories in the index (staging area) -- See https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F.

At this point, we removed images/wat.jpg from the index AND from our working directory. And yet we can still see the compressed file in the .git/objects/ hierarchy under d2/dd63f0a1e8f1c07f4c33e5af782a60fda3546f which we can peek at with git cat-file -p d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f | imgcat. Thus the arbiter of what Git knows is .git/index not the assortment of things in the .git/objects directory. In other words, you may find that you've removed a file from your working directory, git status doesn't know about it, but the file is still safe in the .git/objects directory.

Now, re-do the curl command to get the image,

curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg

and git add images. What's in .git/objects doesn't change (the image was already saved there) but the index changes.

As you can see, the fact that wat.jpg lives in images/ is only known to the index. You can get a feel for this by removing images/wat.jpg from the index with git rm --cached images/wat.jpg, then adding a new directory images2 and then curl'ing a new wat.jpg, and then adding images2/:

git rm --cached images/wat.jpg
mkdir images2
curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images2/wat.jpg
git add images2

You will see that only the index changes -- the content has already been hashed into the objects hierarchy. Reverse the experiment with . . .

git rm --cached images2/wat.jpg
rm -fR images2
git add images

Finally, before we commit, let's add one more file to the working directory that will help us understand the relationship between git status, the index, and the last commit.

echo -e '### Image list\n\n* more images.md ![](images/wat.jpg)' >images.md

Now let's see what happens if we do a commit

Commit files with

git commit -m "Add pages and images"

You will notice that this immediately creates some new items in objects/, a new directory logs/ with some items, and a new directory refs/ with some items. What has happened?

First, the new objects. There are three of them. Use git cat-file -p and git cat-file -t to inspect them.

  1. .git/objects/f9/c298b9e62cecfc65ede840f48fa025347909fe is the tree with a blob entry for wat.jpg. Doing git cat-file -p f9c298b9e62cecfc65ede840f48fa025347909fe we see

    100644 blob d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f	wat.jpg

    Do we know the name of the tree? No. Do we know wat.jpg is kept in a tree? Yes.

  2. .git/objects/11/dd91adca91dc2445dfcee4d3418e8ad219db7c is another tree created from the index that represents the current state of the whole project. git cat-file -t 11dd91 tells us that it's a "tree," and git cat-file -p 11dd91 shows that it looks like this:

    100644 blob 81d4aaeff611587c378d25a90e1d2f178b484727	home.md
040000 tree f9c298b9e62cecfc65ede840f48fa025347909fe	images
100644 blob ed91a30546d771a1c22ab2d1e846bdea6b52a624	software.md

  3. .git/objects/31/8ff038bef2f31f61729fdebaaa02e8415c659a (your name will be different; make a note of the hash you have) is a commit representing the new root for your repo. git cat-file -t tells us that it's a "commit."

    The contents for mine are

    tree 11dd91adca91dc2445dfcee4d3418e8ad219db7c
author John Norman <john@iorahealth.com> 1475414291 -0500
committer John Norman <john@iorahealth.com> 1475414291 -0500

Add pages and images

Now we can trace this backwards. We want to do git cat-file -t and git cat-file -p on each of the following SHAs:

  • 318ff038bef2f31f61729fdebaaa02e8415c659a commit
  • 11dd91adca91dc2445dfcee4d3418e8ad219db7c tree: references blob home.md, tree images [f9c298b9e62cecfc65ede840f48fa025347909fe], and blob software.md)
  • f9c298b9e62cecfc65ede840f48fa025347909fe tree: references blob wat.jpg
  • d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f blob: for wat.jpg

Meanwhile, the contents of .git/index do not change. Tthis is because .git/index still represents what is being tracked, which, after a commit, is whatever is in the last commit -- i.e., git diff --cached returns nothing. git status will compare the state of the working directory with the index, and will find that there is one file that that is "untracked": images.md.

Making our entire repo with commands manipulating git objects directly

Git provides for manipulating the object database directly, without using the commands to add to the index from your working directory. Here's the complete list of commands we used to set up our repo, add files, and do the two commits:

# Set up our directory and initialize the repo
cd $HOME
cd src
rm -fR wiki
mkdir wiki
cd wiki
echo "### Welcome to the wiki" >home.md
git init

# Add a file and commit
git add home.md
git commit -m "Initial commit"

# Create another file and add it
echo "Software we like: git, atom" > software.md
git add software.md

# Create an image directory, put a file in there, add the directory and image, and commit
mkdir images
curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg
git add images
git commit -m "Add pages and images"

Now let's do all of this without creating any files in the working directory. We are going to add stuff to the index and objects/ hierarchy directly.

First, create our directory and initialize our repo.

cd $HOME
cd src
rm -fR wiki
mkdir wiki
cd wiki
git init

We will now use the hash-object command with -w to create our object. Notice that no file is created in the working directory.

echo "### Welcome to the wiki" | git hash-object -w --stdin
# returns hash: 81d4aaeff611587c378d25a90e1d2f178b484727

This only created the compressed object in object/ -- we don't yet have anything in the index. To put a reference to our object in the index, we use update-index:

git update-index --add --cacheinfo 100644 81d4aaeff611587c378d25a90e1d2f178b484727 home.md

The next thing we want to do is commit this object so that it is referenced from HEAD. Recall that the commit object needs to reference a tree object. Let's make a tree object from the current contents of the index. This command will report the hash for the newly-created tree.

git write-tree
# returns hash: 8dfcc6e23dec849d71d7428290cf66aab644a22f

The hash value should look familiar from above.

Now we will make the commit object with commit-tree:

echo "Initial commit" | git commit-tree 8dfcc6
# returns hash (yours will be different): 256b5975726098c52222d01787688681b959a991
# make a note of this hash: You will use it later

At this point we can do git log if we pass in the commit hash

git log 256b5975726098c52222d01787688681b959a991

Add software.md as above, but without creating a file in the working directory:

echo "Software we like: git, atom" | git hash-object -w --stdin
# hash = ed91a30546d771a1c22ab2d1e846bdea6b52a624
git update-index --add --cacheinfo 100644 ed91a30546d771a1c22ab2d1e846bdea6b52a624 software.md

Add the binary for our image:

curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg | git hash-object -w --stdin
# returns hash: d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f

Add our file to the index and then make a new tree:

git update-index --add --cacheinfo 100644 d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f images/wat.jpg
git write-tree
# returns hash: 11dd91adca91dc2445dfcee4d3418e8ad219db7c

And commit. Note the additional -p: This specifies the prior commit so that this commit references the first one.

echo "Add pages and images" | git commit-tree 11dd91 -p 256b59
# returns hash (yours will be different): 914b2fef46cd6cd1c58c90def3a751d2fb738e39

And now do git log with this new hash . . .

git log 914b2f

Now let's create the master branch, and look at the log for that branch:

git update-ref refs/heads/master 914b2fef46cd6cd1c58c90def3a751d2fb738e39
# NOTE: Basically the same as . . .
#   echo "914b2fef46cd6cd1c58c90def3a751d2fb738e39" > .git/refs/heads/master
git log --pretty=oneline master

What's in refs/?

So now that we've committed something, we have a reference to the tip of our master branch. The SHA for the tip of our master branch is stored in the ordinary file .git/refs/heads/master -- the content is 318ff038bef2f31f61729fdebaaa02e8415c659a , which is just the last thing we did with our repo: The last commit. Can we add another head? Yep. Do this:

git checkout -b jn-foobar

Now we get a new file .git/refs/head/jn-foobar. And its content is also 318ff038bef2f31f61729fdebaaa02e8415c659a, the SHA for our last commit. In other words, master and jn-foobar point to the same thing; meanwhile, .git/HEAD now is jn-foobar.

We can play around with this endlessly, checking out master, then jn-foobar and see the content of .git/HEAD change accordingly.

What's in logs/?

As you work, git will track your commits and keep them in various places in logs/. It is possible to reset your repo and lose references to hashes. But typically the compressed data is still somewhere in objects/ -- you just need to be able to find the hash for the lost file. By consulting the logs/ via the git reflog command, you can sometimes recover lost work. By default, the log goes back 90 does for commits in your current history, and 30 days for commits that are no longer relevant.

In my experience, if you find that you have to deal with the reflog, get help from someone who's dealt with it before.

What does it mean to do git add?

When do you git add, you add a compressed version of the file(s) to the objects/ directory (if it isn't there already), and add (a) line(s) to the index.

What's the opposite of git add?

git rm --cached filename. Note, this will not delete the compressed binary version from the objects/ directory.

What is the index? What is the "staging area"?

git keeps a list of all of the files for the "next" commit. This list is called the index. It's also called the "staging area." It's also called the "cache." To get a list of all of the files in the index, type git ls-files. To get the same list, along with the hashes for each file and the file mode, do git ls-files --stage.

It's all the files currently being tracked since the last commit, including removed files. So, for instance, if you clone a large repo such as what's at git@github.com:IoraHealth/IoraHealth.git and do git ls-files | wc -l, as of October 2016 you'll get 3913, which is the same the number of files in the repo not including what's in .git/ (compare find . -type f -not -path "./.git/*" | wc -l).

What does git check -b branch do?

What does git commit do?

What does git reset do?

And what is the difference between git reset --hard and regular git reset?

What is the difference between git rm --cached [file] and git reset HEAD [file] and git reset [file]?

watch-do

On the Mac,

brew install tree fswatch

Create a file like this, call it watch-do and put it in your patch

#!/bin/bash

# usage
#   watch-do <dir> <path-to-watch> <command-to-run>

DIR=$1
WATCH=$2
COMMAND="clear && cd $DIR && $3"
bash -c "$COMMAND"
fswatch -l 0.1 -o "$WATCH" | xargs -n1 bash -c "$COMMAND"