- How it really works
We are going to make a repo for a wiki. Along the way we are going to watch what Git is doing.
In my opinion, the reason Git is so hard is that the commands (
git checkout, etc.),
have names whose semantics seem to be mapped onto the history of a variety of versioning tools, especially Subversion. The actual git commands, however, have behaviors with very precise meanings regarding specific git entities. For instance,
git add means: Compress the files and add them to a git-managed directory, and add references to those files to the binary "index" file. Even when the official Git docs use the language of the implementation (for instance, we are told by
git help add that
git-add means "Add file contents to the index") it isn't very clear what
"index" means in this context. A lot would be gained by banning words such as "staging area" and "cache" and just sticking with "index."
Create a couple of extra terminal tabs, and in the second tab,
watch-do $HOME/src/wiki $HOME/src/wiki "tree -t -r -C -a ."
This lets us observe the directory hierarchy Git creates to manage the state of our repository. [To set up
watch-do see the end of this document.]
In the first tab,
cd $HOME cd src rm -fR wiki mkdir wiki cd wiki
echo "### Welcome to the wiki" >home.md git init
What are these directories and files?
home.md (and everything else except for
.git/) is your content in what Git calls the "working directory."
.git/ directory holds the entire history of the repository. Within that,
||(You won't see this until something has been added.) A binary file with a sorted list, each item being a hash of a blob object and its permission. The index becomes the basis for the next "tree" object that will be committed.|
||A text file with configuration information for the repository.|
||An ordinary text file that tracks a reference to whatever branch is the current head.|
||(You may not see this) If present, contains a name for the repository. Used by the GitWeb program.|
||(You may not see this) Contains files that define behaviors that are triggered at various times during Git operations.|
||(You may not see this) Contains a global exclude file for patterns you don’t want to track in a .gitignore file|
git add home.md
In a third tab:
watch-do $HOME/src/wiki $HOME/src/wiki/.git/index "git ls-files --stage"
The concept: We are adding
home.md to the "index." The index is a place where changes accumulate
that you will eventually commit to your repo. Think of the index, then, as "the next commit." (Ryan Tomayko
has a nice post on this, though he calls the index "the next patch" -- see http://2ndscale.com/rtomayko/2008/the-thing-about-git.) This is the place where you set up your changes properly before making them official.
A directory has been added in
81 and within that, a file
.git/index has changed. The idea
objects/ is effectively a database of compressed files, and the file names are based
on the hashes, which are based on the contents. By this means,
git is a content-addressable
database. Notice that the contents of
objects/ has no information regarding filenames; it's
all about content.
objects/ just contained compressed objects, there needs to be a structure
that manages the current state: That's what the
index is for: A list of hashes pointing into
objects/ store, with the names used when you added.
Indeed, if you added a second identical file with a different name with
echo "### Welcome to the wiki" >home2.md you will find that what's in
objects/ doesn't change.
If you then add that file with
git add home2.md the only change you will find is in the index,
which will reference the name object, but with a different name.
Again, the exact sequence:
81/is created. Within
d4aaeff611587c378d25a90e1d2f178b484727is created. [The path
81/d4aaeff611587c378d25a90e1d2f178b484727is a hash of the content of the
home.mdfile, which you can get with
git hash-object home.md(try it).
git hash-objectis pretty simple; here's a Ruby version of how to get a git-compatible hash for a blob: https://gist.github.com/jgn/b3dddf091db54b2d719fa02e996dc5a3
The compressed contents of the file
home.mdare put the file
81/d4aaeff611587c378d25a90e1d2f178b484727-- to see this, type
git show 81d4aa(
81d4aais the opening part of the hash). Or you can manually uncompress it with
ruby -rzlib -e 'print Zlib::Inflate.new.inflate(STDIN.read)' < .git/objects/81/d4aaeff611587c378d25a90e1d2f178b484727
The git way to see the content (
-pmeans "pretty print"):
git cat-file -p 81d4aaeff611587c378d25a90e1d2f178b484727
And to see the type . . .
git cat-file -t 81d4aaeff611587c378d25a90e1d2f178b484727
A binary file
.git/indexis added. An entry is added to
.git/indexwith the mode of the file, the hash, and the filename. To see this, type
git ls-files --stage(which is what is being shown in your third terminal tab, if you've been following along. Also,
git ls-files --cacheddoes the same thing as
--stagebut just lists the filenames only --
--cachedis the default.)
Commit everything that has been added (everything in the index)
git commit -m "Initial commit"
The index stays the same
There are now a total of three objects; two are new. One is a tree.
$ git cat-file -t 8dfcc6 tree $ git cat-file -p 8dfcc6 100644 blob 81d4aaeff611587c378d25a90e1d2f178b484727 home.md
It contains a pointer to the blob, and also includes a name, so that when the tree is applied, the blob will get a name.
All of the paths are relative to the root of the project.
The other new object is a commit. Note that the hash will be different from what I'm using, because the hash is based on data that is different on your system, namely your name/email and the date/time of the commit, which are included in the commit messages and change the hash.
$ git cat-file -t 0e12bc commit $ git cat-file -p 0e12bc tree 8dfcc6e23dec849d71d7428290cf66aab644a22f author John Norman <firstname.lastname@example.org> 1475667520 -0500 committer John Norman <email@example.com> 1475667520 -0500 Initial commit
The commit points to the tree. The tree points to the blob.
In the first tab,
echo "Software we like: git, atom" > software.md git add software.md
You should see a new item
ed/91a30546d771a1c22ab2d1e846bdea6b52a624 in the
objects/ directory, and a new line added to the index.
In the first tab,
Notice that the heading before the files in the index says: "Changes to be committed" and that we are told that the command to "unstage" a file is to type
git rm --cached <file> .... Thus already we have a proliferation of terms regarding the files that are compressed in
.git/objects and listed in
.git/index -- the terms are (1) changes to be committed; (2) "staged"; (3) "cached." These all mean the same thing. Nowhere is the word "index" mentioned, even though it is in the docs if you type
git help add. But we know what all this stuff means:
git status is telling us what is in
Let's create an
images/ directory and put something in there:
mkdir images curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg git add images
Notice that we have added the compressed image file
.git/objects as well as an entry in
.git/index. The newly added directory (
is only indicated as the directory for the
wat.jpg file (there is no separate addition for the index for
images/ itself. Also, if we
git rm --cached images/wat.jpg
the lines is taken out of the index, but the compressed file remains in
objects/. [NOTE: The fact that the compressed file remains in
objects/ provides you some safety should you remove files from the index and from your working copy. But if you have not "pushed" your repo to a remote, deleting stuff from the
objects/ hierarchy can result in loss-of-data. In short, leave
objects/ alone, except for manipulations via git commands.]
rm images/wat.jpg git add images
are not files there, and . . . nothing happens to the index. This is because Git cannot represent empty directories in the index (staging area) -- See https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F.
At this point, we removed
images/wat.jpg from the index AND from our working directory. And yet we can still see the compressed
file in the
.git/objects/ hierarchy under
d2/dd63f0a1e8f1c07f4c33e5af782a60fda3546f which we can peek at with
git cat-file -p d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f | imgcat. Thus the arbiter of what Git knows is
.git/index not the assortment of
things in the
.git/objects directory. In other words, you may find that you've removed a file from your working directory,
git status doesn't know about it, but the file is still safe in the
Now, re-do the curl command to get the image,
curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg
git add images. What's in
.git/objects doesn't change (the image was already saved there) but
the index changes.
As you can see, the fact that
wat.jpg lives in
images/ is only known to the index. You can get a feel for this
images/wat.jpg from the index with
git rm --cached images/wat.jpg, then adding a new directory
and then curl'ing a new
wat.jpg, and then adding
git rm --cached images/wat.jpg mkdir images2 curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images2/wat.jpg git add images2
You will see that only the index changes -- the content has already been hashed into the objects hierarchy. Reverse the experiment with . . .
git rm --cached images2/wat.jpg rm -fR images2 git add images
Finally, before we commit, let's add one more file to the working directory that will help us understand the relationship
git status, the index, and the last commit.
echo -e '### Image list\n\n* more images.md !(images/wat.jpg)' >images.md
Commit files with
git commit -m "Add pages and images"
You will notice that this immediately creates some new items in
objects/, a new directory
logs/ with some items, and a new directory
refs/ with some items. What has happened?
First, the new objects. There are three of them. Use
git cat-file -p and
git cat-file -t to inspect them.
.git/objects/f9/c298b9e62cecfc65ede840f48fa025347909feis the tree with a blob entry for
git cat-file -p f9c298b9e62cecfc65ede840f48fa025347909fewe see
100644 blob d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f wat.jpg
Do we know the name of the tree? No. Do we know
wat.jpgis kept in a tree? Yes.
.git/objects/11/dd91adca91dc2445dfcee4d3418e8ad219db7cis another tree created from the index that represents the current state of the whole project.
git cat-file -t 11dd91tells us that it's a "tree," and
git cat-file -p 11dd91shows that it looks like this:
100644 blob 81d4aaeff611587c378d25a90e1d2f178b484727 home.md 040000 tree f9c298b9e62cecfc65ede840f48fa025347909fe images 100644 blob ed91a30546d771a1c22ab2d1e846bdea6b52a624 software.md
.git/objects/31/8ff038bef2f31f61729fdebaaa02e8415c659a(your name will be different; make a note of the hash you have) is a commit representing the new root for your repo.
git cat-file -ttells us that it's a "commit."
The contents for mine are
tree 11dd91adca91dc2445dfcee4d3418e8ad219db7c author John Norman <firstname.lastname@example.org> 1475414291 -0500 committer John Norman <email@example.com> 1475414291 -0500 Add pages and images
Now we can trace this backwards. We want to do
git cat-file -t and
git cat-file -p on each of the following SHAs:
11dd91adca91dc2445dfcee4d3418e8ad219db7ctree: references blob home.md, tree images [
f9c298b9e62cecfc65ede840f48fa025347909fe], and blob software.md)
f9c298b9e62cecfc65ede840f48fa025347909fetree: references blob wat.jpg
d2dd63f0a1e8f1c07f4c33e5af782a60fda3546fblob: for wat.jpg
Meanwhile, the contents of
.git/index do not change. Tthis is because
.git/index still represents what is
being tracked, which, after a commit, is whatever is in the last commit -- i.e.,
git diff --cached returns nothing.
git status will compare the state of the working directory with the index, and will find that there is one file that that is "untracked":
Git provides for manipulating the object database directly, without using the commands to add to the index from your working directory. Here's the complete list of commands we used to set up our repo, add files, and do the two commits:
# Set up our directory and initialize the repo cd $HOME cd src rm -fR wiki mkdir wiki cd wiki echo "### Welcome to the wiki" >home.md git init # Add a file and commit git add home.md git commit -m "Initial commit" # Create another file and add it echo "Software we like: git, atom" > software.md git add software.md # Create an image directory, put a file in there, add the directory and image, and commit mkdir images curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg -o images/wat.jpg git add images git commit -m "Add pages and images"
Now let's do all of this without creating any files in the working directory. We are going to
add stuff to the index and
objects/ hierarchy directly.
First, create our directory and initialize our repo.
cd $HOME cd src rm -fR wiki mkdir wiki cd wiki git init
We will now use the
hash-object command with
-w to create our object. Notice that no file is
created in the working directory.
echo "### Welcome to the wiki" | git hash-object -w --stdin # returns hash: 81d4aaeff611587c378d25a90e1d2f178b484727
This only created the compressed object in
object/ -- we don't yet have anything in the index. To
put a reference to our object in the index, we use
git update-index --add --cacheinfo 100644 81d4aaeff611587c378d25a90e1d2f178b484727 home.md
The next thing we want to do is commit this object so that it is referenced from
HEAD. Recall that
commit object needs to reference a
tree object. Let's make a
tree object from the current
contents of the index. This command will report the hash for the newly-created tree.
git write-tree # returns hash: 8dfcc6e23dec849d71d7428290cf66aab644a22f
The hash value should look familiar from above.
Now we will make the commit object with
echo "Initial commit" | git commit-tree 8dfcc6 # returns hash (yours will be different): 256b5975726098c52222d01787688681b959a991 # make a note of this hash: You will use it later
At this point we can do
git log if we pass in the commit hash
git log 256b5975726098c52222d01787688681b959a991
software.md as above, but without creating a file in the working directory:
echo "Software we like: git, atom" | git hash-object -w --stdin # hash = ed91a30546d771a1c22ab2d1e846bdea6b52a624 git update-index --add --cacheinfo 100644 ed91a30546d771a1c22ab2d1e846bdea6b52a624 software.md
Add the binary for our image:
curl -s http://i2.kym-cdn.com/photos/images/original/000/173/575/25810.jpg | git hash-object -w --stdin # returns hash: d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f
Add our file to the index and then make a new tree:
git update-index --add --cacheinfo 100644 d2dd63f0a1e8f1c07f4c33e5af782a60fda3546f images/wat.jpg git write-tree # returns hash: 11dd91adca91dc2445dfcee4d3418e8ad219db7c
And commit. Note the additional
-p: This specifies the prior commit so that this commit references
the first one.
echo "Add pages and images" | git commit-tree 11dd91 -p 256b59 # returns hash (yours will be different): 914b2fef46cd6cd1c58c90def3a751d2fb738e39
And now do git log with this new hash . . .
git log 914b2f
Now let's create the master branch, and look at the log for that branch:
git update-ref refs/heads/master 914b2fef46cd6cd1c58c90def3a751d2fb738e39 # NOTE: Basically the same as . . . # echo "914b2fef46cd6cd1c58c90def3a751d2fb738e39" > .git/refs/heads/master git log --pretty=oneline master
So now that we've committed something, we have a reference to the tip of our master branch. The SHA for the tip of our
master branch is stored in the ordinary file
.git/refs/heads/master -- the content is
318ff038bef2f31f61729fdebaaa02e8415c659a , which
is just the last thing we did with our repo: The last commit. Can we add another head? Yep. Do this:
git checkout -b jn-foobar
Now we get a new file
.git/refs/head/jn-foobar. And its content is also
318ff038bef2f31f61729fdebaaa02e8415c659a, the SHA
for our last commit. In other words,
jn-foobar point to the same thing; meanwhile,
.git/HEAD now is
We can play around with this endlessly, checking out
jn-foobar and see the content of
.git/HEAD change accordingly.
As you work, git will track your commits and keep them in various places in
logs/. It is possible to reset
your repo and lose references to hashes. But typically the compressed data is still somewhere in
you just need to be able to find the hash for the lost file. By consulting the
logs/ via the
git reflog command, you can sometimes recover lost work. By default, the log goes back 90 does for commits in your current history,
and 30 days for commits that are no longer relevant.
In my experience, if you find that you have to deal with the reflog, get help from someone who's dealt with it before.
When do you
git add, you add a compressed version of the file(s) to the
objects/ directory (if it isn't there already), and add (a) line(s) to the
git rm --cached filename. Note, this will not delete the compressed binary version from the
git keeps a list of all of the files for the "next" commit. This list is called the
index. It's also called the "staging area." It's also called the "cache." To get a list of all of the files in the index, type
git ls-files. To get the same list, along with the hashes for each file and the file mode, do
git ls-files --stage.
It's all the files currently being tracked since the last commit, including removed files. So, for instance, if you clone a large repo such as what's at
firstname.lastname@example.org:IoraHealth/IoraHealth.git and do
git ls-files | wc -l, as of October 2016 you'll get 3913, which is the same the number of files in the repo not including what's in
find . -type f -not -path "./.git/*" | wc -l).
What is the difference between git rm --cached [file] and git reset HEAD [file] and git reset [file]?
On the Mac,
brew install tree fswatch
Create a file like this, call it
watch-do and put it in your patch
#!/bin/bash # usage # watch-do <dir> <path-to-watch> <command-to-run> DIR=$1 WATCH=$2 COMMAND="clear && cd $DIR && $3" bash -c "$COMMAND" fswatch -l 0.1 -o "$WATCH" | xargs -n1 bash -c "$COMMAND"