How to Remove a Large File from Commit History in Git?

Eric Wei
3 Min To Read
07 Dec, 2021
- Unity

If you develop a game in Unity and save assets like images or audio files on Github without using Git FS, your repo’s size might become bigger and bigger when you make any changes to Github. Given the fact that Git History will record all the changes of codes, images, audio files, or even binary files.

This could make the building time of games longer and delay your work process because it will take too much time in the CI/CD Pipeline while downloading assets from Github and even building the games.

How to clean up those large files in Git Commit History?

In this post, we will provide a guide and practical example to resolve this problem.

Step 1: Check the Size of Git History

du -hs .git/objects

Step 2: Create a script to find larges files

Create git_find_big.sh script

vim git_find_big.sh


#!/bin/bash  
#set -x


# Shows you the largest objects in your repo's pack file.  
# Written for osx.  
#  
# @see http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/  
# @author Antony Stubbs


# set the internal field spereator to line break, so that we can iterate easily over the verify-pack output  
IFS=$'\n';


# list all objects including their size, sort by size, take top 10  
objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v chain | sort -k3nr | head`


echo "All sizes are in kB's. The pack column is the size of the object, compressed, inside the pack file."


output="size,pack,SHA,location"  
for y in $objects  
do  
    # extract the size in bytes  
    size=$((`echo $y | cut -f 5 -d ' '`/1024))  
    # extract the compressed size in bytes  
    compressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))  
    # extract the SHA  
    sha=`echo $y | cut -f 1 -d ' '`  
    # find the objects location in the repository tree  
    other=`git rev-list --all --objects | grep $sha`  
    #lineBreak=`echo -e "\n"`  
    output="${output}\n${size},${compressedSize},${other}"  
done


echo -e $output | column -t -s ', '

Step 3: Find Top 10 Large files

Run

./git_find_big.sh

Step 4. Install git-filter-repo

Run

pip3 install git-filter-repo

Step 5: Remove large files from the .git history

You can get the Top 10 large file names from Step 3.

Then run below to delete them in .git history.

git filter-repo --force --path <large file path> --invert-paths

This command will rewrite your .git history and delete it.

Step 6: Clean Unused Fils

git reflog expire --expire=now --all


git gc --prune=now

Step 7. Add Git Remote URL and Push Back the Result

# Add Your Github URL  
git remote add origin <Your Github URL>


# Push the result to all the branches  
git push --all --force


# Push the result to origin repo  
git push -u origin --all  
git push -u origin --tags -f

Conclusion

This tutorial provides a solution to optimize your repo’s size on Github. Remember you should use Git FS to save a large size of contents.

If not, don’t forget to regularly check whether there are some large files in Git History or not.

Making your game codes as small as possible not only can boost work efficiency, but also provide a high-quality game on time and on budget. Because it’s easy to maintain codes and add new features with fewer efforts.