transfer.unpackLimit defaults to 100, meaning that if less than 100
objects exist in the repository, git will automatically unpack the
objects to be loose as part of the clone operation. So, if there are no
packs and less than 100 objects, consider the repo to be freshly packed
for purposes of our fresh clone sanity checks.
Signed-off-by: Elijah Newren <newren@gmail.com>
analyze_commit() calls add_commit_and_parents() which does a sanity
check that we have seen all parents previously. --refs breaks that
assumption, so we need to workaround that check when ref limiting is in
effect.
Signed-off-by: Elijah Newren <newren@gmail.com>
Users may have long lists of --path, --path-rename, --path-regex, etc.
flags (or even a --paths-from-file option with a lot of entries in the
file). In such cases, we may have to compare any given path against a
lot of different values. In order to avoid having to repeat that long
list of comparisons every time a given path is updated, we long ago
added a cache of the renames so that we can compute the new name for a
path once and then just reuse it each time a new commit updates the old
filepath.
Sadly, I flubbed the implementation and instead of setting
cache[oldname] = newname
I somehow did the boneheaded
cache[newname] = newname
For most repositories and rewrites, this would just have the effect of
making the cache useless, but it could wreak various kinds of havoc if
a newname matched the oldname of some other file.
Make sure we record the mapping from OLDNAME to newname to fix these
issues.
Signed-off-by: Elijah Newren <newren@gmail.com>
The --analyze mode was extremely slow for the freebsd/freebsd repo on
github; digging in, the is_ancestor() function was being called a huge
number of times -- about 22 times per commit on average (and about 17
million times overall). The analyze mode uses is_ancestor() to
determine whether a rename equivalency class should be broken (i.e.
renaming A->B mean all versions of A and B are just different versions
of the same file, but if someone adds a new A in some commit which
contains the A->B rename in its history then this equivalence class no
longer holds). Each is_ancestor() call potentially has to walk a tree
of dependencies all the way back to a sufficient depth where it can
realize that the commit cannot be an ancestor; this can be a very long
walk.
We can speed this up by keeping track of some previous is_ancestor()
results. If commit F is not an ancestor of commit G, then F cannot be
an ancestor of children of G (unless that child has multiple parents;
but even in that case F can only be an ancestor through one of the
parents other than G). Similarly, if F is an ancestor of commit G, then
F will always be an ancestor of any children of G. Cache results from
previous calls to is_ancestor() and use them to accelerate subsequent
calls.
Signed-off-by: Elijah Newren <newren@gmail.com>
Translating "Processed %d blob sizes" or "Processed %d commits" hundreds
of thousands or millions of times is a waste and turns out to be pretty
expensive. Translate it once, cache the string, and then re-use it.
Note that a similar issue was noted in commit 3999349be4 (filter-repo:
fix perf regression; avoid excessive translation, 2019-05-21), but I did
not think to check --analyze mode for similar issues back then. Fix it
now.
Signed-off-by: Elijah Newren <newren@gmail.com>
There was code to allow the argument of --to-subdirectory-filter and
--subdirectory-filter to have a trailing slash, but it was broken due to
a bug in python3's bytestring design: b'somestring/'[-1] != b'/',
despite that being the obvious expectation. One either has to compare
b'somestring/'[-1:] to b'/' or else compare b'somestring/'[-1] to
b'/'[0]. So lame. Note that this is essentially a follow-up to commit
385b0586ca ("filter-repo (python3): bytestr splicing and iterating is
different", 2019-04-27).
Signed-off-by: Elijah Newren <newren@gmail.com>
Blob callbacks, either implicit (via e.g. --replace-text) or explicit,
can modify blobs in ways that make them match other blobs, which in turn
can result in some commits becoming empty. We need to detect such cases
and ensure we prune these empty commits when --prune-empty=auto.
Reported-by: John Gietzen <john@gietzen.us>
Signed-off-by: Elijah Newren <newren@gmail.com>
Some projects have a strict --no-ff merging policy. With the default
behavior of --prune-degenerate, we can prune merge commits in a way that
transforms the history into a fast-forward merge. Consider this
example:
* There are two independent commits or branches, named B & C, which
are both built on top of A so that history look like this diagram:
A
\ \
\ B
\
-C
* Someone runs the following sequence of commands:
* git checkout A
* git merge --no-ff B
* git merge --no-ff C
* This will result in a history that looks like:
A---AB---AC
\ \ / /
\ B /
\ /
-C-
* Later, someone comes along and runs filter-repo, specifying to
remove the only path(s) that were modified by B. That would
naturally remove commit B and the no-longer-necessary merge
commit AB. For someone using a strict no-ff policy, the desired
history is
A---AC
\ /
C
However, the default handling for --prune-degenerate would
notice that AC merely merges C into its own ancestor A, whereas
the original AC merged C into something separate (namely, AB).
So, it would say that AC has become degenerate and prune it,
leaving the simple history of
A
\
C
For projects not using a strict no-ff policy, this simpler history
is probably better, but for folks that want a strict no-ff policy,
it is unfortunate.
Provide a --no-ff option to tweak the --prune-degenerate behavior so
that it ignores the first parent being an ancestor of another parent
(leaving the first parent unpruned even if it is or becomes degenerate
in this fashion).
Signed-off-by: Elijah Newren <newren@gmail.com>
Prior to this commit, git-filter-repo could only be used with either the
--dry-run flag or the --debug flag, not both. When run in debug mode,
git-filter-repo expected to be able to read from the output stream,
which obviously isn't created when doing a dry run, so it stack traced
when it tried to use the non-existent output stream. This commit fixes
that bug with an equally simple sanity check for the existence of the
output stream when run in debug mode.
Signed-off-by: Karl Lenz <xorangekiller@gmail.com>
The mailmap format parsed by the "git shortlog" command allows for
matching mailmap entries with no email address. This is admittedly an
edge case, because most Git commits will have an email address
associated with them as well as a name, but technically the address
isn't required, and "git shortlog" accommodates that in its mailmap
format. This commit teaches git-filter-repo to do the same thing.
Signed-off-by: Karl Lenz <xorangekiller@gmail.com>
It's hard to be exhaustive, but if users try something like:
--path-rename foo/bar/baz:.
or
--path ../other-dir
then bad things happen. In the first case, filter-repo will try to
ask fast-import to create a directory named '.' and move everything
from foo/bar/baz/ into it but of course '.' is a reserved directory
name so we can't create it. In the second case, they are probably
running from a subdirectory, but filter-repo doesn't work from a
subdirectory. I hard-coded the assumption that everything was in the
toplevel directory and all paths were relative from there pretty
early on. So, if the user tries to use any of these components
anywhere, just throw an early error.
Signed-off-by: Elijah Newren <newren@gmail.com>
In commit f2729153 (filter-repo: workaround Windows' insistence that cwd
not be a bytestring, 2019-10-19), filter-repo was made to use a special
SubprocessWrapper class instead of the normal subprocess calls, due to
what appears to be in bugs in the python implementation on Windows not
working with arguments being bytestrings. Add the same workarounds to
bfg-ish and filter-lamely.
Signed-off-by: Elijah Newren <newren@gmail.com>
git fast-import, in an attempt to be friendly, allows the same file to
be specified multiple times within a commit and just takes the last
version of the file listed. It determines whether files are the same
via fspathncmp, which is defined differently depending on the setting of
core.ignorecase. Unfortunately, this means that if someone is trying to
do filtering of history and using a broken (case-insensitive) filesystem
and the history they are filtering has some paths that differed in name
only, then fast-import will delete whichever of the "colliding" files is
listed first.
Avoid these problems by just turning off core.ignorecase while
fast-import is running. This will prevent silently modifying the repo
in an unexpected way. Users on such filesystems may have difficulty
checking out commits with files which differ in case only, but that is
a separate problem for them to deal with after rewriting history.
Signed-off-by: Elijah Newren <newren@gmail.com>
In commit 509a624 (filter-repo: fix issue with pruning of empty commits,
2019-10-03), it was noted that when the first parent is pruned away,
then we need to generate a corrected list of file changes relative to
the new first parent. Unfortunately, we did not apply our set of file
filters to that new list of file changes, causing us to possibly
introduce many unwanted files from the second parent into the history.
The testcase added at the time was rather lax and totally missed this
problem (which possibly exacerbated the original bug being fixed rather
than helping). Tighten the testcase, and fix the error by filtering the
generated list of file changes.
Signed-off-by: Elijah Newren <newren@gmail.com>
RepoFilter._tweak_commit() was a bit unwieldy, and we have a reason for
wanting to re-use the file filtering logic in it, so break that out into
a separate function.
Signed-off-by: Elijah Newren <newren@gmail.com>
The analyze mode will handle type changes (e.g. normal file to symlink)
in combination with adds and modifies, but the similar logic below
didn't allow for type changes in combination with renames. Fix the
oversight.
Signed-off-by: Elijah Newren <newren@gmail.com>
It appears that in addition to Windows requiring cwd be a string (and
not a bytestring), it also requires the command line arguments to be
unicode strings. This appears to be a python-on-Windows issue at the
surface (attempts to quote things that assumes the arguments are all
strings), but whether it's solely a python-on-Windows issue or there is
also a deeper Windows issue, we can workaround this brain-damage by
extending the SubprocessWrapper slightly. As with the cwd changes, only
apply this on Windows and not elsewhere because there are perfectly
legitimate reasons to pass non-unicode parameters (e.g. filenames that
are not valid unicode).
Signed-off-by: Elijah Newren <newren@gmail.com>
Unfortunately, it appears that Windows does not allow the 'cwd' argument
of various subprocess calls to be a bytestring. That may be functional
on Windows since Windows-related filesystems are allowed to require that
all file and directory names be valid unicode, but not all platforms
enforce such restrictions. As such, I certainly cannot change
cwd=directory
to
cwd=decode(directory)
because that could break on other platforms (and perhaps even on Windows
if someone is trying to read a non-native filesystem). Instead, create
a SubprocessWrapper class that will always call decode on the cwd
argument before passing along to the real subprocess class. Use these
wrappers on Windows, and do not use them elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
During the python3 transition, StringIO was renamed to io -- but the
import wasn't moved to preserve appropriate sorting.
Signed-off-by: Elijah Newren <newren@gmail.com>
Note that this isn't a version *number* or even the more generalized
version string that folks are used to seeing, but a version hash (or
leading portion thereof).
A few import points:
* These version hashes are not strictly monotonically increasing
values. Like I said, these aren't version numbers. If that
bothers you, read on...
* This scheme has incredibly nice semantics satisfying a pair of
properties that most version schemes would assume are mutually
incompatible:
This scheme works even if the user doesn't have a clone of
filter-repo and doesn't require any build step to inject the
version into the program; it works even if people just download
git-filter-repo.py off GitHub without any of the other sources.
And:
This scheme means that a user is running precisely version X of
the code, with the version not easily faked or misrepresented
when third parties edit the code.
Given the wonderful semantics provided by satisfying this pair of
properties that all other versioning schemes seem to miss out on, I
think I should name this scheme. How about "Semantic Versioning"?
(Hehe...)
* The version hash is super easy to use; I just go to my own clone of
filter-repo and run either:
git show $VERSION_HASH
or
git describe $VERSION_HASH
* A human consumable version might suggest to folks that this software
is something they might frequently use and upgrade. This program
should only be used in exceptional cases (because rewriting history
is not for the faint of heart).
* A human consumable version (i.e. a version number or even the
more relaxed version strings in more common use) might suggest to
folks that they can rely on strict backward compatibility. It's
nice to subtly undercut any such assumption.
* Despite all that, I will make releases (downloadable tarballs with
real version numbers in the tarball name; I'm just going to re-use
whatever version git is released with at the time). But those
version numbers won't be used by the --version option; instead the
version hash will.
Signed-off-by: Elijah Newren <newren@gmail.com>
Partial history rewrites were possible before with the (previously
hidden) --refs flag, but the defaults were wrong. That could be worked
around with the --source or --target flags, but that disabled --no-data
for fast-export and thus slowed things down, and also would require
overridding --replace-refs. And the defaults for --source and --target
may diverge further from what is wanted/needed for partial history
rewrites in the future.
So, add --partial as a first-class supported option with scary
documentation about how it permits mixing new and old history. Make
--refs imply that flag. Make the behavioral similarities (in regards to
which steps are skipped) between --source, --target, and --partial more
clear. Add relevant documentation to round it out.
Signed-off-by: Elijah Newren <newren@gmail.com>
In order to build the correct tree for a commit, git-fast-import always
takes a list of file changes for a merge commit relative to the first
parent.
When the entire first-parent history of a merge commit is pruned away
and the merge had paths with no difference relative to the first parent
but which differed relative to later parents, then we really need to
generate a new list of file changes in order to have one of those other
parents become the new first parent. An example might help clarify...
Let's say that there is a merge commit, and:
* it resolved differences in pathA between its two parents by taking
the version of pathA from the first parent.
* pathB was added in the history of the second parent (it is not
present in the first parent) and is NOT included in the merge commit
(either being deleted, or via rename treated as deleted and added as
something else)
For this merge commit, neither pathA nor pathB differ from the first
parent, and thus wouldn't appear in the list of file changes shown by
fast-export. However, when our filtering rules determine that the first
parent (and all its parents) should be pruned away, then the second
parent has to become the new first parent of the merge commit. But to
end up with the right files in the merge commit despite using a
different parent, we need a list of file changes that specifies the
changes for both pathA and pathB.
Signed-off-by: Elijah Newren <newren@gmail.com>
git.git wants to move more towards core-only rather than batteries
included, and as such, filter-repo will not be part of the git
distribution. Therefore, due to keeping the projects apart, there will
need to be separate translation files (assuming filter-repo ever gains
any translations) and as such we will need a different textdomain
definition.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow folks to periodically update the export of a live repo without
re-exporting from the beginning. This is a performance improvement, but
can also be important for collaboration. For example, for sensitivity
reasons, folks might want to export a subset of a repo and update the
export periodically. While this could be done by just re-exporting the
repository anew each time, there is a risk that the paths used to
specify the wanted subset might need to change in the future; making the
user verify that their paths (including globs or regexes) don't also
pick up anything from history that was previously excluded so that they
don't get a divergent history is not very user friendly. Allowing them
to just export stuff that is new since the last export works much better
for them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Being able to find the new commit hash for either an abbreviated commit
hash or a full commit hash is much more useful than only working for a
full commit hash.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit 346f2ba891 (filter-repo: make reencoding of commit messages
togglable, 2019-05-11) made reencoding of commit messages togglable but
forgot to add parsing and outputting of the encoding header itself. Add
such ability now.
Signed-off-by: Elijah Newren <newren@gmail.com>
Now that we are tracking exported and imported refs, we no longer need
to rely on _orig_refs and _seen_refs for deletion of "unused" refs at
the end of the run. Verify that we correctly tracked exported and
imported refs by using them instead for the post-run ref deletion. This
removes the last use of _seen_refs, which will be removed in a
subsequent commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
We previously nuked all refs not seen in the import using _seen_refs, by
comparing to a full list of original refs. That works okay when doing a
full repository rewrite, but fails for partial history rewrites.
Further, external rewriting tools that wants to implement a tweak of
this behavior would have had to access the internal _seen_refs field,
but might not be able to rely on _orig_refs if they were doing a partial
history rewrite. Fix both by tracking both which refs were exported
from the source repository, and which were ultimately imported into the
target repository (they may differ due to pruned commits, renamed
branches or tags, etc.). Make both available via a new public API,
get_exported_and_imported_refs().
Signed-off-by: Elijah Newren <newren@gmail.com>
External rewrite tools using filter-repo as a library may want to add
additional objects into the stream. Some examples in t/t9391 did this
using an internal _output field and using syntax that did not seem so
clear. Provide an insert() method for doing this, and convert existing
cases over to it.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we prune a commit for being empty, there is no update to the branch
associated with the commit in the fast-import stream. If the parent
commit had been associated with a different branch, then the branch
associated with the pruned commit would not be updated without
additional measures. In the past, we resolved this by recording that
the branch needed an update in _seen_refs. While this works, it is a
bit more complicated than just issuing an immediate Reset. Also, note
that we need to avoid calling callbacks on that Reset because those
could rename branches (again, if the commit-callback already renamed
once) causing us to not update the intended branch.
There was actually one testcase where the old method didn't work: when a
branch was pruned away to nothing. A testcase accidentally encoded the
wrong behavior, hiding this problem. Fix the testcase to check for
correct behavior.
Signed-off-by: Elijah Newren <newren@gmail.com>
We previously did this incorrectly, but due to our assumptions of
full-history rewriting and deleting of unseen refs, we got away with it.
Fix this for partial history rewrites.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit 1f0e57bada ("filter-repo: avoid pruning annotated tags that we
have seen", 2019-03-07) left behind the setting of a variable,
full_ref, that is no longer used. Remove it.
Signed-off-by: Elijah Newren <newren@gmail.com>
Add a flag allowing for specifying a file filled with blob-ids which
will be stripped from the repository.
Signed-off-by: Elijah Newren <newren@gmail.com>
Fix a few issues and add a token testcase for partial repo filtering.
Add a note about how I think this is not a particularly interesting or
core usecase for filter-repo, even if I have put some good effort into
the fast-export side to ensure it worked. If there is a core usecase
that can be addressed without causing usability problems (particularly
the "don't mix old and new history" edict for normal rewrites), then
I'll be happy to add more testcases, document it better, etc.
Signed-off-by: Elijah Newren <newren@gmail.com>
Make several fixes around --source and --target:
* Explain steps we skip when source or target locations are specified
* Only write reports to the target directory, never the source
* Query target git repo for final ref values, not the source
* Make sure --debug messages avoid throwing TypeErrors due to mixing
strings and bytes
* Make sure to include entries in ref-map that weren't in the original
target repo
* Don't:
* worry about mixing old and new history (i.e. nuking refs
that weren't updated, expiring reflogs, gc'ing)
* attempt to map refs/remotes/origin/* -> refs/heads/*
* disconnect origin remote
* Continue (but only in target repo):
* fresh-clone sanity checks
* writing replace refs
* doing a 'git reset --hard'
Signed-off-by: Elijah Newren <newren@gmail.com>
Add a flag for filtering out blob based on their size, and allow the
size to be specified using 'K', 'M', or 'G' suffixes.
Signed-off-by: Elijah Newren <newren@gmail.com>
Create a new function, GitUtils.get_blob_sizes() to hold some logic
that used to be at the beginning of RepoAnalyze.gather_data(). This
will allow reuse of this functionality within RepoFilter.
Signed-off-by: Elijah Newren <newren@gmail.com>
Although fast-import can take file changes in any order, trying to debug
by comparing the original fast-export stream to the filtered version is
difficult if the files are randomly reordered. Sometimes we aren't
comparing the filtered version to the original but just looking at the
stream passed to fast-import, in which case having the files in sorted
order may help.
Our accumulation of file_changes into a dict() in order to check for
collisions when renaming had the unfortunate side effect of sorting
files by internals of dictionary ordering. Although the files started
in sorted order, we don't in general want to use the original order
because renames can cause filenames to become out-of-order. Just apply
a simple sort at the end.
Signed-off-by: Elijah Newren <newren@gmail.com>
I suspect at some point someone will try to pass -M or -C to
fast-export; may as well leave a note in the code about another place
that's incompatible while I'm thinking about it.
Signed-off-by: Elijah Newren <newren@gmail.com>
Imperative form sounds better than --empty-pruning and
--degenerate-pruning, and it probably works better with command line
completion.
Signed-off-by: Elijah Newren <newren@gmail.com>
The reset directive can specify a commit hash for the 'from' directive,
which can be used to reset to a specify commit, or, if the hash is all
zeros, then it can be used to delete the ref. Support such operations.
Signed-off-by: Elijah Newren <newren@gmail.com>
For other programs importing git-filter-repo as a library and passing a
blob, commit, tag, or reset callback to RepoFilter, pass a second
parameter to these functions with extra metadata they might find useful.
For simplicity of implementation, this technically changes the calling
signature of the --*-callback functions passed on the command line, but
we hide that behind a _do_not_use_this_variable parameter for now, leave
it undocumented, and encourage folks who want to use it to write an
actual python program that imports git-filter-repo. In the future, we
may modify the --*-callback functions to not pass this extra parameter,
or if it is deemed sufficiently useful, then we'll rename the second
parameter and document it.
As already noted in our API compatibilty caveat near the top of
git-filter-repo, I am not guaranteeing API backwards compatibility.
That especially applies to this metadata argument, other than the fact
that it'll be a dict mapping strings to some kind of value. I might add
more keys, rename them, change the corresponding value, or even remove
keys that used to be part of metadata.
Signed-off-by: Elijah Newren <newren@gmail.com>
Location of filtering logic was previously split in a confusing fashion
between FastExportFilter and RepoFilter. Move all filtering logic from
FastExportFilter into RepoFilter, and rename the former to
FastExportParser to reflect this change.
One downside of this change is that FastExportParser's _parse_commit
holds two pieces of information (orig_parents and had_file_changes)
which are not part of the commit object but which are now needed by
RepoFilter. Adding those bits of info to the commit object does not
make sense, so for now we pass an auxiliary dict with the
commit_callback that has these two fields. This information is not
passed along to external commit_callbacks passed to RepoFilter, though,
which seems suboptimal. To be fair, though, commit_callbacks to
RepoFilter never had access to this information so this is not a new
shortcoming, it just seems more apparent now.
Signed-off-by: Elijah Newren <newren@gmail.com>
I introduced this over a decade ago thinking it would come in handy in
some special case, and the only place I used it was in a testcase that
existed almost solely to increase code coverage. Modify the testcase to
instead demonstrate how it is trivial to get the effects of the
everything_callback without it being present.
Signed-off-by: Elijah Newren <newren@gmail.com>
The specially constructed callbacks in RepoFilter.run() were
superfluous; we already had special callback functions. Instead of
creating new local functions that call the real callbacks and then do
one extra step, just put the extra wanted code into the real callbacks.
Signed-off-by: Elijah Newren <newren@gmail.com>
Using fast-import's feature done capability, any output sent to it after
the 'done' directive will be ignored. We do not intend to send any such
information, but there have been a couple cases where an accident while
refactoring the code resulted in some information being sent after the
done directive. To avoid having to debug that again, just close the
output stream after sending the 'done' directive to ensure that we get
an immediate and clear error if we ever run into such a situation again.
Signed-off-by: Elijah Newren <newren@gmail.com>
The upcoming git-2.22 release will not have the --reencode option to
fast-export; however, since we default to --reencode=yes and that was
the default behavior in all existing versions of git (only to change in
git-2.23), we can just silently leave the option off if we detect we are
running with this version. However, the diff-tree --combined-all-paths
option from git-2.22 is still mandatory; we cannot run with git versions
older than that (well, with -rc or built-from-source versions, but that
won't matter to most users).
Signed-off-by: Elijah Newren <newren@gmail.com>
Translating "Parsed %d commits" a hundred thousand times (once per
commit), turned out to be somewhat expensive -- especially since we
were only going to print it out once every few thousand commits.
Translate it once and cache the result, shaving off about 20% of
execution time for a simple rewrite of a test repository (rails).
Signed-off-by: Elijah Newren <newren@gmail.com>
This allows the user to put a whole bunch of paths they want to keep (or
want to remove) in a file and then just provide the path to it. They
can also use globs or regexes (similar to --replace-text) and can also
do renames. In fact, this allows regex renames, despite the fact that I
never added a --path-rename-regex option.
Signed-off-by: Elijah Newren <newren@gmail.com>
Using an exact path (file or directory) for --path-rename instead of a
prefix removes an ugly caveat from the documentation, makes it operate
similarly to --path, and will make it easier to reuse common code when I
add the --paths-from-file option. Switch over, and replace the
startswith() check by a call to filename_matches().
Signed-off-by: Elijah Newren <newren@gmail.com>
This new flag allows people to filter files solely based on their
basename rather than on their full path within the repo, making it
easier to e.g. remove all .DS_Store files or keep all README.md
files.
Signed-off-by: Elijah Newren <newren@gmail.com>
This class only represents one FileChange, so fix the misnomer and make
it clearer to others the purpose of this object.
Signed-off-by: Elijah Newren <newren@gmail.com>
For most repos, --fake-missing-tagger will be a no-op. But if a repo
out there has a missing tagger, then fast-import will choke on it unless
one is inserted; ask fast-export to do that.
Signed-off-by: Elijah Newren <newren@gmail.com>
This adds the ability to automatically add new replacement refs for each
rewritten commit (as well as delete or update replacement refs that
existed before the run). This will allow users to use either new or old
commit hashes to reference commits locally, though old commit hashes
will need to be unabbreviated. The only requirement for this to work,
is that the person who does the rewrite also needs to push the replace
refs up where other users can grab them, and users who want to use them
need to modify their fetch refspecs to grab the replace refs.
However, other tools external to git may not understand replace refs...
Tools like Gerrit and GitHub apparently do not yet natively understand
replace refs. Trying to view "commits" by the replacement ref will
yield various forms of "Not Found" in each tool. One has to instead try
to view it as a branch with an odd name (including "refs/replace/"), and
often branches are accessed via a different URL style than commits so it
becomes very non-obvious to users how to access the info associated with
an old commit hash.
* In Gerrit, instead of being able to search on the sha1sum or use a
pre-defined URL to search and auto-redirect to the appropriate code
review with
https://gerrit.SITE.COM/#/q/${OLD_SHA1SUM},n,z
one instead has to have a special plugin and go to a URL like
https://gerrit.SITE.COM/plugins/gitiles/ORG/REPO/+/refs/replace/${OLD_SHA1SUM}
but then the user isn't shown the actual code review and will need
to guess which link to click on to get to it (and it'll only be
there if the user included a Change-Id in the commit message).
* In GitHub, instead of being able to go to a URL like
https://github.SITE.COM/ORG/REPO/commit/${OLD_SHA1SUM}
one instead has to navigate based on branch using
https://github.SITE.COM/ORG/REPO/tree/refs/replace/${OLD_SHA1SUM}
but that will show a listing of commits instead of information about
a specific commit; the user has to manually click on the first commit
to get to the desired location.
For now, providing replace refs at least allows users to access
information locally using old IDs; perhaps in time as other external
tools will gain a better understanding of how to use replace refs, the
barrier to history rewrites will decrease enough that big projects that
really need it (e.g. those that have committed many sins by commiting
stupidly large useless binary blobs) can at least seriously contemplate
the undertaking. History rewrites will always have some drawbacks and
pain associated with them, as they should, but when warranted it's nice
to have transition plans that are more smooth than a massive flag day.
Signed-off-by: Elijah Newren <newren@gmail.com>
We have a good default for pruning of empty commits and degenerate merge
commits: only pruning such commits that didn't start out that way (i.e.
that couldn't intentionally have been empty or degenerate). However,
users may have reasons to want to aggressively prune such commits (maybe
they used BFG repo filter or filter-branch previously and have lots of
cruft commits that they want remoed), and we may as well allow them to
specify that they don't want pruning too, just to be flexible.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a merge in the original repo, and its ancestors on at
least one side have all been filtered away, and the commit has no
filechanges relative to its remaining parent (if any), then this commit
should be pruned. We had a small logic error preventing such pruning;
fix it by checking len(parents) instead of len(orig_parents).
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import syntax declares how to specify the parents of a commit with
'from' and possibly 'merge' directives, but it oddly also allows parents
to be implicitly specified via branch name. The documentation is easy
to misread:
"Omitting the from command in the first commit of a new branch will
cause fast-import to create that commit with no ancestor."
Note that the "in the first commit of a new branch" is key here. It is
reinforced later in the document with:
"Omitting the from command on existing branches is usually desired, as
the current commit on that branch is automatically assumed to be the
first ancestor of the new commit."
Desirability of operating this way aside, this raises an interesting
question: what if you only have one branch in some repository, but that
branch has more than one root commit? How does one use the fast-import
format to import such a repository? The fast-import documentation
doesn't state as far as I can tell, but using a 'reset' directive
without providing a 'from' reference for it is the way to go.
Modify filter-repo to understand implicit 'from' commits, and to
appropriately issue 'reset' directives when we need additional root
commits.
Signed-off-by: Elijah Newren <newren@gmail.com>
This is by far the largest python3 change; it consists basically of
* using b'<str>' instead of '<str>' in lots of places
* adding a .encode() if we really do work with a string but need to
get it converted to a bytestring
* replace uses of .format() with interpolation via the '%' operator,
since bytestrings don't have a .format() method.
Signed-off-by: Elijah Newren <newren@gmail.com>
Unlike how str works, if we grab an array index of a bytestr we get an
integer (corresponding to the ASCII value) instead of a bytestr of
length 1. Adjust code accordingly.
Signed-off-by: Elijah Newren <newren@gmail.com>
python3 forces a couple issues for us with the conversion of globs to
regexes:
* fnmatch.translate() will ONLY operate on unicode strings, not
bytestrings. Super lame.
* newer versions of python3 modified the regex style used by
fnmatch.translate() causing us to need extra logic to 'fixup'
the regex into the form we want.
Split the code for translating the glob to a regex out into a separate
function which now houses more complicated logic to handle these extra
conditions.
Signed-off-by: Elijah Newren <newren@gmail.com>
We need a function to transform byte strings into unicode strings for
printing error messages and occasional other uses.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit ca32c5d9afe2 ("filter-repo: workaround python<2.7.9 exec bug",
2019-04-30) put in a workaround for python versions prior to 2.7.9, but
which was incompatible with python3. Revert it as one step towards
migrating to python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Python issue 21591 will cause SyntaxError messages to by thrown if using
python versions prior to 2.7.9. Use the workaround identified in the
bug report: use the exec statement instead of the exec function, even if
this will need to be reverted for python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Extra LFs are permitted in git-fast-import syntax, and they serve to
make it easier to read the stream (from --dry-run or --debug), if they
are so inclined.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we invoked the 'ls' command of fast-import, we just passed the
filename as-is. That will work for most filenames, but some have to
be quoted. Make sure we do so.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
Karl Lenz
Matthisk Heimensen