Although fast-import can take file changes in any order, trying to debug
by comparing the original fast-export stream to the filtered version is
difficult if the files are randomly reordered. Sometimes we aren't
comparing the filtered version to the original but just looking at the
stream passed to fast-import, in which case having the files in sorted
order may help.
Our accumulation of file_changes into a dict() in order to check for
collisions when renaming had the unfortunate side effect of sorting
files by internals of dictionary ordering. Although the files started
in sorted order, we don't in general want to use the original order
because renames can cause filenames to become out-of-order. Just apply
a simple sort at the end.
Signed-off-by: Elijah Newren <newren@gmail.com>
I suspect at some point someone will try to pass -M or -C to
fast-export; may as well leave a note in the code about another place
that's incompatible while I'm thinking about it.
Signed-off-by: Elijah Newren <newren@gmail.com>
Imperative form sounds better than --empty-pruning and
--degenerate-pruning, and it probably works better with command line
completion.
Signed-off-by: Elijah Newren <newren@gmail.com>
The reset directive can specify a commit hash for the 'from' directive,
which can be used to reset to a specify commit, or, if the hash is all
zeros, then it can be used to delete the ref. Support such operations.
Signed-off-by: Elijah Newren <newren@gmail.com>
For other programs importing git-filter-repo as a library and passing a
blob, commit, tag, or reset callback to RepoFilter, pass a second
parameter to these functions with extra metadata they might find useful.
For simplicity of implementation, this technically changes the calling
signature of the --*-callback functions passed on the command line, but
we hide that behind a _do_not_use_this_variable parameter for now, leave
it undocumented, and encourage folks who want to use it to write an
actual python program that imports git-filter-repo. In the future, we
may modify the --*-callback functions to not pass this extra parameter,
or if it is deemed sufficiently useful, then we'll rename the second
parameter and document it.
As already noted in our API compatibilty caveat near the top of
git-filter-repo, I am not guaranteeing API backwards compatibility.
That especially applies to this metadata argument, other than the fact
that it'll be a dict mapping strings to some kind of value. I might add
more keys, rename them, change the corresponding value, or even remove
keys that used to be part of metadata.
Signed-off-by: Elijah Newren <newren@gmail.com>
Location of filtering logic was previously split in a confusing fashion
between FastExportFilter and RepoFilter. Move all filtering logic from
FastExportFilter into RepoFilter, and rename the former to
FastExportParser to reflect this change.
One downside of this change is that FastExportParser's _parse_commit
holds two pieces of information (orig_parents and had_file_changes)
which are not part of the commit object but which are now needed by
RepoFilter. Adding those bits of info to the commit object does not
make sense, so for now we pass an auxiliary dict with the
commit_callback that has these two fields. This information is not
passed along to external commit_callbacks passed to RepoFilter, though,
which seems suboptimal. To be fair, though, commit_callbacks to
RepoFilter never had access to this information so this is not a new
shortcoming, it just seems more apparent now.
Signed-off-by: Elijah Newren <newren@gmail.com>
I introduced this over a decade ago thinking it would come in handy in
some special case, and the only place I used it was in a testcase that
existed almost solely to increase code coverage. Modify the testcase to
instead demonstrate how it is trivial to get the effects of the
everything_callback without it being present.
Signed-off-by: Elijah Newren <newren@gmail.com>
The specially constructed callbacks in RepoFilter.run() were
superfluous; we already had special callback functions. Instead of
creating new local functions that call the real callbacks and then do
one extra step, just put the extra wanted code into the real callbacks.
Signed-off-by: Elijah Newren <newren@gmail.com>
Using fast-import's feature done capability, any output sent to it after
the 'done' directive will be ignored. We do not intend to send any such
information, but there have been a couple cases where an accident while
refactoring the code resulted in some information being sent after the
done directive. To avoid having to debug that again, just close the
output stream after sending the 'done' directive to ensure that we get
an immediate and clear error if we ever run into such a situation again.
Signed-off-by: Elijah Newren <newren@gmail.com>
The upcoming git-2.22 release will not have the --reencode option to
fast-export; however, since we default to --reencode=yes and that was
the default behavior in all existing versions of git (only to change in
git-2.23), we can just silently leave the option off if we detect we are
running with this version. However, the diff-tree --combined-all-paths
option from git-2.22 is still mandatory; we cannot run with git versions
older than that (well, with -rc or built-from-source versions, but that
won't matter to most users).
Signed-off-by: Elijah Newren <newren@gmail.com>
Translating "Parsed %d commits" a hundred thousand times (once per
commit), turned out to be somewhat expensive -- especially since we
were only going to print it out once every few thousand commits.
Translate it once and cache the result, shaving off about 20% of
execution time for a simple rewrite of a test repository (rails).
Signed-off-by: Elijah Newren <newren@gmail.com>
This allows the user to put a whole bunch of paths they want to keep (or
want to remove) in a file and then just provide the path to it. They
can also use globs or regexes (similar to --replace-text) and can also
do renames. In fact, this allows regex renames, despite the fact that I
never added a --path-rename-regex option.
Signed-off-by: Elijah Newren <newren@gmail.com>
Using an exact path (file or directory) for --path-rename instead of a
prefix removes an ugly caveat from the documentation, makes it operate
similarly to --path, and will make it easier to reuse common code when I
add the --paths-from-file option. Switch over, and replace the
startswith() check by a call to filename_matches().
Signed-off-by: Elijah Newren <newren@gmail.com>
This new flag allows people to filter files solely based on their
basename rather than on their full path within the repo, making it
easier to e.g. remove all .DS_Store files or keep all README.md
files.
Signed-off-by: Elijah Newren <newren@gmail.com>
This class only represents one FileChange, so fix the misnomer and make
it clearer to others the purpose of this object.
Signed-off-by: Elijah Newren <newren@gmail.com>
For most repos, --fake-missing-tagger will be a no-op. But if a repo
out there has a missing tagger, then fast-import will choke on it unless
one is inserted; ask fast-export to do that.
Signed-off-by: Elijah Newren <newren@gmail.com>
This adds the ability to automatically add new replacement refs for each
rewritten commit (as well as delete or update replacement refs that
existed before the run). This will allow users to use either new or old
commit hashes to reference commits locally, though old commit hashes
will need to be unabbreviated. The only requirement for this to work,
is that the person who does the rewrite also needs to push the replace
refs up where other users can grab them, and users who want to use them
need to modify their fetch refspecs to grab the replace refs.
However, other tools external to git may not understand replace refs...
Tools like Gerrit and GitHub apparently do not yet natively understand
replace refs. Trying to view "commits" by the replacement ref will
yield various forms of "Not Found" in each tool. One has to instead try
to view it as a branch with an odd name (including "refs/replace/"), and
often branches are accessed via a different URL style than commits so it
becomes very non-obvious to users how to access the info associated with
an old commit hash.
* In Gerrit, instead of being able to search on the sha1sum or use a
pre-defined URL to search and auto-redirect to the appropriate code
review with
https://gerrit.SITE.COM/#/q/${OLD_SHA1SUM},n,z
one instead has to have a special plugin and go to a URL like
https://gerrit.SITE.COM/plugins/gitiles/ORG/REPO/+/refs/replace/${OLD_SHA1SUM}
but then the user isn't shown the actual code review and will need
to guess which link to click on to get to it (and it'll only be
there if the user included a Change-Id in the commit message).
* In GitHub, instead of being able to go to a URL like
https://github.SITE.COM/ORG/REPO/commit/${OLD_SHA1SUM}
one instead has to navigate based on branch using
https://github.SITE.COM/ORG/REPO/tree/refs/replace/${OLD_SHA1SUM}
but that will show a listing of commits instead of information about
a specific commit; the user has to manually click on the first commit
to get to the desired location.
For now, providing replace refs at least allows users to access
information locally using old IDs; perhaps in time as other external
tools will gain a better understanding of how to use replace refs, the
barrier to history rewrites will decrease enough that big projects that
really need it (e.g. those that have committed many sins by commiting
stupidly large useless binary blobs) can at least seriously contemplate
the undertaking. History rewrites will always have some drawbacks and
pain associated with them, as they should, but when warranted it's nice
to have transition plans that are more smooth than a massive flag day.
Signed-off-by: Elijah Newren <newren@gmail.com>
We have a good default for pruning of empty commits and degenerate merge
commits: only pruning such commits that didn't start out that way (i.e.
that couldn't intentionally have been empty or degenerate). However,
users may have reasons to want to aggressively prune such commits (maybe
they used BFG repo filter or filter-branch previously and have lots of
cruft commits that they want remoed), and we may as well allow them to
specify that they don't want pruning too, just to be flexible.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a merge in the original repo, and its ancestors on at
least one side have all been filtered away, and the commit has no
filechanges relative to its remaining parent (if any), then this commit
should be pruned. We had a small logic error preventing such pruning;
fix it by checking len(parents) instead of len(orig_parents).
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import syntax declares how to specify the parents of a commit with
'from' and possibly 'merge' directives, but it oddly also allows parents
to be implicitly specified via branch name. The documentation is easy
to misread:
"Omitting the from command in the first commit of a new branch will
cause fast-import to create that commit with no ancestor."
Note that the "in the first commit of a new branch" is key here. It is
reinforced later in the document with:
"Omitting the from command on existing branches is usually desired, as
the current commit on that branch is automatically assumed to be the
first ancestor of the new commit."
Desirability of operating this way aside, this raises an interesting
question: what if you only have one branch in some repository, but that
branch has more than one root commit? How does one use the fast-import
format to import such a repository? The fast-import documentation
doesn't state as far as I can tell, but using a 'reset' directive
without providing a 'from' reference for it is the way to go.
Modify filter-repo to understand implicit 'from' commits, and to
appropriately issue 'reset' directives when we need additional root
commits.
Signed-off-by: Elijah Newren <newren@gmail.com>
This is by far the largest python3 change; it consists basically of
* using b'<str>' instead of '<str>' in lots of places
* adding a .encode() if we really do work with a string but need to
get it converted to a bytestring
* replace uses of .format() with interpolation via the '%' operator,
since bytestrings don't have a .format() method.
Signed-off-by: Elijah Newren <newren@gmail.com>
Unlike how str works, if we grab an array index of a bytestr we get an
integer (corresponding to the ASCII value) instead of a bytestr of
length 1. Adjust code accordingly.
Signed-off-by: Elijah Newren <newren@gmail.com>
python3 forces a couple issues for us with the conversion of globs to
regexes:
* fnmatch.translate() will ONLY operate on unicode strings, not
bytestrings. Super lame.
* newer versions of python3 modified the regex style used by
fnmatch.translate() causing us to need extra logic to 'fixup'
the regex into the form we want.
Split the code for translating the glob to a regex out into a separate
function which now houses more complicated logic to handle these extra
conditions.
Signed-off-by: Elijah Newren <newren@gmail.com>
We need a function to transform byte strings into unicode strings for
printing error messages and occasional other uses.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit ca32c5d9afe2 ("filter-repo: workaround python<2.7.9 exec bug",
2019-04-30) put in a workaround for python versions prior to 2.7.9, but
which was incompatible with python3. Revert it as one step towards
migrating to python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Python issue 21591 will cause SyntaxError messages to by thrown if using
python versions prior to 2.7.9. Use the workaround identified in the
bug report: use the exec statement instead of the exec function, even if
this will need to be reverted for python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Extra LFs are permitted in git-fast-import syntax, and they serve to
make it easier to read the stream (from --dry-run or --debug), if they
are so inclined.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we invoked the 'ls' command of fast-import, we just passed the
filename as-is. That will work for most filenames, but some have to
be quoted. Make sure we do so.
Signed-off-by: Elijah Newren <newren@gmail.com>
While the underlying fast-export and fast-import streams explicitly
separate 'from' commit (first parent) and 'merge' commits (all other
parents), foisting that separation into the Commit object for
filter-repo forces additional places in the code to deal with that
distinction. It results in less clear code, and especially does not
make sense to push upon folks who may want to use filter-repo as a
library.
Signed-off-by: Elijah Newren <newren@gmail.com>
The sorting order of entries written to files in the analysis directory
didn't specify a secondary sort, thus making the order dependent on the
random-ish sorting order of dictionaries and making it inconsistent
between python versions. While the secondary order didn't matter much,
having a defined order makes it slightly easier to define a single
testcase that can work across versions.
Signed-off-by: Elijah Newren <newren@gmail.com>
Assuming filter-repo will be merged into git.git, use "git" for the
TEXTDOMAIN, and assume its build system will replace "@@LOCALEDIR@@"
appropriately.
Note that the xgettext command used to grab string translations is
nearly identical to the one for C files in git.git; just use
--language=python instead and add --join-existing to avoid overwriting
the po/git.pot file. In other words, use the command:
xgettext -o../git/po/git.pot --join-existing --force-po \
--add-comments=TRANSLATORS: \
--msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
--from-code=UTF-8 --language=python \
--keyword=_ --keyword=N_ --keyword="Q_:1,2" \
git-filter-repo
To create or update the translation, go to git.git/po and run either of:
msginit --locale=XX
msgmerge --add-location --backup=off -U XX.po git.pot
Once you've updated the translation, within git.git just build as
normal. That's all that's needed.
Signed-off-by: Elijah Newren <newren@gmail.com>
Over a decade ago, I added code to deal with splitting and splicing
repositories where you weren't always dealing with first parents and
linear histories, and in particular where the mainline tended to be the
second parent (because there was no integrator or special central
gatekeeper like gerrit or github; instead, everyone pushed directly to
the main repository after locally testing, and integration happened via
everyone running 'git pull'). When attempting to splice repositories
the fact that fast-export always gave changes relative to the first
parent caused some grief with my splitting and splicing efforts.
It has been over a decade, I don't know of a good testcase of this
functionality separate from the live repositories I lost access to over
six years ago, git-subtree was released in the meantime which I'm
certain handled the task better, git-fast-export since gained a
--full-tree option which might have provided a better way to attack the
problem (though with splicing repos you often want work with additive
changes rather than recreating from scratch), and I just don't
quite understand the code anymore anyway. I think it had some
fundamental limitations that I knew my usecase avoided, but I don't
remember the details (and I'm not certain if this is true).
Even though code coverage hits all but one of the lines, I'd rather
rewrite any needed functionality if the usecase arises, and in view of
what facilities exist today rather than what I was working with a decade
ago. So, just nuke this code.
Signed-off-by: Elijah Newren <newren@gmail.com>
The original idea was to add --path-rename-(glob|regex) options, but
I like the general flexibility of --filename-callback better for
special cases and keeping the number of command line options at least
slightly in check.
Signed-off-by: Elijah Newren <newren@gmail.com>
There are several lines equivalent to BUG() calls in git that are
supposed to be unreachable, and which exist just to make debugging the
fundamental system problem or refactoring of the code slightly easier by
trying to give a more immediate notification of a problem. If these
error cases are ever hit and happen to be wrong, then the individual
will at worst get a stacktrace and the program will abort...but that
might arguably be even more helpful. Since there is no harm in avoiding
the work of finding ways to break the system to force these lines to be
covered, simply exclude them from line coverage counting.
Signed-off-by: Elijah Newren <newren@gmail.com>
The AncestryGraph setup assumed we had previously seen all commits which
would be used as parents; that interacted badly with doing an
incremental import. Add a function which can be used to record external
commits, each of which we'll treat like a root commit (i.e. depth 1 and
having no parents of its own). Add a test to prevent regressions.
Signed-off-by: Elijah Newren <newren@gmail.com>
While most users of filter-repo will just use it as a tool and
RepoFilter.run() is the final function, filter-repo can be used as a
library with additional work being done after calling that function.
So, simply return from that function when it is done rather than calling
sys.exit.
Signed-off-by: Elijah Newren <newren@gmail.com>
The everything_callback took two arguments, the first being a string
that was the name of the type of the second argument. There was no
point for this argument; someone can just compare type(second) to the
relevant classes. Remove it.
Signed-off-by: Elijah Newren <newren@gmail.com>
The only times this is ever printed is when debugging filter-repo
itself, or when trying to add tests to get to 100% line coverage. But
the printing was broken when objects were skipped (which caused a
mapping from int -> None). Fix the format specifier to handle this
case too.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't expect to ever get progress or checkpoint directives in normal
operation, but the --stdin flag makes it a possibility. In such a case,
the progress directives could actually break our parsing since
git-fast-import will just print it to its stdout, which is what we read
from to find new commit names so we can do commit message hash updating.
So, pass these along to a progress_callback, but don't dump them by
default. Also, it is not clear checkpoint directives make sense given
that we'll be filtering and only getting a subset of history (and I'm
dubious on checkpoint's utility in general anyway as fast-import is
relatively quick), so pass these along to a callback but don't use them
by default.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't run fast-export with rename detection, even though we have
code for handling it, because we decided to use a rev-list|diff-tree
pipeline instead. The code was manually tested and determined to be
working and it might be useful in the future so I don't want to just
outright delete it, but since we know we can't trigger it right now,
add a
# pragma: no cover
on these lines so it doesn't show up on coverage reports.
Signed-off-by: Elijah Newren <newren@gmail.com>
Since there were multiple places in the code where we returned early
knowing that we didn't have a translation of old_hash to a new_hash, we
need to update _commits_referenced_but_removed from each of them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to the invariants we maintain with _commit_renames and
_commit_short_old_hashes (the latter always gets an extra entry in
either a key or a value whenever _commit_renames gains a new key/value
pair), there were a few lines of code that we could not ever reach.
Replace them with an assertion that the condition used for them is
never true.
Signed-off-by: Elijah Newren <newren@gmail.com>
The former logic for keeping track of whether we had seen annotated tags
(and thus whether they were interesting and should avoid being pruned)
was just plain buggy. I do not know if it was that bad from the start
or there was other surrounding code that made it different that got lost
in one of my history rewrites, but fix it. I'll include tests of it
with --subdirectory-filter shortly.
Signed-off-by: Elijah Newren <newren@gmail.com>
We previously would abort if we had been requested to rename files and
that caused two different files to go to the same path. However, if
the files have identical contents and mode, then we can treat the
request as a desire from the user to just coalesce the extra copies.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to pruning of empty commits, merge commits can become degenerate
(same commit serving as both parents, or one parent is an ancestor of
one of the others). While we usually want to allow such degenerate
merge commits to themselves be pruned (assuming they add no additional
file changes), we do not want to prune them if the merge commit in the
original repository had the same degenerate topology. So, we need to
keep track of the ancestry graph of the original repository as well and
include it in the logic about whether to allow merge commits to be
pruned.
Signed-off-by: Elijah Newren <newren@gmail.com>
The reason we want to sometimes keep commits that start empty is because
they may have been intentionally added for build or versioning reasons.
Not all commits that start empty are useful, even if intentional,
though, because they could have pre-dated the introduction of a
directory we are filtering for. So, we always allowed an exception that
if the number of parents had been reduced, we also allow pruning commits
that started empty.
However, there is a similar case: one or more contiguous chunks of
history may only touch some directories/files that are not of interest;
empty commits within that range of history are likewise uninteresting to
us. Since "interesting" empty commits are of the form some new commit
on top of interesting history (because otherwise it loses its special
build or versioning utility), we should loosen the rules to also
consider that empty commits whose parent was pruned are also prunable;
we no longer use the existence of some other distant ancestor of that
empty commit in determining whether the empty commit is prunable.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to the special handling of 'from' in the fast_export stream and the
aggregation of the 'from' commit with the 'merge'd commits, a parentless
commit has its parents represented as [None] rather than []. We had
fixed this up in other places, but forgot to do so with orig_parents,
breaking our comparison. Handle it for orig_parents too.
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import from versions of git up to at least 2.21.0 had a bug in the
handling of the get-mark directive that would cause it to abort with a
parsing error on valid input. While a fix has been submitted upstream
for this, add some extra newlines in a way that will work with both old
and new git versions.
Signed-off-by: Elijah Newren <newren@gmail.com>
Many of the callback functions might only be a single line, and as such
instead of forcing the user to write a full blown program with an import
and everything, let them just specify the body of the callback function
as a command line parameter. Add several tests of this functionality as
well.
Signed-off-by: Elijah Newren <newren@gmail.com>
Add callbacks for:
* filename
simplifies filtering/renaming based solely on filename; return
None to have file removed, or original or new name for file
* message
simplifies tweaking both commit and tag messages; if you want to
tweak just one of the two, use either tag_callback or
commit_callback
* person_name
simplifies tweaking actual names of people without worrying where
they come from (author, committer, or tagger)
* email:
simplifies tweaking email addresses without worrying where they
come from (author, committer, or tagger)
* refname:
simplifies tweaking reference names, regardless of whether they
come from FastExport commit objects, reset objects, or tag objects
Signed-off-by: Elijah Newren <newren@gmail.com>
I want to allow callbacks that could operate on similar pieces of commit
or reset or tag objects (e.g. reference names, email addresses);
restructure the current ones slightly to both allow more general ones to
be added and to make the existing ones slightly clearer.
Signed-off-by: Elijah Newren <newren@gmail.com>
Users may want to run --analyze both before and after filtering in
order to both find the big objects to remove and to verify they are
gone and the overall repository size and filenames are as expected.
As such, aborting and telling the user there's a previous analysis
directory in the way is annoying; just remove it instead.
Signed-off-by: Elijah Newren <newren@gmail.com>
Users may want to run multiple filtering operations, either because it's
easier for them to do it that way, or because they want to combine both
path inclusion and exclusion. For example:
git filter-repo --path subdir
git filter-repo --invert-paths --path subdir/some-big-file
cannot be done in a single step. However, the first filtering operation
would make the repo not look like a clean clone anymore (because it is
not a clean clone anymore), causing the safety check to trigger and
requiring the --force flag. But once we've allowed them to do
repository rewriting, there's no point disallowing further rewriting.
So, write a .git/filter-repo/already_ran file when we run and treat the
presence of that file the same as providing a --force flag.
Signed-off-by: Elijah Newren <newren@gmail.com>
We need to have a list of references to rewrite (which we pass along to
fast-export), but we accepted arbitrary rev-list args. This could
backfire pretty badly if a user tried the wrong but somewhat
straightforward
git filter-repo --invert-paths --path foo bar
instead of the expected
git filter-repo --invert-paths --path foo --path bar
because the passing of 'bar' as a rev-list arg means that fast-export
happily notices that some kind of rev-list was specified but not a
meaningful one so it gives an empty output...and filter-repo interprets
an empty output as "This is a history with no commits" and promptly
deletes everything.
Partial history rewrites aren't yet properly supported anyway (I would
need to stop doing the disconnect-of-origin-remote, the deleting of
references that were filtered-away or otherwise didn't show up, and
the post-rewrite gc+prune). When I add such support, I'll revisit
how these arguments can be specified.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit.dump() showed up in a profile. Reorganize the code slightly to
build up much of the string into one big chunk before calling
file_.write(); this shaves a few percent off the total runtime. (Where
total runtime is again measured in terms of the
cat fast-export.original | git filter-repo --stdin --dry-run ...
trick mentioned a few commits back.) Trying to make a [c]StringIO
object in order to build more of the string up into a single place
to reduce the number of file_.write() calls was apparently
counter-productive, so only the header before the parents is combined
into a single string.
Signed-off-by: Elijah Newren <newren@gmail.com>
We repeatedly hit the same filenames over and over as we traverse
history, but our expressions for renaming or filtering within the
newname() function are based solely on the filename and thus will always
give the same answer. So record any answer we get and just use it
whenever we hit the same filename again.
If the filtering expressions contain only a single short pathname, this
has no measurable effect, but for several paths (e.g. listing all
builtin/*.c files individually in git.git) it can add up to a few
percent of overall runtime.
Signed-off-by: Elijah Newren <newren@gmail.com>
Repeatedly using non-compiled regexes is rather wasteful of resources.
Pre-compile these and use the cached versions.
I ran
git filter-repo --invert-paths --path configure.ac --dry-run
and then for timing ran
cat .git/filter-repo/fast-export.original | time git filter-repo \
--invert-paths --path configure.ac --dry-run --stdin
on the git.git repository (with tags of blobs and tags of tags deleted).
Comparing the timings before and after this change, I see about a 13%
overall speedup just from caching the regexes.
Signed-off-by: Elijah Newren <newren@gmail.com>
Most filtering operations are not interested in the time that commits
were authored or committed, or when tags were tagged. As such,
translating the string representation of the date into a datetime object
is wasted effort, and causes us to waste more time later as we have to
translate it back into a string.
Instead, provide string_to_date() and date_to_string() functions so that
callers can perform the translation if wanted, and let the normal case
be fast.
Provides a small but noticable speedup when just filtering based on
paths; about a 3.5% improvement in execution time for writing the new
history.
Signed-off-by: Elijah Newren <newren@gmail.com>
We have to ask fast-import for the new names of commits, but doing so
immediately upon dumping out the commit related information requires
context switches and waiting for fast-import to parse and handle more
information. We don't need to know the new name of the commit until we
run across a subsequent commit that referenced it in the commit message
by its old ID.
So, speed things up dramatically by waiting until we need the new name
of the commit message (or the fast-import output pipe we are
communicating with should be getting kind of full) before blocking on
reading new commit hashes.
Signed-off-by: Elijah Newren <newren@gmail.com>
Treat fast_import_pipes more like the other parameters to
FastExportFilter.run(), both for consistency, and because it will allow
us to more easily defer doing blocking reads for new commit names until
we actually need to know the new commit hashes corresponding to old
commit ids.
Signed-off-by: Elijah Newren <newren@gmail.com>
Once we rewrite history, our history will be unrelated to our original
upstream. As such, migrate refs/remotes/origin/* to refs/heads/*
(our sanity check already verified that if both names exist they are
equal; if the user used --force then just delete the remote tracking
branch and leave the local branch as is), and then delete the 'origin'
remote.
This has a few advantages:
* People expect to work with refs/heads/*, not refs/remotes/origin/*,
and will be more likely to write filters based on those.
* People will probably need to push the new history somewhere when
they are done, and it's easier if we have it in refs/heads/* than
if it's under refs/remotes/origin/*.
* It encourages people to use good hygiene and not mix old and new
histories. If users really want, they can push the repo (or parts
thereof) back over their original history, but they should have to
take extra steps to do it instead of just a 'git push'.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, despite the git-fast-import.txt documentation, tagger is
optional for both fast-export and fast-import. I suspect this is
because there are several (old) tags in the linux.git repository that
have no tagger, so if we want to test rewriting linux.git history then
we need to make it optional for filter-repo too.
Signed-off-by: Elijah Newren <newren@gmail.com>
Make it easy for users to search and replace text throughout the
repository history. Instead of inventing some new syntax, reuse the
same syntax used by BFG repo filter's --replace-text option, namely,
a file with one expression per line of the form
[regex:|glob:|literal:]$MATCH_EXPR[==>$REPLACEMENT_EXPR]
Where "$MATCH_EXPR" is by default considered to be literal text, but
could be a regex or a glob if the appropriate prefix is used. Also,
$REPLACEMENT_EXPR defaults to '***REMOVED***' if not specified. If
you want a literal '==>' to be part of your $MATCH_EXPR, then you
must also manually specify a replacement expression instead of taking
the default. Some examples:
sup3rs3kr3t
(replaces 'sup3rs3kr3t' with '***REMOVED***')
HeWhoShallNotBeNamed==>Voldemort
(replaces 'HeWhoShallNotBeNamed' with 'Voldemort')
very==>
(replaces 'very' with the empty string)
regex:(\d{2})/(\d{2})/(\d{4})==>\2/\1/\3
(replaces '05/17/2012' with '17/05/2012', and vice-versa)
The format for regex is as from
re.sub(<pattern>, <repl>, <string>) from
https://docs.python.org/2/library/re.html
The <string> comes from file contents of the repo, and you specify
the <pattern> and <repl>.
glob:Copy*t==>Cartel
(replaces 'Copyright' or 'Copyleft' or 'Copy my st' with 'Cartel')
Signed-off-by: Elijah Newren <newren@gmail.com>
When non-merge commits have files in the _files_tweaked set (they were
modified by a blob or commit callback), they may become empty. However,
new_1st_parent is more accurately named
new_1st_parent_if_would_become_non_merge; it will always be None for
non-merge commits. So we need to get the correct parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we only have an output and no input of our own, filter.run() seems
weird to call, especially since it'll only be closing a handle and waiting
for fast-import to finish. Add a finish() synonym for such a case to make
external code callers more legible.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will allow exporting from one repo into a different repo, and
combined with chained RepoFilter instances from commit 81016821a1
(filter-repo: allow chaining of RepoFilter instances, 2019-01-07), will
even allow things like splicing separate repositories together.
Signed-off-by: Elijah Newren <newren@gmail.com>
We do not want to kill fast-import processes unused; it's better
to abort before those processes are created when we know we need to.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow each instance to be just input or just output so that we can splice
repos together or split one into multiple different repos.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we are using --stdin, it should be okay to import into a bare repo,
but the checks were enforcing that we were in a clone with a packfile.
Relax the check to work within a bare repo as well.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we have blob callbacks, we cannot pass --no-data to fast-export. Also,
with blob callbacks, any file the callback modifies could match the
modification done to the file by a subsequent commit, possibly making the
later commit empty. As such, we keep a record of all filenames modified
(by blob or commit callbacks), and then check all these filenames for all
subsequent commits to see if it causes empty commits. In particular, if
files other than these are modified in a non-merge commit, we know that
the commit will not become empty so we can bypass the empty-pruning
checks.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a non-merge commit previously, then since we do not do
any kind of blob modifications (or funny parent grafting), there is no
way for a filemodify instruction to introduce the same version of the
file that already existed in the parent, as such the only check we need
to do to determine whether a commit becomes empty is whether
file_changes is empty. Subsequent more expensive checks can be skipped.
Signed-off-by: Elijah Newren <newren@gmail.com>
Split a lot of the logic out into separate functions, and avoid
flattening parents when the original commit history itself had
redundant parents (such as --no-ff merges).
Signed-off-by: Elijah Newren <newren@gmail.com>
commits may not have any parents at all. As such,
parse_optional_parent_ref() is used expecting that it will sometimes
return None.
Now, when commits are skipped, we have a scheme to translate anyone that
depends on such commits to instead depend on the nearest ancestor of
such commits. If the entire ancestry of a commit was skipped along with
a comit, then that commit will be translated to None, which is
indistinguishable from there having been no parent to begin with.
Sometimes our scheme needs to distinguish between a commit that started
with no parents and one which ended up with no parents, so we need a way
to tell these apart.
Also, not knowing the original parent makes it hard for us to
determine if the original had the same weird topology that the current
commit does. For example, it is possible for a merge commit to have
one parent be the ancestor of another (particularly when --no-ff is
passed to git merge), or even for a merge commit to have the same
commit used as both parents (if you use low-level commands to create
a crazy commit). There are cases where the pruning of some commits
could cause either of these situations to arise, and it's useful to be
able to distinguish between intentionally "weird" history and history
that has been made weird due to other pruning, because the latter we
may have reason to do additional pruning on.
Signed-off-by: Elijah Newren <newren@gmail.com>
Oh, boy, timezone +051800 exists in the wild. Is that 0518 hours and 00
minutes? Or 05 hours and 1800 minutes? Or 051 hours and 800 minutes?
Attempt to do something sane with these broken commits that fast-import
barfs on. Also, fix an old bug in the handling of ahead-of-UTC timezones.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, the default for subprocess stdout is unbuffered; switching
it to buffered yields a huge 40% speedup. Doing this also exposes the
need to add fi_input.flush() calls, highlighting another performance
issue. We may be able to have fewer such calls with some refactoring,
but that is a bigger separate change. Just having them highlighted to
remind about them as a performance issue is good for now.
Signed-off-by: Elijah Newren <newren@gmail.com>
As suggested by Peff, use rev-list & diff-tree to get the information we
need, instead of relying on fast-export (with some out-of-tree patches)
to get that information.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a tag points at a commit whose changes are all filtered out and thus
becomes empty and gets pruned, and all of its ancestors are likewise
pruned, then there is no need for the tag; just nuke it.
Signed-off-by: Elijah Newren <newren@gmail.com>
This option walks through the repository history and creates a report
with basic statistics, rename related information, and sizes of objects
and when/if those have been deleted. It primarily looks at unpacked
sizes (i.e. size of object ignoring delta-ing and compression), and
sums the size of each version of the file for each path. Additionally,
it aggregates these sums by extension and by directory, and tracks
whether paths, extensions, and directories have been deleted. This can
be very useful in determining what the big things are, and whether they
might have been considered to have been mistakes to add to the
repository in the first place.
There are numerous caveats with the determination of "deleted" and
"renamed", and can give both false positives and false negatives. But
they are only meant as a helpful heuristic to give others a starting
point for an investigation, and the information provide so far is useful.
I do want to improve the equivalence classes (rename handling), but that
is for a future commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.
In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash. Instead of dying, note the
suboptimal commit in the suboptimal-issues file.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.
Signed-off-by: Elijah Newren <newren@gmail.com>
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well. Empty commits
from the original repository should only be retained if they have at
least one retained parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is. (e.g. "Merge branch maint into
master"). There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.
Signed-off-by: Elijah Newren <newren@gmail.com>
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated. A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.
Signed-off-by: Elijah Newren <newren@gmail.com>
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself. This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors. In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.
Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.
Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates. However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit. However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
Use the 'feature done' ability to mark when the fast-import stream is
finished, so that an aborted run (due to running into some kind of bug
while filtering, whether a bug in the code, or an error in the repo or
flags specified for the case under consideration) won't cause the repo
to be rewritten.
Signed-off-by: Elijah Newren <newren@gmail.com>
It may be that the only time a reference is shown in the fast-export stream
is for a commit which will become empty due to the filtering. We do not
want such refs to be left out and thus not be updated; we want them to
instead be set to the nearest non-empty ancestor. Only if it has no
non-empty ancestor would we want it to be stripped out.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren