While the underlying fast-export and fast-import streams explicitly
separate 'from' commit (first parent) and 'merge' commits (all other
parents), foisting that separation into the Commit object for
filter-repo forces additional places in the code to deal with that
distinction. It results in less clear code, and especially does not
make sense to push upon folks who may want to use filter-repo as a
library.
Signed-off-by: Elijah Newren <newren@gmail.com>
The sorting order of entries written to files in the analysis directory
didn't specify a secondary sort, thus making the order dependent on the
random-ish sorting order of dictionaries and making it inconsistent
between python versions. While the secondary order didn't matter much,
having a defined order makes it slightly easier to define a single
testcase that can work across versions.
Signed-off-by: Elijah Newren <newren@gmail.com>
Assuming filter-repo will be merged into git.git, use "git" for the
TEXTDOMAIN, and assume its build system will replace "@@LOCALEDIR@@"
appropriately.
Note that the xgettext command used to grab string translations is
nearly identical to the one for C files in git.git; just use
--language=python instead and add --join-existing to avoid overwriting
the po/git.pot file. In other words, use the command:
xgettext -o../git/po/git.pot --join-existing --force-po \
--add-comments=TRANSLATORS: \
--msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
--from-code=UTF-8 --language=python \
--keyword=_ --keyword=N_ --keyword="Q_:1,2" \
git-filter-repo
To create or update the translation, go to git.git/po and run either of:
msginit --locale=XX
msgmerge --add-location --backup=off -U XX.po git.pot
Once you've updated the translation, within git.git just build as
normal. That's all that's needed.
Signed-off-by: Elijah Newren <newren@gmail.com>
Over a decade ago, I added code to deal with splitting and splicing
repositories where you weren't always dealing with first parents and
linear histories, and in particular where the mainline tended to be the
second parent (because there was no integrator or special central
gatekeeper like gerrit or github; instead, everyone pushed directly to
the main repository after locally testing, and integration happened via
everyone running 'git pull'). When attempting to splice repositories
the fact that fast-export always gave changes relative to the first
parent caused some grief with my splitting and splicing efforts.
It has been over a decade, I don't know of a good testcase of this
functionality separate from the live repositories I lost access to over
six years ago, git-subtree was released in the meantime which I'm
certain handled the task better, git-fast-export since gained a
--full-tree option which might have provided a better way to attack the
problem (though with splicing repos you often want work with additive
changes rather than recreating from scratch), and I just don't
quite understand the code anymore anyway. I think it had some
fundamental limitations that I knew my usecase avoided, but I don't
remember the details (and I'm not certain if this is true).
Even though code coverage hits all but one of the lines, I'd rather
rewrite any needed functionality if the usecase arises, and in view of
what facilities exist today rather than what I was working with a decade
ago. So, just nuke this code.
Signed-off-by: Elijah Newren <newren@gmail.com>
The original idea was to add --path-rename-(glob|regex) options, but
I like the general flexibility of --filename-callback better for
special cases and keeping the number of command line options at least
slightly in check.
Signed-off-by: Elijah Newren <newren@gmail.com>
There are several lines equivalent to BUG() calls in git that are
supposed to be unreachable, and which exist just to make debugging the
fundamental system problem or refactoring of the code slightly easier by
trying to give a more immediate notification of a problem. If these
error cases are ever hit and happen to be wrong, then the individual
will at worst get a stacktrace and the program will abort...but that
might arguably be even more helpful. Since there is no harm in avoiding
the work of finding ways to break the system to force these lines to be
covered, simply exclude them from line coverage counting.
Signed-off-by: Elijah Newren <newren@gmail.com>
The AncestryGraph setup assumed we had previously seen all commits which
would be used as parents; that interacted badly with doing an
incremental import. Add a function which can be used to record external
commits, each of which we'll treat like a root commit (i.e. depth 1 and
having no parents of its own). Add a test to prevent regressions.
Signed-off-by: Elijah Newren <newren@gmail.com>
While most users of filter-repo will just use it as a tool and
RepoFilter.run() is the final function, filter-repo can be used as a
library with additional work being done after calling that function.
So, simply return from that function when it is done rather than calling
sys.exit.
Signed-off-by: Elijah Newren <newren@gmail.com>
The everything_callback took two arguments, the first being a string
that was the name of the type of the second argument. There was no
point for this argument; someone can just compare type(second) to the
relevant classes. Remove it.
Signed-off-by: Elijah Newren <newren@gmail.com>
The only times this is ever printed is when debugging filter-repo
itself, or when trying to add tests to get to 100% line coverage. But
the printing was broken when objects were skipped (which caused a
mapping from int -> None). Fix the format specifier to handle this
case too.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't expect to ever get progress or checkpoint directives in normal
operation, but the --stdin flag makes it a possibility. In such a case,
the progress directives could actually break our parsing since
git-fast-import will just print it to its stdout, which is what we read
from to find new commit names so we can do commit message hash updating.
So, pass these along to a progress_callback, but don't dump them by
default. Also, it is not clear checkpoint directives make sense given
that we'll be filtering and only getting a subset of history (and I'm
dubious on checkpoint's utility in general anyway as fast-import is
relatively quick), so pass these along to a callback but don't use them
by default.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't run fast-export with rename detection, even though we have
code for handling it, because we decided to use a rev-list|diff-tree
pipeline instead. The code was manually tested and determined to be
working and it might be useful in the future so I don't want to just
outright delete it, but since we know we can't trigger it right now,
add a
# pragma: no cover
on these lines so it doesn't show up on coverage reports.
Signed-off-by: Elijah Newren <newren@gmail.com>
Since there were multiple places in the code where we returned early
knowing that we didn't have a translation of old_hash to a new_hash, we
need to update _commits_referenced_but_removed from each of them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to the invariants we maintain with _commit_renames and
_commit_short_old_hashes (the latter always gets an extra entry in
either a key or a value whenever _commit_renames gains a new key/value
pair), there were a few lines of code that we could not ever reach.
Replace them with an assertion that the condition used for them is
never true.
Signed-off-by: Elijah Newren <newren@gmail.com>
The former logic for keeping track of whether we had seen annotated tags
(and thus whether they were interesting and should avoid being pruned)
was just plain buggy. I do not know if it was that bad from the start
or there was other surrounding code that made it different that got lost
in one of my history rewrites, but fix it. I'll include tests of it
with --subdirectory-filter shortly.
Signed-off-by: Elijah Newren <newren@gmail.com>
We previously would abort if we had been requested to rename files and
that caused two different files to go to the same path. However, if
the files have identical contents and mode, then we can treat the
request as a desire from the user to just coalesce the extra copies.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to pruning of empty commits, merge commits can become degenerate
(same commit serving as both parents, or one parent is an ancestor of
one of the others). While we usually want to allow such degenerate
merge commits to themselves be pruned (assuming they add no additional
file changes), we do not want to prune them if the merge commit in the
original repository had the same degenerate topology. So, we need to
keep track of the ancestry graph of the original repository as well and
include it in the logic about whether to allow merge commits to be
pruned.
Signed-off-by: Elijah Newren <newren@gmail.com>
The reason we want to sometimes keep commits that start empty is because
they may have been intentionally added for build or versioning reasons.
Not all commits that start empty are useful, even if intentional,
though, because they could have pre-dated the introduction of a
directory we are filtering for. So, we always allowed an exception that
if the number of parents had been reduced, we also allow pruning commits
that started empty.
However, there is a similar case: one or more contiguous chunks of
history may only touch some directories/files that are not of interest;
empty commits within that range of history are likewise uninteresting to
us. Since "interesting" empty commits are of the form some new commit
on top of interesting history (because otherwise it loses its special
build or versioning utility), we should loosen the rules to also
consider that empty commits whose parent was pruned are also prunable;
we no longer use the existence of some other distant ancestor of that
empty commit in determining whether the empty commit is prunable.
Signed-off-by: Elijah Newren <newren@gmail.com>
Due to the special handling of 'from' in the fast_export stream and the
aggregation of the 'from' commit with the 'merge'd commits, a parentless
commit has its parents represented as [None] rather than []. We had
fixed this up in other places, but forgot to do so with orig_parents,
breaking our comparison. Handle it for orig_parents too.
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import from versions of git up to at least 2.21.0 had a bug in the
handling of the get-mark directive that would cause it to abort with a
parsing error on valid input. While a fix has been submitted upstream
for this, add some extra newlines in a way that will work with both old
and new git versions.
Signed-off-by: Elijah Newren <newren@gmail.com>
Many of the callback functions might only be a single line, and as such
instead of forcing the user to write a full blown program with an import
and everything, let them just specify the body of the callback function
as a command line parameter. Add several tests of this functionality as
well.
Signed-off-by: Elijah Newren <newren@gmail.com>
Add callbacks for:
* filename
simplifies filtering/renaming based solely on filename; return
None to have file removed, or original or new name for file
* message
simplifies tweaking both commit and tag messages; if you want to
tweak just one of the two, use either tag_callback or
commit_callback
* person_name
simplifies tweaking actual names of people without worrying where
they come from (author, committer, or tagger)
* email:
simplifies tweaking email addresses without worrying where they
come from (author, committer, or tagger)
* refname:
simplifies tweaking reference names, regardless of whether they
come from FastExport commit objects, reset objects, or tag objects
Signed-off-by: Elijah Newren <newren@gmail.com>
I want to allow callbacks that could operate on similar pieces of commit
or reset or tag objects (e.g. reference names, email addresses);
restructure the current ones slightly to both allow more general ones to
be added and to make the existing ones slightly clearer.
Signed-off-by: Elijah Newren <newren@gmail.com>
Users may want to run --analyze both before and after filtering in
order to both find the big objects to remove and to verify they are
gone and the overall repository size and filenames are as expected.
As such, aborting and telling the user there's a previous analysis
directory in the way is annoying; just remove it instead.
Signed-off-by: Elijah Newren <newren@gmail.com>
Users may want to run multiple filtering operations, either because it's
easier for them to do it that way, or because they want to combine both
path inclusion and exclusion. For example:
git filter-repo --path subdir
git filter-repo --invert-paths --path subdir/some-big-file
cannot be done in a single step. However, the first filtering operation
would make the repo not look like a clean clone anymore (because it is
not a clean clone anymore), causing the safety check to trigger and
requiring the --force flag. But once we've allowed them to do
repository rewriting, there's no point disallowing further rewriting.
So, write a .git/filter-repo/already_ran file when we run and treat the
presence of that file the same as providing a --force flag.
Signed-off-by: Elijah Newren <newren@gmail.com>
We need to have a list of references to rewrite (which we pass along to
fast-export), but we accepted arbitrary rev-list args. This could
backfire pretty badly if a user tried the wrong but somewhat
straightforward
git filter-repo --invert-paths --path foo bar
instead of the expected
git filter-repo --invert-paths --path foo --path bar
because the passing of 'bar' as a rev-list arg means that fast-export
happily notices that some kind of rev-list was specified but not a
meaningful one so it gives an empty output...and filter-repo interprets
an empty output as "This is a history with no commits" and promptly
deletes everything.
Partial history rewrites aren't yet properly supported anyway (I would
need to stop doing the disconnect-of-origin-remote, the deleting of
references that were filtered-away or otherwise didn't show up, and
the post-rewrite gc+prune). When I add such support, I'll revisit
how these arguments can be specified.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit.dump() showed up in a profile. Reorganize the code slightly to
build up much of the string into one big chunk before calling
file_.write(); this shaves a few percent off the total runtime. (Where
total runtime is again measured in terms of the
cat fast-export.original | git filter-repo --stdin --dry-run ...
trick mentioned a few commits back.) Trying to make a [c]StringIO
object in order to build more of the string up into a single place
to reduce the number of file_.write() calls was apparently
counter-productive, so only the header before the parents is combined
into a single string.
Signed-off-by: Elijah Newren <newren@gmail.com>
We repeatedly hit the same filenames over and over as we traverse
history, but our expressions for renaming or filtering within the
newname() function are based solely on the filename and thus will always
give the same answer. So record any answer we get and just use it
whenever we hit the same filename again.
If the filtering expressions contain only a single short pathname, this
has no measurable effect, but for several paths (e.g. listing all
builtin/*.c files individually in git.git) it can add up to a few
percent of overall runtime.
Signed-off-by: Elijah Newren <newren@gmail.com>
Repeatedly using non-compiled regexes is rather wasteful of resources.
Pre-compile these and use the cached versions.
I ran
git filter-repo --invert-paths --path configure.ac --dry-run
and then for timing ran
cat .git/filter-repo/fast-export.original | time git filter-repo \
--invert-paths --path configure.ac --dry-run --stdin
on the git.git repository (with tags of blobs and tags of tags deleted).
Comparing the timings before and after this change, I see about a 13%
overall speedup just from caching the regexes.
Signed-off-by: Elijah Newren <newren@gmail.com>
Most filtering operations are not interested in the time that commits
were authored or committed, or when tags were tagged. As such,
translating the string representation of the date into a datetime object
is wasted effort, and causes us to waste more time later as we have to
translate it back into a string.
Instead, provide string_to_date() and date_to_string() functions so that
callers can perform the translation if wanted, and let the normal case
be fast.
Provides a small but noticable speedup when just filtering based on
paths; about a 3.5% improvement in execution time for writing the new
history.
Signed-off-by: Elijah Newren <newren@gmail.com>
We have to ask fast-import for the new names of commits, but doing so
immediately upon dumping out the commit related information requires
context switches and waiting for fast-import to parse and handle more
information. We don't need to know the new name of the commit until we
run across a subsequent commit that referenced it in the commit message
by its old ID.
So, speed things up dramatically by waiting until we need the new name
of the commit message (or the fast-import output pipe we are
communicating with should be getting kind of full) before blocking on
reading new commit hashes.
Signed-off-by: Elijah Newren <newren@gmail.com>
Treat fast_import_pipes more like the other parameters to
FastExportFilter.run(), both for consistency, and because it will allow
us to more easily defer doing blocking reads for new commit names until
we actually need to know the new commit hashes corresponding to old
commit ids.
Signed-off-by: Elijah Newren <newren@gmail.com>
Once we rewrite history, our history will be unrelated to our original
upstream. As such, migrate refs/remotes/origin/* to refs/heads/*
(our sanity check already verified that if both names exist they are
equal; if the user used --force then just delete the remote tracking
branch and leave the local branch as is), and then delete the 'origin'
remote.
This has a few advantages:
* People expect to work with refs/heads/*, not refs/remotes/origin/*,
and will be more likely to write filters based on those.
* People will probably need to push the new history somewhere when
they are done, and it's easier if we have it in refs/heads/* than
if it's under refs/remotes/origin/*.
* It encourages people to use good hygiene and not mix old and new
histories. If users really want, they can push the repo (or parts
thereof) back over their original history, but they should have to
take extra steps to do it instead of just a 'git push'.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, despite the git-fast-import.txt documentation, tagger is
optional for both fast-export and fast-import. I suspect this is
because there are several (old) tags in the linux.git repository that
have no tagger, so if we want to test rewriting linux.git history then
we need to make it optional for filter-repo too.
Signed-off-by: Elijah Newren <newren@gmail.com>
Make it easy for users to search and replace text throughout the
repository history. Instead of inventing some new syntax, reuse the
same syntax used by BFG repo filter's --replace-text option, namely,
a file with one expression per line of the form
[regex:|glob:|literal:]$MATCH_EXPR[==>$REPLACEMENT_EXPR]
Where "$MATCH_EXPR" is by default considered to be literal text, but
could be a regex or a glob if the appropriate prefix is used. Also,
$REPLACEMENT_EXPR defaults to '***REMOVED***' if not specified. If
you want a literal '==>' to be part of your $MATCH_EXPR, then you
must also manually specify a replacement expression instead of taking
the default. Some examples:
sup3rs3kr3t
(replaces 'sup3rs3kr3t' with '***REMOVED***')
HeWhoShallNotBeNamed==>Voldemort
(replaces 'HeWhoShallNotBeNamed' with 'Voldemort')
very==>
(replaces 'very' with the empty string)
regex:(\d{2})/(\d{2})/(\d{4})==>\2/\1/\3
(replaces '05/17/2012' with '17/05/2012', and vice-versa)
The format for regex is as from
re.sub(<pattern>, <repl>, <string>) from
https://docs.python.org/2/library/re.html
The <string> comes from file contents of the repo, and you specify
the <pattern> and <repl>.
glob:Copy*t==>Cartel
(replaces 'Copyright' or 'Copyleft' or 'Copy my st' with 'Cartel')
Signed-off-by: Elijah Newren <newren@gmail.com>
When non-merge commits have files in the _files_tweaked set (they were
modified by a blob or commit callback), they may become empty. However,
new_1st_parent is more accurately named
new_1st_parent_if_would_become_non_merge; it will always be None for
non-merge commits. So we need to get the correct parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we only have an output and no input of our own, filter.run() seems
weird to call, especially since it'll only be closing a handle and waiting
for fast-import to finish. Add a finish() synonym for such a case to make
external code callers more legible.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will allow exporting from one repo into a different repo, and
combined with chained RepoFilter instances from commit 81016821a1
(filter-repo: allow chaining of RepoFilter instances, 2019-01-07), will
even allow things like splicing separate repositories together.
Signed-off-by: Elijah Newren <newren@gmail.com>
We do not want to kill fast-import processes unused; it's better
to abort before those processes are created when we know we need to.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow each instance to be just input or just output so that we can splice
repos together or split one into multiple different repos.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we are using --stdin, it should be okay to import into a bare repo,
but the checks were enforcing that we were in a clone with a packfile.
Relax the check to work within a bare repo as well.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we have blob callbacks, we cannot pass --no-data to fast-export. Also,
with blob callbacks, any file the callback modifies could match the
modification done to the file by a subsequent commit, possibly making the
later commit empty. As such, we keep a record of all filenames modified
(by blob or commit callbacks), and then check all these filenames for all
subsequent commits to see if it causes empty commits. In particular, if
files other than these are modified in a non-merge commit, we know that
the commit will not become empty so we can bypass the empty-pruning
checks.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a non-merge commit previously, then since we do not do
any kind of blob modifications (or funny parent grafting), there is no
way for a filemodify instruction to introduce the same version of the
file that already existed in the parent, as such the only check we need
to do to determine whether a commit becomes empty is whether
file_changes is empty. Subsequent more expensive checks can be skipped.
Signed-off-by: Elijah Newren <newren@gmail.com>
Split a lot of the logic out into separate functions, and avoid
flattening parents when the original commit history itself had
redundant parents (such as --no-ff merges).
Signed-off-by: Elijah Newren <newren@gmail.com>
commits may not have any parents at all. As such,
parse_optional_parent_ref() is used expecting that it will sometimes
return None.
Now, when commits are skipped, we have a scheme to translate anyone that
depends on such commits to instead depend on the nearest ancestor of
such commits. If the entire ancestry of a commit was skipped along with
a comit, then that commit will be translated to None, which is
indistinguishable from there having been no parent to begin with.
Sometimes our scheme needs to distinguish between a commit that started
with no parents and one which ended up with no parents, so we need a way
to tell these apart.
Also, not knowing the original parent makes it hard for us to
determine if the original had the same weird topology that the current
commit does. For example, it is possible for a merge commit to have
one parent be the ancestor of another (particularly when --no-ff is
passed to git merge), or even for a merge commit to have the same
commit used as both parents (if you use low-level commands to create
a crazy commit). There are cases where the pruning of some commits
could cause either of these situations to arise, and it's useful to be
able to distinguish between intentionally "weird" history and history
that has been made weird due to other pruning, because the latter we
may have reason to do additional pruning on.
Signed-off-by: Elijah Newren <newren@gmail.com>
Oh, boy, timezone +051800 exists in the wild. Is that 0518 hours and 00
minutes? Or 05 hours and 1800 minutes? Or 051 hours and 800 minutes?
Attempt to do something sane with these broken commits that fast-import
barfs on. Also, fix an old bug in the handling of ahead-of-UTC timezones.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, the default for subprocess stdout is unbuffered; switching
it to buffered yields a huge 40% speedup. Doing this also exposes the
need to add fi_input.flush() calls, highlighting another performance
issue. We may be able to have fewer such calls with some refactoring,
but that is a bigger separate change. Just having them highlighted to
remind about them as a performance issue is good for now.
Signed-off-by: Elijah Newren <newren@gmail.com>
As suggested by Peff, use rev-list & diff-tree to get the information we
need, instead of relying on fast-export (with some out-of-tree patches)
to get that information.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a tag points at a commit whose changes are all filtered out and thus
becomes empty and gets pruned, and all of its ancestors are likewise
pruned, then there is no need for the tag; just nuke it.
Signed-off-by: Elijah Newren <newren@gmail.com>
This option walks through the repository history and creates a report
with basic statistics, rename related information, and sizes of objects
and when/if those have been deleted. It primarily looks at unpacked
sizes (i.e. size of object ignoring delta-ing and compression), and
sums the size of each version of the file for each path. Additionally,
it aggregates these sums by extension and by directory, and tracks
whether paths, extensions, and directories have been deleted. This can
be very useful in determining what the big things are, and whether they
might have been considered to have been mistakes to add to the
repository in the first place.
There are numerous caveats with the determination of "deleted" and
"renamed", and can give both false positives and false negatives. But
they are only meant as a helpful heuristic to give others a starting
point for an investigation, and the information provide so far is useful.
I do want to improve the equivalence classes (rename handling), but that
is for a future commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.
In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash. Instead of dying, note the
suboptimal commit in the suboptimal-issues file.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.
Signed-off-by: Elijah Newren <newren@gmail.com>
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well. Empty commits
from the original repository should only be retained if they have at
least one retained parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is. (e.g. "Merge branch maint into
master"). There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.
Signed-off-by: Elijah Newren <newren@gmail.com>
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated. A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.
Signed-off-by: Elijah Newren <newren@gmail.com>
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself. This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors. In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.
Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.
Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates. However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit. However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
Use the 'feature done' ability to mark when the fast-import stream is
finished, so that an aborted run (due to running into some kind of bug
while filtering, whether a bug in the code, or an error in the repo or
flags specified for the case under consideration) won't cause the repo
to be rewritten.
Signed-off-by: Elijah Newren <newren@gmail.com>
It may be that the only time a reference is shown in the fast-export stream
is for a commit which will become empty due to the filtering. We do not
want such refs to be left out and thus not be updated; we want them to
instead be set to the nearest non-empty ancestor. Only if it has no
non-empty ancestor would we want it to be stripped out.
Signed-off-by: Elijah Newren <newren@gmail.com>
Slightly re-order the code to make input, output, and filtering sections
distinct. Also, avoid running `git fast-import` at all when we're in
--dry-run mode.
Signed-off-by: Elijah Newren <newren@gmail.com>
We always called record_id_rename with handle_transitivity set to True, and
I do not know of a use case that would do otherwise so let's just hardcode
that value.
Signed-off-by: Elijah Newren <newren@gmail.com>
My idea to use --export-marks and --import-marks to avoid the need for the
id_offset was not tested and apparently a bad idea. When splicing together
multiple repositories, the second will croak if we pass it --import-marks
with a file having sha1sums that don't exist in that repository.
I'm afraid this might conflict with the --import-marks stuff used in collab
so I've only enabled it for streams beyond the first. So there might be an
issue using --import-marks on a second or later fast-export output stream,
but I can't think of a use case for that...
Signed-off-by: Elijah Newren <newren@gmail.com>
Filtering input from multiple repositories can still be done; however, to
avoid overloading of mark numbers, one should pass --export-marks=<file>
to the first git fast-export and pass --import-marks=<file> to the second.
Signed-off-by: Elijah Newren <newren@gmail.com>
git-fast-import requires that file changes listed in a merge commit be
relative to the first parent. Thus, if I've added new files on a branch
being merged in from the second or later parents, I need to manually
modify the list of files in the merge commit as well. In order to do that,
as soon as I splice in any commit, I have to record the list of new files
for both that commit and every descendant it has.
Signed-off-by: Elijah Newren <newren@gmail.com>
Also, provide an OutputStream class, to make it easy to still direct all
output to some file rather than always sending to git fast-import.
Signed-off-by: Elijah Newren <newren@gmail.com>
Automatically do renaming of references to commits that were skipped, and
automatically remove skipped blobs from the output of commits that
reference them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Explicitly specify --topo-order; git-fast-export fails on some topologies
unless it traverses in topological order.
Signed-off-by: Elijah Newren <newren@gmail.com>
pyparsing sucks a whole file into memory at a time and then parses, which
is really bad in this case since the output from git-fast-export is huge.
I entered disk swapping madness pretty easily. So, now I just do my own
manual parsing.
Signed-off-by: Elijah Newren <newren@gmail.com>
* Allow hooking up (and filtering) multiple git fast-export's to one import
* Allow user callbacks to force dumping of object in order to reference it
with subsequent inserted objects
* Put the separate callbacks and global vars in the calling program into a
combined class
Signed-off-by: Elijah Newren <newren@gmail.com>
My prior handlings of marks would only work if there were not additions
or removals from the fast-export stream. Further, I referred to these as
marks even though I really only accept idnum values, not sha1s or anything
else. So, now I refer to these as ids everywhere, and I am much more
careful in my handling of ids.
Signed-off-by: Elijah Newren <newren@gmail.com>
The commit_callback call was trying to pass a Reset object, which was
not defined. Copy-n-paste-n-forget-to-replace isn't good. Now it passes
a Commit object.
Signed-off-by: Elijah Newren <newren@gmail.com>
We still only parse a single blob, but this should put the infrastructure
in place for parsing more output from git-fast-export.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
James Foucar