darcs

Issue 2669 cant add linux kernel sources

Title cant add linux kernel sources
Priority bug Status unknown
Milestone Resolved in
Superseder Nosy List tankf33der
Assigned To
Topics

Created on 2020-12-09.14:21:29 by tankf33der, last changed 2022-04-12.13:15:43 by bfrk.

Messages
msg22539 (view) Author: tankf33der Date: 2020-12-09.14:21:24
1. Summarise the issue (what were doing, what went wrong?)

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.tar.xz
$ darcs init linux5x
$ cd linux5x
$ mv ../linux-5.0.tar.xz .
$ tar -xf linux-5.0.tar.xz --strip-components=1
$ rm linux-5.0.tar.xz
$ darcs add -r .
<SKIP>
Adding './Documentation/fb'
Adding './Documentation/fb/api.txt'
Adding './Documentation/fb/arkfb.txt'
Adding './Documentation/fb/aty128fb.txt'
Adding './Documentation/fb/cirrusfb.txt'
Adding './Documentation/fb/cmap_xfbdev.txt'
Adding './Documentation/fb/deferred_io.txt'
Adding './Documentation/fb/efifb.txt'
Adding './Documentation/fb/ep93xx-fb.txt'
Adding './Documentation/fb/fbcon.txt'
Adding './Documentation/fb/framebuffer.txt'
Adding './Documentation/fb/gxfb.txt'
Adding './Documentation/fb/intel810.txt'
Adding './Documentation/fb/intelfb.txt'
Adding './Documentation/fb/internals.txt'
Adding './Documentation/fb/lxfb.txt'
Adding './Documentation/fb/matroxfb.txt'
Adding './Documentation/fb/metronomefb.txt'
Adding './Documentation/fb/modedb.txt'
Adding './Documentation/fb/pvr2fb.txt'
Adding './Documentation/fb/pxafb.txt'
Adding './Documentation/fb/s3fb.txt'
Adding './Documentation/fb/sa1100fb.txt'
Adding './Documentation/fb/sh7760fb.txt'
Adding './Documentation/fb/sisfb.txt'
Adding './Documentation/fb/sm501.txt'
Adding './Documentation/fb/sm712fb.txt'
Adding './Documentation/fb/sstfb.txt'
Adding './Documentation/fb/tgafb.txt'
Adding './Documentation/fb/tridentfb.txt'
Adding './Documentation/fb/udlfb.txt'
Adding './Documentation/fb/uvesafb.txt'
Adding './Documentation/fb/vesafb.txt'
Adding './Documentation/fb/viafb.modes'
Adding './Documentation/fb/viafb.txt'
Adding './Documentation/fb/vt8623fb.txt'
Adding './Documentation/features'
Adding './Documentation/features/arch-support.txt'
Skipping './Documentation/features/core/cBPF-JIT' ... couldn't add parent directory './Documentation/features/core' to repository
No files were added
$

2. What behaviour were you expecting instead?
add all files without error


3. What darcs version are you using? (Try: darcs --exact-version)
 darcs --exact-version
darcs compiled on Oct 24 2020, at 14:39:23

Weak Hash: not available

Context:


[TAG 2.16.3
Ben Franksen <ben.franksen@online.de>**20201022094642
 Ignore-this: bd76d7e31488721e4a5ce7267115e71c3b68d680c155c75c3dd275d9e54933a607d208fab502a143
]


4. What operating system are you running?
void linux, 5.9.13
msg22540 (view) Author: bfrk Date: 2020-12-12.15:11:36
> Skipping './Documentation/features/core/cBPF-JIT' ... couldn't add parent directory './Documentation/features/core' to repository
> No files were added

This is due to the default boring file created for your repo when you
darcs init. You have to remove or comment out the lines containing
"core" and "tags" in _darcs/prefs/boring.

However, you will find that it is not a good idea to use "add, then
record" with so many files. It is far more efficient to use 'darcs
record -l'. But for the linux kernel even that doesn't work, unless you
have huge amounts of memory. On my laptop which has 16G a

  darcs record -lam XXX

runs out of memory pretty soon and then gets oom killed. The only method
I am aware of that works here is to break the initial patch into a
sequence of patches that individually aren't too large. One way to do
that is to first record all directories:

  find . -not -path */_darcs* -type d | xargs darcs add
  darcs record -am "initial add of all directories"

To record the files I created a small shell script; this is only needed
so we can assign each patch another name. The following one-liner does
the trick:

  darcs record -am "add file $1..." "$@" --skip-long-comment

Save as ../record_bunch and chmod+x, then do something like

  find . -type f | grep -vw _darcs | xargs -n 1000 ../record_bunch

Since I did it with a git clone I could use git ls-files instead of find:

  git ls-files | xargs -n 1000 ../record_bunch

This succeeds for me, though it takes some time (72 patches). You can
play with the bunch size, it seems that 2000 works, too, and is overall
faster.
msg22541 (view) Author: bfrk Date: 2020-12-12.16:06:05
A few more experiments show that in practice xargs limits the number of
arguments to some value below 5000 anyway (so that the full command line
doesn't become too long). This means that the simplest method that also
results in the least amount of patches is

  git ls-files | xargs ../record_bunch

Or, if you have a tar ball, do a

  find . -type f > FILES

first (before the darcs init) and then use

  cat FILES | xargs ../record_bunch

(after adapting the boring file and recording the directories as
explained previously). This gives me a repo with 22 patches and the
above command takes ~ 6 minutes to complete. My slightly revised
record_bunch "script" now reads:

  darcs record -qlam "add $1 ... ($# files)" "$@" --skip-long-comment
History
Date User Action Args
2020-12-09 14:21:29tankf33dercreate
2020-12-12 15:11:41bfrksetmessages: + msg22540
2020-12-12 16:06:08bfrksetmessages: + msg22541
2022-04-12 13:15:43bfrksetpriority: bug