Git clone very slow

Archive Issues
1

Hello,

After our VPN issue that has been dealt with, we now have another issue, our git clone command takes about 30min, to clone a repo of about 400Mo.

We did a curl on a file of 1Go and it succeeded in 2min35 sec

curl -GO http://url_of_our_1Go_file
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 1024M 0 321k 0 0 366k 0 0:47:38 --:–:-- 0:47:38 366k
0 1024M 0 6954k 0 0 3570k 0 0:04:53 0:00:01 0:04:52 3570k
1 1024M 1 14.7M 0 0 5185k 0 0:03:22 0:00:02 0:03:20 5183k
2 1024M 2 23.5M 0 0 6199k 0 0:02:49 0:00:03 0:02:46 6198k
3 1024M 3 32.6M 0 0 6744k 0 0:02:35 0:00:04 0:02:31 6743k
3 1024M 3 39.7M 0 0 6868k 0 0:02:32 0:00:05 0:02:27 7998k
4 1024M 4 47.2M 0 0 7020k 0 0:02:29 0:00:06 0:02:23 8377k
5 1024M 5 55.3M 0 0 7203k 0 0:02:25 0:00:07 0:02:18 8391k
6 1024M 6 63.8M 0 0 7325k 0 0:02:23 0:00:08 0:02:15 8194k
7 1024M 7 72.0M 0 0 7447k 0 0:02:20 0:00:09 0:02:11 8151k
7 1024M 7 80.0M 0 0 7544k 0 0:02:18 0:00:10 0:02:08 8352k
8 1024M 8 89.0M 0 0 7636k 0 0:02:17 0:00:11 0:02:06 8480k
9 1024M 9 97.1M 0 0 7709k 0 0:02:16 0:00:12 0:02:04 8500k
10 1024M 10 103M 0 0 7647k 0 0:02:17 0:00:13 0:02:04 8228k
10 1024M 10 109M 0 0 7481k 0 0:02:20 0:00:14 0:02:06 7550k
11 1024M 11 114M 0 0 7343k 0 0:02:22 0:00:15 0:02:07 6912k
11 1024M 11 119M 0 0 7235k 0 0:02:24 0:00:16 0:02:08 6266k
12 1024M 12 125M 0 0 7141k 0 0:02:26 0:00:17 0:02:09 5685k
12 1024M 12 130M 0 0 7067k 0 0:02:28 0:00:18 0:02:10 5469k
13 1024M 13 135M 0 0 7001k 0 0:02:29 0:00:19 0:02:10 5548k
13 1024M 13 141M 0 0 6952k 0 0:02:30 0:00:20 0:02:10 5703k
14 1024M 14 147M 0 0 6884k 0 0:02:32 0:00:21 0:02:11 5709k
14 1024M 14 152M 0 0 6839k 0 0:02:33 0:00:22 0:02:11 5744k
15 1024M 15 158M 0 0 6812k 0 0:02:33 0:00:23 0:02:10 5845k
16 1024M 16 164M 0 0 6776k 0 0:02:34 0:00:24 0:02:10 5887k
16 1024M 16 171M 0 0 6770k 0 0:02:34 0:00:25 0:02:09 6007k
17 1024M 17 178M 0 0 6800k 0 0:02:34 0:00:26 0:02:08 6429k
18 1024M 18 186M 0 0 6826k 0 0:02:33 0:00:27 0:02:06 6764k
19 1024M 19 194M 0 0 6897k 0 0:02:32 0:00:28 0:02:04 7303k
19 1024M 19 204M 0 0 7008k 0 0:02:29 0:00:29 0:02:00 8178k
20 1024M 20 214M 0 0 7110k 0 0:02:27 0:00:30 0:01:57 8877k
21 1024M 21 222M 0 0 7151k 0 0:02:26 0:00:31 0:01:55 9025k
22 1024M 22 229M 0 0 7146k 0 0:02:26 0:00:32 0:01:54 8950k
23 1024M 23 236M 0 0 7125k 0 0:02:27 0:00:33 0:01:54 8430k
23 1024M 23 242M 0 0 7116k 0 0:02:27 0:00:34 0:01:53 7758k
24 1024M 24 249M 0 0 7114k 0 0:02:27 0:00:35 0:01:52 7137k
25 1024M 25 256M 0 0 7124k 0 0:02:27 0:00:36 0:01:51 6949k
25 1024M 25 263M 0 0 7115k 0 0:02:27 0:00:37 0:01:50 6915k
26 1024M 26 270M 0 0 7117k 0 0:02:27 0:00:38 0:01:49 7061k
27 1024M 27 277M 0 0 7126k 0 0:02:27 0:00:39 0:01:48 7194k
27 1024M 27 284M 0 0 7121k 0 0:02:27 0:00:40 0:01:47 7177k
28 1024M 28 291M 0 0 7126k 0 0:02:27 0:00:41 0:01:46 7140k
29 1024M 29 299M 0 0 7143k 0 0:02:26 0:00:42 0:01:44 7356k
29 1024M 29 306M 0 0 7151k 0 0:02:26 0:00:43 0:01:43 7416k
30 1024M 30 314M 0 0 7176k 0 0:02:26 0:00:44 0:01:42 7575k
31 1024M 31 323M 0 0 7225k 0 0:02:25 0:00:45 0:01:40 8083k
32 1024M 32 333M 0 0 7273k 0 0:02:24 0:00:46 0:01:38 8499k
33 1024M 33 342M 0 0 7331k 0 0:02:23 0:00:47 0:01:36 8936k
34 1024M 34 352M 0 0 7395k 0 0:02:21 0:00:48 0:01:33 9556k
35 1024M 35 362M 0 0 7437k 0 0:02:20 0:00:49 0:01:31 9745k
36 1024M 36 370M 0 0 7453k 0 0:02:20 0:00:50 0:01:30 9527k
37 1024M 37 379M 0 0 7483k 0 0:02:20 0:00:51 0:01:29 9466k
37 1024M 37 389M 0 0 7535k 0 0:02:19 0:00:52 0:01:27 9494k
38 1024M 38 398M 0 0 7573k 0 0:02:18 0:00:53 0:01:25 9300k
39 1024M 39 407M 0 0 7594k 0 0:02:18 0:00:54 0:01:24 9186k
40 1024M 40 414M 0 0 7601k 0 0:02:17 0:00:55 0:01:22 9113k
41 1024M 41 423M 0 0 7616k 0 0:02:17 0:00:56 0:01:21 8976k
42 1024M 42 431M 0 0 7635k 0 0:02:17 0:00:57 0:01:20 8687k
42 1024M 42 440M 0 0 7657k 0 0:02:16 0:00:58 0:01:18 8575k
43 1024M 43 449M 0 0 7683k 0 0:02:16 0:00:59 0:01:17 8648k
44 1024M 44 458M 0 0 7706k 0 0:02:16 0:01:00 0:01:16 8870k
45 1024M 45 467M 0 0 7728k 0 0:02:15 0:01:01 0:01:14 9022k
46 1024M 46 476M 0 0 7751k 0 0:02:15 0:01:02 0:01:13 9087k
47 1024M 47 483M 0 0 7745k 0 0:02:15 0:01:03 0:01:12 8772k
47 1024M 47 489M 0 0 7726k 0 0:02:15 0:01:04 0:01:11 8256k
48 1024M 48 495M 0 0 7691k 0 0:02:16 0:01:05 0:01:11 7511k
48 1024M 48 500M 0 0 7664k 0 0:02:16 0:01:06 0:01:10 6881k
49 1024M 49 506M 0 0 7642k 0 0:02:17 0:01:07 0:01:10 6247k
50 1024M 50 513M 0 0 7619k 0 0:02:17 0:01:08 0:01:09 6024k
50 1024M 50 518M 0 0 7600k 0 0:02:17 0:01:09 0:01:08 5975k
51 1024M 51 524M 0 0 7582k 0 0:02:18 0:01:10 0:01:08 6125k
51 1024M 51 531M 0 0 7569k 0 0:02:18 0:01:11 0:01:07 6278k
52 1024M 52 537M 0 0 7545k 0 0:02:18 0:01:12 0:01:06 6240k
53 1024M 53 543M 0 0 7530k 0 0:02:19 0:01:13 0:01:06 6276k
53 1024M 53 549M 0 0 7521k 0 0:02:19 0:01:14 0:01:05 6401k
54 1024M 54 556M 0 0 7508k 0 0:02:19 0:01:15 0:01:04 6476k
55 1024M 55 563M 0 0 7506k 0 0:02:19 0:01:16 0:01:03 6615k
55 1024M 55 571M 0 0 7515k 0 0:02:19 0:01:17 0:01:02 7077k
56 1024M 56 580M 0 0 7532k 0 0:02:19 0:01:18 0:01:01 7560k
57 1024M 57 588M 0 0 7540k 0 0:02:19 0:01:19 0:01:00 7830k
57 1024M 57 593M 0 0 7514k 0 0:02:19 0:01:20 0:00:59 7601k
58 1024M 58 599M 0 0 7492k 0 0:02:19 0:01:21 0:00:58 7263k
59 1024M 59 605M 0 0 7477k 0 0:02:20 0:01:22 0:00:58 6887k
59 1024M 59 611M 0 0 7463k 0 0:02:20 0:01:23 0:00:57 6364k
60 1024M 60 617M 0 0 7447k 0 0:02:20 0:01:24 0:00:56 5950k
60 1024M 60 623M 0 0 7437k 0 0:02:20 0:01:25 0:00:55 6196k
61 1024M 61 629M 0 0 7425k 0 0:02:21 0:01:26 0:00:55 6339k
62 1024M 62 636M 0 0 7411k 0 0:02:21 0:01:27 0:00:54 6312k
62 1024M 62 642M 0 0 7402k 0 0:02:21 0:01:28 0:00:53 6396k
63 1024M 63 648M 0 0 7393k 0 0:02:21 0:01:29 0:00:52 6479k
64 1024M 64 655M 0 0 7385k 0 0:02:21 0:01:30 0:00:51 6481k
64 1024M 64 662M 0 0 7385k 0 0:02:21 0:01:31 0:00:50 6675k
65 1024M 65 670M 0 0 7388k 0 0:02:21 0:01:32 0:00:49 6984k
66 1024M 66 678M 0 0 7398k 0 0:02:21 0:01:33 0:00:48 7325k
67 1024M 67 687M 0 0 7423k 0 0:02:21 0:01:34 0:00:47 7956k
68 1024M 68 697M 0 0 7449k 0 0:02:20 0:01:35 0:00:45 8619k
68 1024M 68 705M 0 0 7461k 0 0:02:20 0:01:36 0:00:44 8859k
69 1024M 69 712M 0 0 7455k 0 0:02:20 0:01:37 0:00:43 8674k
70 1024M 70 717M 0 0 7432k 0 0:02:21 0:01:38 0:00:43 8059k
70 1024M 70 722M 0 0 7409k 0 0:02:21 0:01:39 0:00:42 7148k
71 1024M 71 728M 0 0 7387k 0 0:02:21 0:01:40 0:00:41 6215k
71 1024M 71 732M 0 0 7363k 0 0:02:22 0:01:41 0:00:41 5492k
71 1024M 71 736M 0 0 7333k 0 0:02:22 0:01:42 0:00:40 4927k
72 1024M 72 741M 0 0 7303k 0 0:02:23 0:01:43 0:00:40 4784k
72 1024M 72 745M 0 0 7278k 0 0:02:24 0:01:44 0:00:40 4684k
73 1024M 73 750M 0 0 7254k 0 0:02:24 0:01:45 0:00:39 4542k
73 1024M 73 754M 0 0 7227k 0 0:02:25 0:01:46 0:00:39 4465k
74 1024M 74 758M 0 0 7194k 0 0:02:25 0:01:47 0:00:38 4352k
74 1024M 74 761M 0 0 7166k 0 0:02:26 0:01:48 0:00:38 4273k
74 1024M 74 766M 0 0 7135k 0 0:02:26 0:01:49 0:00:37 4183k
75 1024M 75 770M 0 0 7116k 0 0:02:27 0:01:50 0:00:37 4208k
75 1024M 75 775M 0 0 7095k 0 0:02:27 0:01:51 0:00:36 4230k
76 1024M 76 780M 0 0 7072k 0 0:02:28 0:01:52 0:00:36 4449k
76 1024M 76 784M 0 0 7051k 0 0:02:28 0:01:53 0:00:35 4569k
77 1024M 77 788M 0 0 7031k 0 0:02:29 0:01:54 0:00:35 4687k
77 1024M 77 793M 0 0 7010k 0 0:02:29 0:01:55 0:00:34 4670k
77 1024M 77 798M 0 0 6992k 0 0:02:29 0:01:56 0:00:33 4704k
78 1024M 78 803M 0 0 6975k 0 0:02:30 0:01:57 0:00:33 4775k
78 1024M 78 808M 0 0 6961k 0 0:02:30 0:01:58 0:00:32 4933k
79 1024M 79 814M 0 0 6953k 0 0:02:30 0:01:59 0:00:31 5183k
80 1024M 80 820M 0 0 6951k 0 0:02:30 0:02:00 0:00:30 5576k
80 1024M 80 828M 0 0 6959k 0 0:02:30 0:02:01 0:00:29 6179k
81 1024M 81 836M 0 0 6972k 0 0:02:30 0:02:02 0:00:28 6894k
82 1024M 82 844M 0 0 6978k 0 0:02:30 0:02:03 0:00:27 7376k
83 1024M 83 851M 0 0 6985k 0 0:02:30 0:02:04 0:00:26 7759k
84 1024M 84 860M 0 0 7000k 0 0:02:29 0:02:05 0:00:24 8174k
84 1024M 84 867M 0 0 7003k 0 0:02:29 0:02:06 0:00:23 8069k
85 1024M 85 874M 0 0 7003k 0 0:02:29 0:02:07 0:00:22 7763k
86 1024M 86 882M 0 0 7006k 0 0:02:29 0:02:08 0:00:21 7694k
86 1024M 86 889M 0 0 7011k 0 0:02:29 0:02:09 0:00:20 7644k
87 1024M 87 896M 0 0 7017k 0 0:02:29 0:02:10 0:00:19 7441k
88 1024M 88 905M 0 0 7024k 0 0:02:29 0:02:11 0:00:18 7558k
89 1024M 89 912M 0 0 7030k 0 0:02:29 0:02:12 0:00:17 7730k
89 1024M 89 920M 0 0 7036k 0 0:02:29 0:02:13 0:00:16 7828k
90 1024M 90 928M 0 0 7043k 0 0:02:28 0:02:14 0:00:14 7884k
91 1024M 91 935M 0 0 7050k 0 0:02:28 0:02:15 0:00:13 7913k
92 1024M 92 943M 0 0 7058k 0 0:02:28 0:02:16 0:00:12 7958k
92 1024M 92 948M 0 0 7045k 0 0:02:28 0:02:17 0:00:11 7451k
93 1024M 93 953M 0 0 7026k 0 0:02:29 0:02:18 0:00:11 6756k
93 1024M 93 957M 0 0 7011k 0 0:02:29 0:02:19 0:00:10 6126k
94 1024M 94 962M 0 0 6997k 0 0:02:29 0:02:20 0:00:09 5543k
94 1024M 94 967M 0 0 6981k 0 0:02:30 0:02:21 0:00:09 4889k
94 1024M 94 971M 0 0 6960k 0 0:02:30 0:02:22 0:00:08 4613k
95 1024M 95 975M 0 0 6942k 0 0:02:31 0:02:23 0:00:08 4576k
95 1024M 95 980M 0 0 6925k 0 0:02:31 0:02:24 0:00:07 4550k
96 1024M 96 985M 0 0 6912k 0 0:02:31 0:02:25 0:00:06 4541k
96 1024M 96 989M 0 0 6900k 0 0:02:31 0:02:26 0:00:05 4586k
97 1024M 97 994M 0 0 6879k 0 0:02:32 0:02:27 0:00:05 4607k
97 1024M 97 998M 0 0 6862k 0 0:02:32 0:02:28 0:00:04 4587k
97 1024M 97 1002M 0 0 6846k 0 0:02:33 0:02:29 0:00:04 4509k
98 1024M 98 1005M 0 0 6826k 0 0:02:33 0:02:30 0:00:03 4290k
98 1024M 98 1009M 0 0 6803k 0 0:02:34 0:02:31 0:00:03 3988k
98 1024M 98 1013M 0 0 6785k 0 0:02:34 0:02:32 0:00:02 3965k
99 1024M 99 1017M 0 0 6769k 0 0:02:34 0:02:33 0:00:01 3948k
99 1024M 99 1020M 0 0 6744k 0 0:02:35 0:02:34 0:00:01 3729k
99 1024M 99 1023M 0 0 6723k 0 0:02:35 0:02:35 --:–:-- 3631k
100 1024M 100 1024M 0 0 6722k 0 0:02:35 0:02:35 --:–:-- 3666k

So we did an iostat check before the clone and after the clone and here is the result:

            disk0       cpu    load average
  KB/t  tps  MB/s  us sy id   1m   5m   15m
 64.94  443 28.08  10  9 81  2.69 1.53 0.64

Cloning into ‘./directory/app’…
Warning: Permanently added ‘github.com,xxx.xxx.xx.xxx’ (ECDSA) to the list of known hosts.

             disk0       cpu    load average
   KB/t  tps  MB/s  us sy id   1m   5m   15m
   63.39   35  2.14   1  1 98  1.45 1.36 1.18

I’m not sure to understand correctly the results of iostat, but I think it shows that the average speed of the i/o is much slower after the clone.

Does anyone have an idea on how we can optimize this?

2

Nobody has a lead for me? :frowning:

3

Hi @jvossen,

I don’t think iostat would help here, it’s simply just an avg of I/O. It does not mean that that I/O capacity would be worse, it means that there was less I/O than before. That could be caused by multiple things, e.g. that MacOS after the boot might do some I/O which, more than your file download.

based on

I’d say it’s not an I/O issue but rather a network one. We don’t know about any issue on our side, as well as you confirmed that downloading from another host is significantly faster - this usually means that the bottleneck is your git server. I’m not sure without the logs of course, but based on what you wrote (e.g. VPN) I’m pretty sure you use a self hosted git server. My best guess is that either the git server itself is slow, but more likely something else between the build servers & your network (where the git server is hosted). Can be a firewall, VPN or network speed (against the open internet). Might also worth to mention that our build machines & servers are all in the US, and in general traffic between the US and e.g. China is quite slow.

  • In which country do you have your git server?
  • Did you try to do the git clone yourself from outside of your network, through VPN? E.g. from your home internet connection.
4

Hi,

Thanks for your answer and time. As I said I wasn’t sure that iostat was really helpful as I’m not familiar with its use.

So to answer your questions, the server is in France, and when I clone the git myself it takes about 2min on my poor home connection. But it’s not really meaningful since I’m in Belgium.

The 1Go file I downloaded is on the same server as our git, so I don’t think that the download speed is the issue somehow. Maybe I could be sure with it if there is a possibility to show the download speed of the git clone, but I have no idea how to do it :confused:

5

I have tested some way to have more information on what is going on during the clone following this:

export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

And I see that there is one line that take all the time:

01:21:43.374734 pkt-line.c:80 packet: clone< c52a0ce2b228b1e59a5323f855c7be61e698eb06 refs/tags/v4.5.3^{}
01:21:43.374763 pkt-line.c:80 packet: clone< 69e20e1a95e281f212f2577b7f5f4f1c54d8ae48 refs/tags/v4.5.5
01:21:43.374809 pkt-line.c:80 packet: clone< f7573425c92b1ca3e30690cc3e57585d8a6bc5d2 refs/tags/v4.5.5^{}
01:21:43.374825 pkt-line.c:80 packet: clone< 0000
01:21:43.410821 pkt-line.c:80 packet: clone> want da43bcb0b16bd3deeb90b3d6338cfd81dc3e7ca9 multi_ack_detailed side-band-64k thin-pack no-progress include-tag ofs-delta deepen-since deepen-not agent=git/2.15.1
01:21:43.410882 pkt-line.c:80 packet: clone> deepen 1
01:21:43.410895 pkt-line.c:80 packet: clone> 0000
01:21:43.506659 pkt-line.c:80 packet: clone< shallow da43bcb0b16bd3deeb90b3d6338cfd81dc3e7ca9
01:21:43.506714 pkt-line.c:80 packet: clone< 0000
01:21:43.506736 pkt-line.c:80 packet: clone> done
01:21:43.602681 pkt-line.c:80 packet: clone< NAK
01:21:43.602951 run-command.c:626 trace: run_command: ‘–shallow-file’ ‘/Users/vagrant/git/proj_dir/proj/.git/shallow.lock’ ‘index-pack’ ‘–stdin’ ‘–fix-thin’ ‘–keep=fetch-pack 2041 on controlosxprode1bhostondemandosx102.local’
01:21:43.608424 git.c:344 trace: built-in: git ‘index-pack’ ‘–stdin’ ‘–fix-thin’ ‘–keep=fetch-pack 2041 on controlosxprode1bhostondemandosx102.local’
01:21:47.279463 pkt-line.c:80 packet: sideband< PACK …
01:52:19.407909 pkt-line.c:80 packet: sideband< 0000
01:52:19.564824 run-command.c:626 trace: run_command: ‘rev-list’ ‘–objects’ ‘–stdin’ ‘–not’ ‘–all’ ‘–quiet’
01:52:19.571300 git.c:344 trace: built-in: git ‘rev-list’ ‘–objects’ ‘–stdin’ ‘–not’ ‘–all’ ‘–quiet’

It seems that It’s the .pack file that is long to download that is about 400mb, I don’t see how I can have more information than that, and or accelerate that download?

6

after adding --progress to the git clone command, it seems that the connection while cloning is quite slow

Receiving objects: 77% (3093/4015), 93.29 MiB | 56.00 KiB/s

7

Did you manage to find out why?
From what you wrote it doesn’t seem to be related to our build workers/virtual machines/infra directly as downloading from another source does not have this issue.

It’s most likely related to your git server, maybe a firewall or similar which limits access to your git server from our build infrastructure (which is hosted in the US) / from the public internet? :thinking:

8

I haven’t had time to check furthermore why it’s slow, and what I could do to fix that. I fear I won’t be able to make it faster :frowning:

I’ll try to find some times next week to check this.

9

Ok it would seem to be a problem with our VPN, we apparently can’t do anything about it, so I guess we can close this topic :frowning:

10

One thing that might worth a try, to switch between the SSH & HTTP(S) git clone URLs. Certain systems might have special restrictions for either or both, but depending on your system / setup it might worth a try to switch from the SSH (git@...) to the HTTPS (https://git...) or the other way around.

Worth to mention that if you switch between these you’ll have to provide authentication information for the other form or both; SSH keys can only be used for the SSH git clone URLs while username+psw auth can only be used for HTTPS urls. You can of course specify both auth that’s not an issue at all, just like you can use both SSH keys and user+psw auth on your own Mac/PC, but when you use an SSH git clone url you need an SSH key while if you use an HTTPS one you need username+psw auth.

  • For SSH auth you can use the Activate SSH key (RSA private key) step (this is usually added to the default configuration by default).
  • For HTTPS auth you can use the Authenticate host with netrc step.
11

Yeah, It’s worth a try. I’ll test it whenever I have a bit of time :slight_smile: thx for the idea :smiley:

12

Just tried, and the download speed is about the same in HTTPS or SSH :frowning: unfortunately I don’t think we’ll be able to do anything about it :frowning:

13

This topic was automatically closed after 13 days. New replies are no longer allowed.