Cannot find tag when cloning repository

Question & Answer Builds
1

Bitrise Build Issue Report template

Description of the issue

Sometimes when I push tag and start a build manually, the build failed at the clone step saying tag is not present.

It’s really clear why it’s not working correctly as cloning to /tmp dir on my work machine list all the tags correctly.

At 10:29 (EST), I pushed a tag v1.0.1.beta.4 to the repository:

> 10:29:50 directlink-android git:(master)
$ git push origin v1.0.1.beta.4
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 476 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:samsao/directlink-android.git
 * [new tag]           v1.0.1.beta.4 -> v1.0.1.beta.4

Then started a build with a specific tag (https://www.bitrise.io/build/6e06bb5615452093). This one failed saying the tag does not exist:

...
error: pathspec 'v1.0.1.beta.4' did not match any file(s) known to git.
Error: Could not do checkout v1.0.1.beta.4
Clone Is Success?: false
Cloned commit hash: 
 (i) Removing private key file: /root/.ssh/bitrise
rm: invalid option -- 'P'
Try 'rm --help' for more information.

Indeed, in the log tag listing, I do not see it in the list. I rebuild this build and failed with the same error message (https://www.bitrise.io/build/39223fcbbfbc27db).

Then, I clone the repository on my /tmp folder to check if it would work from my side:

> 10:43:17 directlink-android git:(master)
$ cd /tmp

> 10:43:26 /tmp
$ git clone git@github.com:samsao/directlink-android.git
Cloning into 'directlink-android'...
remote: Counting objects: 28918, done.
remote: Compressing objects: 100% (383/383), done.
remote: Total 28918 (delta 663), reused 467 (delta 467), pack-reused 27951
Receiving objects: 100% (28918/28918), 12.08 MiB | 2.42 MiB/s, done.
Resolving deltas: 100% (16149/16149), done.

> 10:43:38 /tmp
$ cd directlink-android

> 10:43:44 /tmp/directlink-android git:(develop)
$ git tag -l
EditConfigurationPreviousInstallFlow
v0.5.0
v0.6.0
v0.7.0
v0.8.0
v1.0.0
v1.0.0.beta.16
v1.0.0.beta.17
v1.0.0.beta.18
v1.0.1.beta.1
v1.0.1.beta.2
v1.0.1.beta.3
v1.0.1.beta.4
v2.0.0.beta.10
v2.0.0.beta.11
v2.0.0.beta.12
v2.0.0.beta.13
v2.0.0.beta.14
v2.0.0.beta.15
v2.0.0.beta.2
v2.0.0.beta.3
v2.0.0.beta.5
v2.0.0.beta.6
v2.0.0.beta.7
v2.0.0.beta.8
v2.0.0.beta.9

> 10:43:45 directlink-android git:(develop)
$ git tag -l | grep v1.0.1.beta.4
v1.0.1.beta.4

Timestamp in EST time. After that, I started another build (https://www.bitrise.io/build/79c19e0278409a66) but it failed again with the same error. You will see that the build was started after the clone on my workstation, so I really don’t see why it cannot find the tag.

It’s really unclear why the tag cannot be retrieved. Seems as a cache problem but why? This happened once in the past for me and waiting a little bit and building again “fixed” the issue.

Environment:

Android Stack

The clone repository step

Reproducibility

  • Does a “Rebuild” help? (You can trigger a rebuild from the Build’s page, by clicking the “Rebuild” button in the top right corner of a finished build) : no (at least no the two other tries I tried)
  • Does a rebuild without caches help? (You can remove the Cache:Pull and Cache:Push steps temporarily to not to use the cache, or you can delete all the caches on the Settings tab of the app. : would need to try
  • If you have multiple different build configurations (workflows), does the issue affect all/more than one? : yes
  • If it’s an issue which happens sporadically, what’s the frequency? (e.g. Once a day ; about x% of the builds) : Hard to say
  • Does upgrading the build Step to the latest version help? : Did not tried

Linux/Android stack builds

Works locally.

Other stacks

Works locally using CLI.

Build log

2

Took a look at your build log, the key part is this:

+---+---------------------------------------------------------------+----------+
| e[31;1mxe[0m | e[31;1mClone repository (exit code: 1)e[0m                               | 5.25 sec |
+---+---------------------------------------------------------------+----------+
| Update available: 3.2.0 -> 3.4.1                                             |
| Issue tracker: https://github.com/bitrise-io/steps-git-clone/issues          |
| Source: https://github.com/bitrise-io/steps-git-clone                        |
+---+---------------------------------------------------------------+----------+

Please upgrade the git clone step to the latest version, and let us know if you’d still have this issue with that! I believe this was fixed in either 3.3.x or 3.4.x - most likely in this release:

3

Thanks, I’m trying it right now.

In the mean time, before your reply, I tried using the commit hash directly. And this failed also.

So the problem is caused because the commit is not reachable via any branch (only via tags). I tried again with latest clone step (3.4.1) and it failed again using commit hash id. So I can simple reproducible step for hash being not reachable is:

# Make some changes to repository

$ git commit -m "Testing"

$ git tag will-fail-for-hash

$ git log --format="%H" -n 1 will-fail-for-hash
# Copy the commit hash to clipboard

# Launch a build using the commit hash
# The clone step will complain that hash does not exist

This is caused because the hash is reachable only via a tag and because the git "fetch" "--tags" seems to be issued only when using the tag field in the UI. The tags should probably be always fetched otherwise, using the hash does not work.

For the tag, it seems indeed latest version fixed the problem by issuing a git "fetch" "--tags" command that make the tag available.

You can close this issue.

4

I’d say you’re right, but also that you probably forgot a git push - at least I don’t see it in your script above. In that case indeed git won’t find the hash, unless you push it to a branch - git push origin TAG won’t push the commit to a branch, so the commit won’t be available for a git checkout directly.

I’m not sure if there’s anything we can/should do, it’s more about how git works, but we’re always open to discuss improvements and changes!

Happy Building! :slight_smile:

5

Haha yes indeed, the git push is indeed missing from the steps, oups.

The tags are valid referential that can leads to commit hash id. I think it’s probably the only to keep commit hash valid in git without using branch.

As such, to fix the problem, you should always make the git "fetch" "--tags" (or use a git clone that fetches tags).

If I do a simple git clone, commit hash referenced by a tag are available. So, I’m expecting they are also reachable from Bitrise.

1 Like
6

That’s a great idea, although it might slow down builds which don’t require the full tag history…

It’s a bit tricky to find the right balance, we try to optimize for the checkout speed because that’s what most users ask for, but we’ll run a couple of tests with this :wink:

7

Yeah I know, not an easy task.

The only suggestion for that would be do have the condition:

if (build is_from tag || build is_from hash) {
   // Fetch tags
}

That would avoid a bunch of tags fetching for build building from a branch.

Anyway, my use cases is more for tag. So, we can see this issue as being solved!

Regards,
Matt

1 Like
8

I definitely see your point, but what you described above (build is_from hash) is actually the most common scenario, as that’s what happens if you have a webhook, you push code into your repository and the webhook triggers a build. That build will get the commit hash and will checkout that specific commit hash (not the branch).

9

Ok ok, didn’t know that, but indeed, make total sense that it’s the hash of the trigger that is picked instead of the branch. My vision was focus on the UI screen where I input the hash manually.

Then maybe it should be only when the commit hash has been specified by the user via the UI or other mean. But that seems like an edge case.

Seems there won’t be an easy one :slight_smile:

10

Indeed, that’s my dilemma too - I’ll discuss this with the team to see if anyone have an elegant solution for this :wink:

11

Actually I might have an idea. I believe this could work reasonably well, especially if you consider this is quite an edge case :

We could do an “extended” retry if the first checkout fails. The git clone step already has retry logic for every network operation, and we have something similar in the CocoaPods step too. In short, if the checkout fails don’t just do a simple retry, but instead do a git fetch tags and then a checkout retry.

This will result in a retry every time but only in this edge case, which I think is reasonable.

WDYT @mvachon ?

12

Indeed, seems like a really good compromise! Nice idea.

1 Like
13

I’ll send a PR with this tomorrow / ASAP! :wink:

14

Hey @mvachon!

First of all, thank you for reporting! :+1:t2: The PR became ready please check it out, it works like a charm for me now. :tada:(+@viktorbenei)

https://github.com/bitrise-io/steps-git-clone/pull/29

Git clone repository
Git clone repository
=> git "init"
=> git "remote" "add" "origin" "git@github.com:trapacska/testtagfetch.git"
=> git "fetch"
=> git "checkout" "fa3dca5d91e0a54a262327a9575205b29ebc28e6"
Checkout failed, error: fatal: reference is not a tree: fa3dca5d91e0a54a262327a9575205b29ebc28e6

Retry with fetching tags...
=> git "fetch" "--tags"
=> git "checkout" "fa3dca5d91e0a54a262327a9575205b29ebc28e6"
=> git "submodule" "update" "--init" "--recursive"

:ok_hand:

Edit: added error log for checkout. So the user knows why was the retry required.

1 Like
15

Yep, good with me. Great job guys!

2 Likes
16

Released! :rocket::tada:

1 Like