1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
|
Description:
oe-go-mod-autogen.py is a helper script for go mod recipes. It is based
on intiial efforts to use only the git fetcher versus golangs module
infrastructure.
Example:
a) generate
cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \
--repo https://github.com/docker/compose --rev v2.20.3
output: src_uri.inc, relocation.inc, modules.txt
b) Copy the three generated files to replace the original ones.
c) update SRCREV
Edit the recipe to update PV and the main SRCREV, and
docker-compose is upgraded.
See --help for more explanations and examples.
Below are some technical details.
* get module's repo from module name
This script checks the following two URLs to determine the module's repo.
1. https://<module_name_tweaked>?=go-get=1
2. https://pkg.go.dev/<module_name_tweaked>
The module_name_tweaked is derived from module_name, with the last components
removed one by one. Let me use two examples to explain this.
For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as
module_name_tweaked for searching. And we can correctly get the repo URL, so
the search stops.
For module_name github.com/k3s-io/etcd/api/v3, the following ones are used
as module_name_tweaked:
github.com/k3s-io/etcd/api/v3
github.com/k3s-io/etcd/api
github.com/k3s-io/etcd
And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search
stops.
* determine the srcdir:destdir mapping in 'vendor' creation
To correctly form the 'vendor' directory, the mapping is critical.
This script makes use of tag matching and path matching to determine
the subpath in the repo for the module.
* avoid subpath being overriden by parent path
We need to avoid subpath being overriden by parent path. This is needed
for both SRC_URI ordering in src_uri.inc and the sites mapping ordering
in relocation.inc. This script simply uses the length as the ordering key,
simply for the reason that if a path is a subpath of another path, it must
be longer.
* the .git suffix is removed to sync with each other
Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script
remove the '.git' suffix from repo URL so that the repo URLs are in sync
with each.
* basic directory hierarchy and caching mechanism
<cwd>/repos: hold the repos downloaded and checked
<cwd>/wget-contents: hold the contents to determine the module's repo
<cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache
This is to avoid unnecessary URL fetching and repo cloning.
* the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script
The script must get the correct repo_url, fullsrc_rev and subpath for
each required module in go.mod to correctly generate the src_uri.inc and
relocation.inc files. If this process fails for any required module, this
script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE
to True in this script. The purpose is to encourage people to report
problems to meta-virt so that we can improve this script according to
these feedbacks. But this variable can set to False, then the script
only records the failed modules in self.modules_unhandled with reasons
added, people can modify the generated src_uri.inc and relocation.inc
to manually handle these unhandled modules if they are urgent to
add/upgrade some go mod recipes.
Debug tips:
If the do_compile task exits with a go mod / fetch error, this is due
to some sort of compilation error during the go build. The log file
can be huge. To figure out what broke, try the following:
% cd ${S}/src/import
% grep -v ^HASH ../../../temp/log.do_compile
This will likely highlight what broke. It is often that either the wrong
revision was used (and there will be go build errors) or that a fetched
repository used the wrong source or destination. Seeing what module
couldn't be found, will allow you to locate the SRC_URI fetch in question,
how it was relocated. Either the source or destination may be wrong.
As an example, k3s was failing with the following error:
go: finding module for package sigs.k8s.io/kustomize/kustomize/v5/commands/build
vendor/k8s.io/kubectl/pkg/cmd/kustomize/kustomize.go:25:2: cannot query module due to -mod=vendor
WARNING: exit code 1 from a shell command.
Which leads to looking at the SRC_URI and vendor fetch, which was not
using the proper source for kustomize.
sigs.k8s.io/kustomize/kustomize/v5:sigs.k8s.io/kustomize/kustomize:force \
had to be changed to:
sigs.k8s.io/kustomize/kustomize/v5:sigs.k8s.io/kustomize/kustomize/v5/kustomize:force \
To get the commands/build subirectory from the source into the proper
vendor location.
|
