diff options
Diffstat (limited to 'scripts/README-oe-go-mod-autogen.txt')
| -rw-r--r-- | scripts/README-oe-go-mod-autogen.txt | 120 |
1 files changed, 120 insertions, 0 deletions
diff --git a/scripts/README-oe-go-mod-autogen.txt b/scripts/README-oe-go-mod-autogen.txt new file mode 100644 index 00000000..80a35e52 --- /dev/null +++ b/scripts/README-oe-go-mod-autogen.txt @@ -0,0 +1,120 @@ +Description: + + oe-go-mod-autogen.py is a helper script for go mod recipes. It is based + on intiial efforts to use only the git fetcher versus golangs module + infrastructure. + + Example: + + a) generate + + cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \ + --repo https://github.com/docker/compose --rev v2.20.3 + output: src_uri.inc, relocation.inc, modules.txt + + b) Copy the three generated files to replace the original ones. + + c) update SRCREV + + Edit the recipe to update PV and the main SRCREV, and + docker-compose is upgraded. + + See --help for more explanations and examples. + + Below are some technical details. + + * get module's repo from module name + + This script checks the following two URLs to determine the module's repo. + 1. https://<module_name_tweaked>?=go-get=1 + 2. https://pkg.go.dev/<module_name_tweaked> + + The module_name_tweaked is derived from module_name, with the last components + removed one by one. Let me use two examples to explain this. + + For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as + module_name_tweaked for searching. And we can correctly get the repo URL, so + the search stops. + + For module_name github.com/k3s-io/etcd/api/v3, the following ones are used + as module_name_tweaked: + github.com/k3s-io/etcd/api/v3 + github.com/k3s-io/etcd/api + github.com/k3s-io/etcd + And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search + stops. + + * determine the srcdir:destdir mapping in 'vendor' creation + + To correctly form the 'vendor' directory, the mapping is critical. + This script makes use of tag matching and path matching to determine + the subpath in the repo for the module. + + * avoid subpath being overriden by parent path + + We need to avoid subpath being overriden by parent path. This is needed + for both SRC_URI ordering in src_uri.inc and the sites mapping ordering + in relocation.inc. This script simply uses the length as the ordering key, + simply for the reason that if a path is a subpath of another path, it must + be longer. + + * the .git suffix is removed to sync with each other + + Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script + remove the '.git' suffix from repo URL so that the repo URLs are in sync + with each. + + * basic directory hierarchy and caching mechanism + + <cwd>/repos: hold the repos downloaded and checked + <cwd>/wget-contents: hold the contents to determine the module's repo + <cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache + This is to avoid unnecessary URL fetching and repo cloning. + + * the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script + + The script must get the correct repo_url, fullsrc_rev and subpath for + each required module in go.mod to correctly generate the src_uri.inc and + relocation.inc files. If this process fails for any required module, this + script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE + to True in this script. The purpose is to encourage people to report + problems to meta-virt so that we can improve this script according to + these feedbacks. But this variable can set to False, then the script + only records the failed modules in self.modules_unhandled with reasons + added, people can modify the generated src_uri.inc and relocation.inc + to manually handle these unhandled modules if they are urgent to + add/upgrade some go mod recipes. + + +Debug tips: + +If the do_compile task exits with a go mod / fetch error, this is due +to some sort of compilation error during the go build. The log file +can be huge. To figure out what broke, try the following: + + % cd ${S}/src/import + % grep -v ^HASH ../../../temp/log.do_compile + +This will likely highlight what broke. It is often that either the wrong +revision was used (and there will be go build errors) or that a fetched +repository used the wrong source or destination. Seeing what module +couldn't be found, will allow you to locate the SRC_URI fetch in question, +how it was relocated. Either the source or destination may be wrong. + +As an example, k3s was failing with the following error: + + go: finding module for package sigs.k8s.io/kustomize/kustomize/v5/commands/build + vendor/k8s.io/kubectl/pkg/cmd/kustomize/kustomize.go:25:2: cannot query module due to -mod=vendor + WARNING: exit code 1 from a shell command. + +Which leads to looking at the SRC_URI and vendor fetch, which was not +using the proper source for kustomize. + + sigs.k8s.io/kustomize/kustomize/v5:sigs.k8s.io/kustomize/kustomize:force \ + +had to be changed to: + + sigs.k8s.io/kustomize/kustomize/v5:sigs.k8s.io/kustomize/kustomize/v5/kustomize:force \ + +To get the commands/build subirectory from the source into the proper +vendor location. |