| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
* Enables fail on warnings and enable nitpicky
* Consolidate string -> str for args
Change-Id: Id5c29d48813fd6f772816dc74c7c4e408097113d
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Each component (builder, publishers...) is loaded everytime dispatch()
is called. Its main purpose is to lookup the type of the component.
However pkg_resources.EntryPoint.load() is a costly operation and it was
done every single time dispatch() is called.
Against Wikimedia configurations files [0] it is extremely noticeable.
The entrypoint is loaded solely to figure out its type. I do not see any
good reason for it to change, thus introduce a new cache
_component_type_cache.
That then let us prevent invoking load() every single time.
Run time to generate Wikimedia configuration files went from 17 seconds
to a mere 4 seconds!
[0] https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/
Change-Id: I49bc5dc91e3ba69b7f15c5f5ded8a42fb61a60c2
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When profiling a JJB run against Wikimedia configuration files [0], I
noticed a lot of calls to package_resource trying to figure out the
version.
Years ago, I already added coded to cache the entrypoints, unfortunately
only the entry point definition got cached not the underlying function.
Thus each entry point ends up invoking a load() which dramatically slow
down JJB.
A low hanging fruit is to invoke load() and cache the returned function.
Doing so, the run time goes down from 28 seconds down to 17 seconds.
[0] https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/
Change-Id: I336b556f28328d8acbdd71c43c8775d55c3720d8
|
| |
|
|
|
|
|
|
|
| |
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
|
| |
|
|
|
|
|
|
|
|
| |
As Jinja2 provides functionality other than interpolation, it's entirely
feasible to use it outside of a context where there are template
variables. This change ensures that we always deep format Jinja2
templates, even when we wouldn't normally interpolate a regular JJB
template.
Change-Id: Ia3f86dd2073d48594eaf791fa2277db7e5ebf0ab
|
| |
|
|
|
|
| |
This reverts commit e645ac2acff34c7e58af2b4715b5bcdd2dffa31a.
Change-Id: I56e8c8282669cbc9f963056f64e9caef8104b6bb
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce the registry.MacroRegistry class to handle:
* registration of macro types via setuptools' entrypoints
* registration of individual macros for lookup by component list type
* expansion of macros references during YAML "parsing"
As a consequence there is a reduction in performance due to moving the
expansion of macros from inline with XML generation, to requiring
multiple passes over macro component lists.
This decrease in efficiency results in approx ~30-50% increase in unit
test time. Since this will allow for jobs to be expanded from
templates/macros in parallel with future changes, it is a reasonable
short term trade-off as the most computationally expensive task is
updating the definitions on the remote master
Change-Id: I292c6b1f8472370282205426cd8ceb847eb969bd
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
when building snapshots, maven will sometimes add some information
*after* the -SNAPSHOT like this:
1.4.6-SNAPSHOT (private-0986edd9-egolov)
the old registry code would transform this to
1.4.6.preview (private-0986edd9-egolov)
which is not a valid version string for pkg_resources.
strip everything after the -SNAPSHOT tag and thus produce a clean
version string that is comparable properly
Change-Id: I765c991ac7632b3c76f548112d2a5323a8ce4489
Signed-off-by: Evgeni Golov <evgeni@golov.de>
|
| |/
|
|
|
|
|
|
| |
Problems occurring deep in the code frequently need to be caught and
additional information included in the error output to help debug
issues.
Change-Id: I5aee523a3cf9e1cb7573fa6fc5a06dc3888be1ef
|
| |
|
|
| |
Change-Id: Iaea087c6c3a024922034423a8d56571184f33b7c
|
| |
|
|
| |
Change-Id: Iec16a5965d62bebb50d3e7307ab93c59304a9ab6
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.
This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.
The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.
Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
|
| |
|
|
|
|
|
|
|
|
|
| |
Remove reference to ConfigParser object in ModuleRegistry. Instead:
* make use of JJBConfig.get_module_config to grab settings for Hipchat
Notifier Plugin
* make use of JJBConfig.yamlparser['allow_empty_variables'] rather
than repeating ConfigParser logic moved out of the YamlParser into
JJBConfig in an earlier commit.
Change-Id: Icb7ef514826005545e48af993335ce120f973b0d
|
| |
|
|
|
|
|
|
|
|
| |
Currently JJB warns over 700 times when checking project-config with:
WARNING:jenkins_jobs.registry:You have a macro ('tox') defined for
'builder' component type that is masking an inbuilt definition
Warn only once for this.
Change-Id: I010765269c499320fc9a84b9493d22949080e9a0
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Prefer user defined macros over inbuilt entry points, as JJB may
accidentally use a name that has already been used by an end-user to
define a custom macro.
Warn users when they have a macro defined that masks an in-built
component to avoid accidentally changing XML generated with new
releases.
Change-Id: I6cae62d7cc40be6c4a5636a74f151fcce4cdd856
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Use the component type entry points to infer modules to scan
corresponding modules and automatically construct entry points for any
functions associated with yaml entries in JJB definitions.
This removes the need for new JJB modules added directly to the project
to also need to be listed in the setup.cfg to be reachable. Simply
adding the needed functionality to the correct file ensures that it is
available.
Change-Id: I2a0fd8f6720ddfd8582b58e47556a87ed1ad553d
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Prior to this patch macros serialized into a yaml document, a simple string
substitution was applied and then they were re-loaded into python. This
created several problems:
- newlines are lost because the yaml loading was not respecting the
newline settings of the original strings (Story: 1387060)
- {obj:} notation was not working for macro parameters (Story: 2000254)
- multiline strings passed into scripts will create a yaml syntax error
when being re-loaded (unticketed)
This patch resolves these problems by replacing the previous interpolation
code with the more robust `deep_format` method. A side effect of this patch
is that there are now fewer disparate code paths for interpolating variables.
Change-Id: Ia06c98286537a50dc52c6e5e000b32ebc9e8ede2
Signed-off-by: Justin Stoller <justin.stoller@gmail.com>
|
|
|
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.
That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.
Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
|
