summaryrefslogtreecommitdiffstats
path: root/jenkins_jobs/local_yaml.py
Commit message (Collapse)AuthorAgeFilesLines
* Allow use of jinja templates in defaultsJames Harris2019-11-051-7/+21
| | | | | | | | | | | | Jinja templates are not deep-copyable so they cannot be used in "defautls" sections or to pick defualts for job groups or projects. This works around the issue by waiting until we render the template to construct the template itself. Story: 2006431 Task: 36337 Change-Id: Ief31fdaac06bb14d0aaba71c8c0e658a7f861671
* Auto-generated output from python-blackThanh Ha2019-09-091-63/+85
| | | | | | | | | Please review the following patch containing the code changes in the repo. This patch is a transition patch and is the auto-generated output of the python-black tool. Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032 Signed-off-by: Thanh Ha <zxiiro@gmail.com>
* Revert "Add support for rendering jinja template as yaml"Daniel Watkins2019-07-291-19/+0
| | | | | | This reverts commit a9e12ed4a9f5866da0af5fdc68f60b09ca8efa4b. Change-Id: I2bbb2cc167a4c2cd95ac6b376ebe83bcf7a263ad
* Add support for rendering jinja template as yamlPhilip Roche2019-07-121-0/+19
| | | | | | | | | | | | | | | | | | | | | | This also adds support for using custom loaders in job definitions not just job-templates and builders. I have also added unit tests for include-jinja2-as-yaml custom loader. Example usage: - axis: type: user-defined name: VERSIONS values: !include-jinja2-as-yaml: versions.j2.yaml.inc Where versions.j2.yaml.inc is {% for possible_versions in configuration["possible_versions"] %} - {{ possible_versions["versions"] }} {% endfor -%} Change-Id: I15a431d5a86b29d704efda8576965ade2b7dcd2f
* Fix loader overwritingIvan Remizov2019-03-121-0/+5
| | | | | | | | | | | In cases when there are many j2 templates and some of them use include tags, correct jinja loader is overriden with one of loaders related to other templates. This patch fixes it with workaround. Which is overwrite incorrect loader with initial. Change-Id: Iec21c24fe6e4bfedf281d7215e520298381f07a7
* upgrade hacking moduleSorin Sbarnea2018-06-191-1/+1
| | | | | | | | | - upgrades hacking to current version - sorts new linting issues - sorts bug with sys.reload on py3 Change-Id: I4a18abc93116667a2733e8aec619ac59ea73d630 Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
* Add retain_anchors config option.Vicky Chijwani2018-05-311-2/+3
| | | | | | | | | If set to True, YAML anchors can be referenced across files, allowing jobs to be composed from bits of YAML defined in separate files. False by default. Story: 2000338 Task: 2547 Change-Id: I034ce3bce0030093cb8d4266dabbdb06d96306d6
* Add helper tag to join arbitrary lists into stringsDarragh Bailey2018-04-121-0/+65
| | | | | | | | | | | | | Adds a yaml application tag '!join:' to support defining data as a list while having the resulting contents automatically joined together with the specified delimiter and passed as a string into the resulting object returned by yaml for JJB to process. This allows users to store long lists of data that is combined together for use with Jenkins plugins that expect a delimited string of arguments instead of needing each module in JJB to provide individual support. Change-Id: I745181ade3926d5c29708963189ae499a0378ece
* Add support for {% include %} in Jinja2 templatesDaniel Watkins2018-02-231-3/+6
| | | | | | | | | | This allows the use of the {% include %} tag[0] within Jinja2 templates, with the same search path used for looking up other templates (i.e. the one which LocalLoader is instantiated with). [0] http://jinja.pocoo.org/docs/2.9/templates/#include Change-Id: I0403c385ee317e1460a80a9bf717aa4712f37e2f
* Add !j2 directive to allow inline Jinja2 templatesDaniel Watkins2018-02-211-0/+17
| | | | | | | This enables the use of Jinja2 templating within strings in the template YAML files. Change-Id: I2e912d2f874c5f2428e1aed4af94897aee8d4a72
* Merge "Revert "Move macro expansion into YamlParser.""Zuul2018-01-051-31/+3
|\
| * Revert "Move macro expansion into YamlParser."Thanh Ha2018-01-011-31/+3
| | | | | | | | | | | | This reverts commit e645ac2acff34c7e58af2b4715b5bcdd2dffa31a. Change-Id: I56e8c8282669cbc9f963056f64e9caef8104b6bb
* | Change file inclusion level from info to debugSorin Sbarnea2018-01-011-1/+1
|/ | | | | | | | | | | File inclusion is debug info, info level would spam console and make user unable to get progress info without it. File inclusion can easily mean >100 lines for users relying on templating. Change-Id: I013d86d8563fd75f5255578c0c735c4f2d5c074b
* Move macro expansion into YamlParser.Wayne Warren2017-08-241-3/+31
| | | | | | | | | | | | | | | | | | | Introduce the registry.MacroRegistry class to handle: * registration of macro types via setuptools' entrypoints * registration of individual macros for lookup by component list type * expansion of macros references during YAML "parsing" As a consequence there is a reduction in performance due to moving the expansion of macros from inline with XML generation, to requiring multiple passes over macro component lists. This decrease in efficiency results in approx ~30-50% increase in unit test time. Since this will allow for jobs to be expanded from templates/macros in parallel with future changes, it is a reasonable short term trade-off as the most computationally expensive task is updating the definitions on the remote master Change-Id: I292c6b1f8472370282205426cd8ceb847eb969bd
* Add !include-jinja2 for rendering templates with Jinja2Daniel Watkins2017-08-091-5/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | This template included using !include-jinja2: """ {{ my_var }} """ is rendered the same as an existing template that looks like this: """ {my_var} """ This also allows the use of Jinja2's richer syntax: """ {% for test_environment in configuration.get("envs", ["py35"]) %} tox -e {{ test_environment }} {% endfor %} """ Story: 2001135 Change-Id: Ia3ee21822d6e9237f5ea46796bc8810ecac61e2c
* Fix !include parsing with variableAleksandr Charykov2017-04-201-2/+4
| | | | | | | Do not overwrite template string when parsing yaml. Story: 2000996 Change-Id: Id1b12be69c3994fa4108a9841c0128da9dc883ef
* Add note regarding a bug with !include-raw-escapeThanh Ha2017-03-161-0/+9
| | | | | | | | This spawned from discussions on the mailing list: http://lists.openstack.org/pipermail/openstack-infra/2017-March/005233.html Change-Id: I7916ebec86350fa760bb02b6bde38e27c71dc359 Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
* Removed some deprecation warnings and enabled py35 on tox.Sorin Sbarnea2017-03-121-4/+4
| | | | | Change-Id: Icc5b28c4636b542a10502b7b1d2d2dc2028cc166 Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
* Merge "Improve logger output for expanding templates"Jenkins2016-11-101-1/+28
|\
| * Improve logger output for expanding templatesDarragh Bailey2016-09-141-1/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Output the variable inputs used that trigger an error when expanding template names to the error logger channel in a sensible format. Ensures that when indented variable inputs for templates result in exceptions when expanding a template name, that the project, template name and variables that failed to be iterated over are outputted in a log error message along with the original set of inputs from the project definition to make it easier for end users to find where the error has been made in a JJB definition. Add code to allow dumping of variables stored in OrderedDict transparently to match the input format used in JJB definitions and hide the implementation detail of using OrderedDict to be within the localyaml library. Change-Id: I660bb0ca3b109e1a861948d6a867f185047b90ae
* | Merge "Support lazy resolving of include yaml tags"Jenkins2016-11-021-7/+92
|\ \ | |/ |/|
| * Support lazy resolving of include yaml tagsDarragh Bailey2016-08-151-7/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow filenames referenced by the application specific yaml tags to contain template job variables, use a lazy loading object that provides a format method that can be called by the deep_format function. Instead of processing the file, when a KeyError occurs on attempting to call format on the filename after the yaml tag, create a LazyLoader instance to wrap the data and provide a format method that can be called at a later stage. In order to call the correct method on the original Loader class, LazyLoader needs to be given the custom tag class, a reference to the loader and the node object. Using the tag class it can call the from_yaml() method with the loader and node object to return the file contents. Since the result from the LazyLoader instance is triggered by calling the format method, there is no need to escape the brackets used by pythons format method since the output will not be passed through it. In order to ensure this behaviour, nodes passed to the method handling the '!include-raw-escape:' tag class, which need to use the LazyLoader approach will convert to the '!include-raw:' tag class to the LazyLoader initialization instead. Due to a bug in sphinx with use of 'note' admonitions and manpage generation, need to update to a version >= 1.2.1. Change-Id: I187eb83ba54740c2c1b627bc99c2d9769687fbc7 Story: 2000522
* | Cleanup various deprecation warnings.Wayne Warren2016-08-201-2/+2
|/ | | | Change-Id: Iec16a5965d62bebb50d3e7307ab93c59304a9ab6
* Remove ordereddict support from py26Thomas Bechtold2016-02-091-4/+1
| | | | | | | | | | python 2.6 is no longer supported and adding a marker to requirements.txt for installing ordereddict only in py2.6 envs was not an option because markers are not supported in older pip versions. So remove python 2.6 support completly. Change-Id: Iebdd999b469c9a9681b0d7e9f50cc488a8820953
* Reorder imports to match hacking guidelinesDarragh Bailey2015-12-231-1/+3
| | | | | | | Ensure that the imports follow the standard OpenStack hacking guidelines. Change-Id: Iaa4326aef118ddfd807dd006934f1d9ca80a1cfa
* Separate out custom yaml tags into separate classesDarragh Bailey2015-09-181-73/+114
| | | | | | | | | | | | Implement custom tags for including files by subclassing YAMLObject to allow PyYAML to easily convert from the Yaml nodes. Change to support both lists and strings under the same tag name and deprecate the old tag naming. Rename tests using the deprecated tags to ensure they are not used for examples. Remove old tests that simply duplicate others. Change-Id: I1a8d3376ea6e4918a2adb05fb856b939a3124d74
* Replace open() with io.open() and force 'utf-8'Darragh Bailey2015-07-251-7/+7
| | | | | | | Use io.open() to allow reading and writing of files in 'utf-8' format irrespective of the terminal encoding selected. Change-Id: Ie952617a34c0719efc59a7729d698beafaa477b0
* Make include_path deterministicMax Rydahl Andersen2015-05-311-5/+5
| | | | | | | | | | | | | In code the `include_path` and defaults like `.` added to a set as opposed a list. Resulting in that the order is machine/hashcode specific and that user cannot make i.e. `.` to have higher precedence than the fallbacks. Without doing this you have to use absolute or unique paths which might not always be possible in your .ini file. Change-Id: I9721b168a67e799dfc19d7df951b99c0749b52e0
* Merge "Allow inclusion of multiple files"Jenkins2015-05-291-0/+29
|\
| * Allow inclusion of multiple filesJan Hruban2015-04-201-0/+29
| | | | | | | | | | | | | | | | Add variants of the include-raw* tags, which accept a list of files. Those files are concatenated and included as a string data into the calling yaml construct. Change-Id: I6af87a298268acc8a73d7a2b50f9f99733d8723a
* | Allow use of aliases defined previously inside included filesVictor Seva2015-05-141-1/+26
|/ | | | | | | | | | | | | | | | | | | | | | | | | | Anchors and aliases were expanded internally within JJB's yaml loading calls so they were limited to individual documents. Now, included files will have access to aliases of anchors already defined at previously processed files. Example: - default: name: default-timeout-wrapper timeout: &timeout fail: true elastic-percentage: 150 elastic-default-timeout: 90 type: elastic - wrapper: !include include002_1.yaml.inc Previously was not possible to use '*timeout' alias inside include002_1.yaml.inc file Closes-Story: 2000173 Change-Id: Ic031ddbb0310bd11748183fbde9502735c3b7169
* Make JJB python 3 compatibleMarc Abramowitz2014-10-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Convert to use idioms that work for both python 3 and python 2.6+ and ensure that a suitable version of dependencies is included for python 3 compatibility. Update python-jenkins to 0.3.4 as the earliest version that supports python 3 without any known regressions. Add an extra parser check for missing 'command' due to changes in how argparse works under python 3. To access the first element of a dict in both python 2 and 3, 'next(iter(dict.items()))' is used as the standard idiom to replace 'dict.items()[0]' as 'items()' returns an iterator in python 3 which cannot be indexed. Using 'next(iter(..))' allows for both lists and iterators to be passed in without unnecessary conversion of iterators to lists which would be true of 'list(dict.items())[0]'. Original change which was reverted due to breaking use of job-groups is If4b35e2ceee8239379700e22eb79a3eaa04d6f0f. This replaces the previous conversion of 'dict.items()[0]' to 'dict.popitem()', which would result in removing a job-group when first called, thus defeating the benefit of being able to reference the group mulitple times. This usage has been replaced with 'next(iter(dict.items()))' as a non-modifying alternative that still avoids creating unnecessary copies of data while working for all supported versions of python. Change-Id: I37e3b67c043dadddb54e16ee584bde3f79e6a770
* Merge "doc update: add references to yaml local include files"Jenkins2014-09-241-0/+24
|\
| * doc update: add references to yaml local include filesKhai Do2014-08-031-0/+24
| | | | | | | | Change-Id: I3e30d4f988ae8d9506d6975d1f8aa913eab1abea
* | Revert "Some tweaks to get closer to Python 3 compat"Clark Boylan2014-09-021-1/+1
| | | | | | | | | | | | | | | | | | | | This reverts commit 1d7647fa857fa718af814f3038d538d758c35201. This change altered the xml output (by forcing it to fail) which a backward compatible change should not do. Revert it in order to get this compat change in without breaking that output and test. Change-Id: I20f66fb1bd9c70a0debbdd5eebacf6ec5d0f5df9
* | Some tweaks to get closer to Python 3 compatMarc Abramowitz2014-09-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Convert to use idioms that work for both python 3 and python 2.6+ and ensure that a suitable version of dependencies is included for python 3 compatibility. Update python-jenkins to 0.3.3 as the earliest version that supports python 3 without any known regressions. Add an extra parser check for missing 'command' due to changes in how argparse works under python 3. Where contents should be retained, to access the first element of a dict in both python 2 and 3, 'next(iter(dict.items()))' is used as the standard idiom to replace 'dict.items()[0]' as 'items()' returns an iterator in python 3 which cannot be indexed. Using 'next(iter(..))' allows for both lists and iterators to be passed in without unnecessary conversion of iterators to lists which would be true of 'list(dict.items())[0]'. Alternatively, where further access to the data is not required, 'dict.popitem()' is used. Change-Id: If4b35e2ceee8239379700e22eb79a3eaa04d6f0f
* | Ensure dict orders are deterministicDarragh Bailey2014-09-011-1/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Python 3 enables hash randomization by default, additionally tox 1.7 turns on the same randomization for earlier versions of python by default. Need to ensure that order of iteration over the yaml data and resulting XML has deterministic order for testing. Adapts https://gist.github.com/enaeseth/844388 which ensures data read by yaml will have its order retained in a predictable manner across multiple python versions. Additionally it seems more sensible to ensure that the order of generated XML snippets corresponding to the input yaml files are consistently in the same order as the entries in the source files. Closes-Bug: #1333349 Change-Id: I6bf6d298a2609cc6ddbbc6b02b7f1a04413a5c89
* | Handle utf8 in JJBClark Boylan2014-08-291-1/+2
|/ | | | | | | | | | | | | | | | | | JJB didn't actually handle unicode data very well for a couple reasons. First the local yaml loader was loading files into yaml as strings instead of unicode which we should just go ahead and do because yaml's built int loader loads utf-8 by default (and we don't override the default). Second we need to do parameter substitution on unicode and regular strings so change the substitution typecheck to use basestring instead of str. Finally we need to use UTF-8 as the encoding when emitting XML so do that. Add tests to actually test this in the yamlparser tests. The addition of these new tests comes with a little bit of cleanup in the test classes to make sure we load unicode files as utf8 more consistently. Change-Id: I2169e19aae2cdc7ddbd1e7217ef7584c786a039a Fixes-bug: 1361090
* Use yaml local tags to support including filesDarragh Bailey2014-06-201-0/+159
Add support for local tags which are application specific to allow including of other yaml files or code from scripts. Allows for code to be maintained and tested as seperate files, as well as reduces duplication of yaml code that cannot be macro'ed or easily templated by including it from a common file. Adds support for the following tags: 'include' - load file as yaml code 'include-raw' - load file as data (for scripts) 'include-raw-escaped' - load file as data with escaping braces '{}' as default for use with job-templates Use configuration file options to provide a search path for the files. - Test behaviour of yaml tags independent of any XML generation by comparing json result of yaml parsing to verify that certain tags do/don't recall the yaml.load() method. - Add examples for the include tags via addition tests for YamlParser class Inspired by http://stackoverflow.com/questions/528281/how-can-i-include-an-yaml-file-inside-another Change-Id: Ib90a07043112d4739d6529ceddbc9817668bcec0
generated by cgit (git 2.34.1) at 2025-09-03 12:38:58 +0000