Remove the whole distgit cache before triggering
AbandonedPublic
Actions

Authored by mkrizek on Apr 25 2016, 9:56 AM.

Details

Reviewers

Summary

Remove the whole distgit cache instead of removing just package-specific
cache and prevent the trigger from eating the disk space.

Test Plan

Tested locally, worked for me. Unittests amended.

Diff Detail

Repository

rTRGR taskotron-trigger

Branch

feature/rm-distgit-cache

Lint

Lint OK

Unit

Unit Tests OK

Build Status

Buildable 552
Build 552: arc lint + arc unit

mkrizek retitled this revision from to Remove the whole distgit cache before triggering.Apr 25 2016, 9:56 AM

mkrizek updated this object.

mkrizek edited the test plan for this revision. (Show Details)

mkrizek added a reviewer: tflink.

Isn't there a chance of multiple tasks interfering with each other if each task trigger is recursively removing all the dist-git checkouts?

In D825#15650, @tflink wrote:

Isn't there a chance of multiple tasks interfering with each other if each task trigger is recursively removing all the dist-git checkouts?

Er, fedmsg consumers run in multiple threads, don't they?

mkrizek abandoned this revision.Apr 25 2016, 5:10 PM

Revision Contents

			Path	Coverage (All)	Coverage (Touched)	Packages
M			jobtriggers/utils.py (16 lines)	83%	37%
M			testing/test_utils.py (1 line)	100%	100%

Commit	Tree	Parents	Author	Summary	Date
8da52807a0b4	f61201f3919f	19d9f6a7f6c7	Martin Krizek	Remove the whole distgit cache before triggering (Show More…)	Apr 25 2016, 9:51 AM

Diff 2096

View Options

jobtriggers/utils.py

1	import os	1		import os
2	import re	2		import re
3	import subprocess	3		import subprocess
4	import csv	4		import csv
5	import datetime	5		import datetime
6	import shutil	6		import shutil
		7		import glob
7		8
8	import yaml	9		import yaml
9		10
10	from jobtriggers import config	11		from jobtriggers import config
11	from jobtriggers import exceptions as exc	12		from jobtriggers import exceptions as exc
12		13
13		14
14	def log_job(logname, item, item_type, task, arch):	15		def log_job(logname, item, item_type, task, arch):
Show All 29 Lines		41	def get_branch(release):
44		45
45	return matches[-1].replace('c', '')	46		return matches[-1].replace('c', '')
46		47
47		48
48	def get_repo(pkgname):	49		def get_repo(pkgname):
49	return '/'.join([config.distgit_repo_url, config.distgit_namespace, '%s.git' % pkgname])	50		return '/'.join([config.distgit_repo_url, config.distgit_namespace, '%s.git' % pkgname])
50		51
51		52
		53		def rm_dir_contents(dirname):
		54		contents = glob.glob(os.path.join(dirname, '')) + glob.glob(os.path.join(dirname, '.'))
		55
		56		for item in contents:
		57		if os.path.isdir(item):
		58		shutil.rmtree(item)
		59		else:
		60		os.unlink(item)
		61
		62
52	def get_distgit_tasks(pkgname, item_type, repo, branch='master'):	63		def get_distgit_tasks(pkgname, item_type, repo, branch='master'):
53	tasks_dir = os.path.join(config.distgit_cache_dir, pkgname)	64		tasks_dir = os.path.join(config.distgit_cache_dir, pkgname)
54		65
55	# for now, just delete the repo and make a new shallow copy	66		# for now, just delete the whole cache (so it doesn't eat the disk) and make a new shallow copy
56	if os.path.exists(tasks_dir):	67		rm_dir_contents(config.distgit_cache_dir)
57	shutil.rmtree(tasks_dir)
58		68
59	cmd = ['git', 'clone', '--branch', branch, '--depth', '1', repo, tasks_dir]	69		cmd = ['git', 'clone', '--branch', branch, '--depth', '1', repo, tasks_dir]
60		70
61	try:	71		try:
62	subprocess.check_output(cmd, stderr=subprocess.STDOUT)	72		subprocess.check_output(cmd, stderr=subprocess.STDOUT)
63	except subprocess.CalledProcessError as e:	73		except subprocess.CalledProcessError as e:
64	if 'Could not find remote branch' in e.output and branch != 'master':	74		if 'Could not find remote branch' in e.output and branch != 'master':
65	return get_distgit_tasks(pkgname, item_type, repo, branch='master')	75		return get_distgit_tasks(pkgname, item_type, repo, branch='master')
Show All 12 Lines