Bug 1189517

Summary:	[RFE][sahara]: Allow placeholders in datasource URLs
Product:	Red Hat OpenStack	Reporter:	RHOS Integration <rhos-integ>
Component:	openstack-sahara	Assignee:	Elise Gafford <egafford>
Status:	CLOSED ERRATA	QA Contact:	Luigi Toscano <ltoscano>
Severity:	high	Docs Contact:
Priority:	high
Version:	unspecified	CC:	ddomingo, jschluet, kbasil, markmc, matt, mimccune, yeylon
Target Milestone:	beta	Keywords:	FutureFeature
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
URL:	https://blueprints.launchpad.net/sahara/+spec/edp-datasource-placeholders
Whiteboard:	upstream_milestone_liberty-1 upstream_definition_approved upstream_status_implemented
Fixed In Version:	openstack-sahara-3.0.0-3.el7ost	Doc Type:	Enhancement
Doc Text:	When creating a job template intended for re-use, you can now register a variable for datasource URLs with OpenStack Data Processing (sahara). Doing so allows you to easily change input and output paths per run, rather than an actual URL (which would require revising the template, or manually revising the URL per run between jobs). This makes it easier to reuse job templates when data source jobs are mutable between runs, as is true for most real-world cases.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-04-07 21:00:33 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description RHOS Integration 2015-02-05 13:50:20 UTC

Cloned from launchpad blueprint https://blueprints.launchpad.net/sahara/+spec/edp-datasource-placeholders.

Description:

Common use case: user wants to run EDP job two times. Now the only way to do that with the same data sources is to erase result of the first run before running job the second time. Allowing to have random part in URL will allow to use output with random suffix.

Specification URL (additional information):

None

Comment 6 Elise Gafford 2015-11-19 15:45:04 UTC

Feature is implemented in upstream L.

Comment 8 Luigi Toscano 2016-04-01 17:01:15 UTC

The placeholders in datasources (%RANDSTR(<len>)%, %JOB_EXEC_ID%) are correctly replaced according the spec for both HDFS and Swift data sources. 

There is only a minor issue: when <len> is really big, for example 2^32+1 but also smaller values, something goes wrong and the job stays in PREPARE state. But this is really a corner case and it comes from a deliberate choice of the user that defines the data source, who can easily avoid the problem.
Therefore I don't think this is a blocker for the general availability of the feature.
The issue has been captured in the following upstream bug:
https://bugs.launchpad.net/sahara/+bug/1560890

Verified on an up-to-date RHEL 7.2, with
openstack-sahara-api-3.0.1-4.el7ost.noarch
openstack-sahara-common-3.0.1-4.el7ost.noarch
openstack-sahara-engine-3.0.1-4.el7ost.noarch
python-sahara-3.0.1-4.el7ost.noarch
python-sahara-tests-3.0.1-4.el7ost.noarch

Comment 10 errata-xmlrpc 2016-04-07 21:00:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html