Thursday, 22 August 2013

Join and 'collectively' manipulate a de-normalized one-to-many split conditionally after the fact

Join and 'collectively' manipulate a de-normalized one-to-many split
conditionally after the fact

Tricky thing to title and explain accurately - let me know if anything's
unclear.
I have a de-normalized dataflow, consider:
Input one:
ID
Value
FKID
Input two:
FKID
DiffValue
CheckValue
Input one can have more than one input two row, relating on FKID obviously.
Now, after the joining I might have something like:
[ID] - [Value] - [DiffValue] - [CheckValue]
1 - A - D1 - C1
1 - A - D2 - C1
1 - A - D3 - C2
I then go through a conditional split, based off the CheckValue -
searching for value: C2. If C2, change Value to DiffValue. Leaving me
with:
Split one:
1 - A - D1 - C1
1 - A - D2 - C1
Split two:
1 - D3 - D3 - C2
Now I'd like to make sure that all of my rows have that D3 value. The
thing here is, that it's possible for the split never to become true, in
which case I need it to just go on, keeping the original A value, but if
just ONE of the rows fulfill the checkvalue condition, all of the rows
need the DiffValue.
A script transformation component won't work, as it's based on buffer, and
I can't cache all the rows, perform code on the entire collection of rows
at once, before sending them to output. I can't even do a simple save ID
to temp SQL and do a lookup based on ID afterwards, because after the
conditional split it's two parallel asynchronous paths, so I can't halt
the first split until after the temp SQL table has been filled.
I COULD do a script destination, caching all the rows then, perform my
manipulation after every row has been cached, and then make a custom
output with the relevant columns, however, there are many more columns
(this is a simplified example), and the maintenance in having to deal with
that custom output is overwhelming.
How would I go about this?

No comments:

Post a Comment