Stage 3 - Pipeline Generation¶
The Stage that produces working SnapLogic pipelines. It is the longest and the most involved, and it builds everything twice on purpose.
Two representations, and why¶
SLIM does not translate your integration straight into SnapLogic. It builds it twice, in order.
First it writes a Functional Representation: a working implementation of what the integration does, in Python, built from the Integration Analysis document. It carries no opinion about SnapLogic at all. Its only job is to state the behavior plainly - what is read, what is transformed, what is written, and what happens when something goes wrong.
Then it builds the SnapLogic pipelines from that representation, together with any design documents you attached to the Project.
The detour separates two questions that get muddled when they are answered at once: what does this integration do, and how should that be built in SnapLogic? Answering the first one on its own means the behavior is written down before any decision about the target platform is layered on top of it. That is what makes the comparison at the end worth reading: the new pipelines are compared against a plain statement of the behavior that a person can check, rather than against the original system’s configuration.
It also gives you two audit points instead of one. There is a record of the move from your source system to the functional representation, and a second record of the move from there into SnapLogic, so a question about why a pipeline behaves as it does has somewhere to be answered. See Traceability.
Design documents reach the second step only. The functional representation is built without seeing them, deliberately, so that what your integration does today is established before what somebody intends it to become is applied to it. See Supporting Documents.
What you get¶
Each integration you generate for gets two folders inside its own folder under /migrations in the Project Tree: /python for the functional representation and /snaplogic for the pipelines.
The /snaplogic folder holds the pipelines themselves, an expression library of the reusable logic they share, the test pipelines described in Stage 4 - Testing and Validation, and three documents - the migration plan, which is what was intended; the comparison, which is what was built set against what it came from, including differences that were deliberate; and the completion report, which is how the testing went and what is still outstanding. Where you attached a design document there is a conformance report as well.
The /python folder holds its own set of the same three documents, describing its own half of the work, alongside the implementation and its tests.
Two things about the pipelines are worth knowing before you open the folder.
Some integrations come out as one pipeline and some as several. Where the work divides naturally, SLIM produces a parent pipeline that calls a set of child pipelines, with the expression library shared between them. It makes that decision itself and records it, with its reasoning, in the migration plan.
Each pipeline is present twice. There is the file SnapLogic imports, ending in .slp, and the text form SLIM built it from. Platform Deploy knows which is which, so you do not need to.
Understanding What Was Generated goes through the documents one at a time and says who should read each.
Running it¶
Read the analysis documents first. The dropdown at the top of the document panel on this page lists every integration that has been analyzed, and this is the Stage Page where those documents are edited, because this is the Stage that consumes them. Correcting one here costs almost nothing; see Verifying the Output.
Then set the integration list at the foot of the page and press Generate Pipelines. The list behaves exactly as it does on the Stage before this one, including the caution about Save Selection - see Stage 2 - Integration Analysis.
Warning
SLIM does not check that an integration has been analyzed before generating pipelines for it. The only thing this Stage requires is that Stage 1 has run and the integration list exists. Generate for an integration that has no analysis document and SLIM does not stop: it reads the exported source itself, quickly, and builds from that instead. You get pipelines either way, and nothing on the page tells you which of the two happened. The thirteen-section analysis is the whole reason the output is worth anything, so run Stage 2 first and read what it produced.
A tick mark against a row means SLIM has already produced something for that integration. On this page that is usually the analysis document rather than pipelines, so do not read the mark as evidence that pipelines exist. The Project Tree is where you confirm that.
While it runs, the Activity Log streams what SLIM is doing and a Stop button is available. The Job Status page lists each integration with its own state - Pending, Generating Pipelines, Pipeline Generation Failed, Pipeline Incomplete or Completed - though it does not refresh itself; see Monitoring Progress. The Project Tree is the most useful view of all, because it fills up as the work lands: /python appears and populates first, then /snaplogic, so you can see how far any single integration has got.
The two banners are the familiar pair. Pipeline Generation ended without finishing is usually worth retrying as it stands. Pipeline Generation failed carries the error underneath it; see Stopping and Restarting. When it finishes, a panel appears at the top of the page: Pipeline Generation complete, with Go to Platform Deploy.
Order and parallelism¶
Selecting eight integrations starts eight separate pieces of work, each of which knows nothing about the others. Within any one of them the two representations are strictly sequential: the functional representation has to be finished before the SnapLogic build starts, because the SnapLogic build reads it.
That shape has three consequences worth planning around.
One integration failing does not stop the rest. The others carry on and finish, and you retry the one that failed on its own.
Several run at once, not all of them. A large selection takes proportionally longer rather than arriving all at once, and nothing is lost while a piece of work waits its turn.
Duplicate integrations are built twice. Two integrations that do substantially the same job are worked on in isolation, so SLIM has no way to notice they are duplicates. You get two near-identical sets of pipelines. Within one integration the shared logic is consolidated into the expression library; across integrations nothing is consolidated, and no instruction can change that. Deciding what to merge or retire is yours, and Stage 2 is the cheapest place to decide it - see Stage 2 - Integration Analysis.
Improvement as it goes¶
Testing is not something that happens after this Stage. It happens inside it, and it changes what you are given.
When a test on a generated pipeline fails, the fix is not confined to the test. Where the fault is in the pipeline itself, or in the expression library the pipelines share, SLIM applies the correction to the pipelines you are going to deploy and then re-checks them. A fault in the shared library is fixed once and every pipeline using it gets the benefit. The completion report has a section recording exactly what was carried back this way, which is worth reading precisely because it tells you what went wrong on the first attempt.
That improvement stays inside one integration. Because each integration is worked in isolation, nothing learned while building one is carried into another in the same run.
How long, and what it costs¶
This is the longest and most expensive Stage by a wide margin, and hours rather than minutes is normal for a complex integration. How long depends on how many integrations you selected and how involved each one is, and the two representations run one after the other within each, so the time for a single integration is the time for both.
When a run finishes, the pill in the Stage header reads Complete and shows how long it took and what it cost alongside. The Model and reasoning effort set for the Project are what drive that figure; see Project Settings.
Adjusting and re-running¶
Instructions for this Stage go in the Pipeline Generation Prompt panel. Edit, type, Save.
These instructions reach the SnapLogic build, not the functional representation. That makes the panel the right place for anything about how the pipelines should be shaped - naming, how the work is divided, which snaps you prefer, which project space to build into. It is the wrong place for anything about what the integration does, because the half of the work that establishes behavior never sees it. Corrections of that kind belong in the analysis document, where they will be read. See Where Guidance Belongs.
One caution about naming. An instruction here applies to every integration in the selection, so an instruction that fixes a single pipeline name gives every integration in the run the same name, and you lose the ability to tell which output belongs to which integration. Ask for a convention rather than a name, or generate one integration at a time.
Design documents are attached on this page too, in the Design Docs panel, and they carry real authority over the structure of what is built. Supporting Documents covers what turning that on commits you to, including the conformance report SLIM then writes, and the longer wait at the end of the Stage that comes with it.
Then re-run, and be deliberate about the ticks first.
A re-run regenerates both representations for every ticked integration, including the ones that finished perfectly well. There is no skip for completed work, and this Stage is the expensive one, so an unconsidered re-run costs the full amount again. Untick what you do not want redone, and save the selection, before you press anything.
Download anything you have edited by hand first. That includes generated pipelines you have adjusted. There is no other copy, and nothing warns you.
If the run failed the control is Retry rather than Re-run; either way the overwrite is the same. See Stopping and Restarting.