XML debatch in pipeline

Last month I had a post were described how to debatch XML message in orchestration
How I wrote it is not the best solution , in case if you will have hundreds of records the debatching using XPath will consume too much memory and processor.

In this post I am going to show how to debatch using pipeline.
The sample will fetch multiple records from MS SQL database and after debatching put them to folder.

How to create simple WCF-SQL ports and SQL polling you can read in my old post
I will use the same procedure and table to generate Schemas and PortTypes

In generated schema change envelop type to yes



Next click on TypePolling which is the root for schema
In properties find Body XPath
and edit it as in picture is

Then change the last node to have Max Occurs = 1

Now we should remove hidden stones :-)

When you added adapter metadata from wizard to your project
PortType was generated, DO NOT use it while creating logical receive port in orchestration.
DO NOT Use multipart message type while creating Message
your mesage now has simple structure



Thats all
now in output folder you should have many files where each will contain a single record from SQL



Comments

Unknown said…
Hey, thanks for your post but I'm not getting it to work right. Richard Serotor has a similar blog about this but neither are working for me. I'm still getting a single XML file with multiple records from my db.

I'm using TypedPolling to get my schema and data from db. I changed the Schema to Envelope=Yes, updated the Body XPath, and set leaf node to Max Occurence = 1. Created a File Send port to consume the receive port (no orchestration) but I'm still getting a single file in output folder.

What am I missing? Thanks!
October 6, 2010 at 7:32 PM
Armen Kirakosyan said…
hi Brad
It should be Ok

leave me your email, I will send a solution MSI file, where you can check it. (next week)
October 8, 2010 at 8:00 PM
Gilson said…
Hello Armen,

Is this possible (debatch using pipeline) with the schemas created by the Select operation in the Consume Adapter Service wizard?

Thanks,

Gilson
October 20, 2010 at 8:50 PM
Gilson said…
Hello Armen,
Is it possible to debatch in the pipeline using the schemas generated by the Select method of the Consume Adapter Service wizard?
Regards,
Gilson
October 21, 2010 at 12:29 PM
Armen Kirakosyan said…
Hi
Sure , if your schema is Taken from WCF-SQL adapter you can debatch in the same way.
Changing that values will not break your XSD
October 21, 2010 at 12:46 PM
Anonymous said…
Hi,
Thanks for your post..
Remove hidden stones section help to solve my problem
thanks again.....

Regards,
Nilesh
November 22, 2010 at 11:10 AM
Unknown said…
Hi I am designing a messaging solutuion not using any Orchs and want to recieve a batch as a whole message in the messge box then debatch on the send pipeline as i call a Stored Proc. Can this be done?

Thanks
December 18, 2010 at 4:26 PM
Armen Kirakosyan said…
hi
sure you can do it in the same way as described latter
December 27, 2010 at 12:00 PM
Anonymous said…
Thanks for the article. Really useful. I managed to get it to work for my project. But the result set is duplicating after debatching. For example if I have two records in the database table, debatch gives me four records. one duplicate record for each record. What I am I doing wrong? Thanks
March 20, 2011 at 2:10 AM
Armen Kirakosyan said…
Hi
Check pooling interval,
also check that you mark that records as retrieved so next time SP will skip records.

run SQL profiler to understand what's going on
March 20, 2011 at 8:11 AM
OldBeno said…
Armen,

Thanks for the details I am now debatching my messages on the pipeline ;)

Much Appreciation,
OldBeno
January 9, 2012 at 2:19 PM
OldBeno said…
Armen,

Thanks for the details I am now debatching my messages on the pipeline ;)

Much Appreciation,
OldBeno
January 9, 2012 at 2:19 PM
Post a Comment

Popular posts from this blog

Testing Azure Functions locally with automated integration test

Image
I am investing more and more time and efforts into building Azure Functions and using serverless instead of WebApps. Greater stability and balancing makes it my first choice when need high quality API. Being big fun of automated testing, faced some issues when coding tests and could not find any useful/working examples. Sources can be found here Some steps here Create function with HTTP trigger using template, dont need anything fancy there. Base it on .Net Core 3.1, you can change it, but then check the path in Test project Create MS Test project Folder structure Project structure Important step, install Azure Function tools from command line, just run npm i -g azure-functions-core-tools after that you can test if it is available from command line The idea is to start functions engine from test project. In local environment you can start it from your command line using Func utility, same can be done from Test project. From command line it will look like this F
Read more

Running Stockfish hosted in Azure function

Image
While working on small chess application had an idea to integrate Stockfish engine, so it will be possible to play with engine. More about Stockfish here : https://stockfishchess.org/ and here https://github.com/mcostalba/Stockfish The idea is to pass FEN and Depth for calculation to some API which will suggest the best move. My main problem is that the goal is to host engine in Azure, as function written in C# while Stockfish is in C++. There are of course some .net ports but really outdated and did not want to experiment with them. First step is just to clone git repo of Stockfish and build exe file, put it into separate folder in my solution. Stockfish is a command line tool. I will send the commands to console and read the output. This is what I will send and get: And in the solution it will look like this: With "No build" and "Copy to output" options The function by itself to get the request is pretty simple: [FunctionName("stockfish")
Read more

Consuming Web API using JavaScript and XMLHttpRequest method

Image
Using JavaScript with remote services by calling REST, JSON, WCF or other web service will decrease data round trips and will allow to create reach web interfaces. For demo I will use Visual Studio 2012 and MVC4. Client call will be hosted in MVC4 page and service will be provided by WebAPI which is also based on MVC4. Lets first see how it works in one server environment, more complex scenario will be in next posts.  Start Visual Studio 2012 and create a web site using  ASP.NET MVC4 Web Application  template.   Then choose  Basic  template without creating test project. Create HomeController and corresponding View which will response to root address I want to skip details about creating controllers and views, you can find a lot of samples and blog posts about it. On next step I will create another controller which will response to JavaScript calls. To create WebAPI page right click on  Controllers  folder and click on  Add / Web API Controller class .
Read more