Thread Question.

[Milleniumlegend]

I am writing an application that would perform a set of tasks such as FTP then email if the FTP is successful as well as Archiving of the Files if successful.

I am thinking of writing a thread which uses the methods from these class to FTP, Email, Archive and Log Events.

What is the best option to use using a Thread pool or a create a new thread for everytime these sequence of tasks is generated.

The workflow is such that whenever there is a new file submitted to the directory these tasks need to be performed.


Does these threads die after they are executed or do they remain in an inactive state?


Many Thanks

 

[Diancecht]

Well, considering that ftp and email tasks can be time consuming, maybe you won't notice a performance improvement by implementing a pool.

You can make the threads to exit, be sure there are no references to them that would force to not exit.

In the other hand, there are already made thread pools (even from Sun) so it's not really much extra work and the code can be clearer, error detecting can be easier.

Cheers,
Dian

 

[timw]

The new JSR208 Java Business Integration specification provides a container which can makes tasks like this easy to build and expand on. The first implementation of this specification is called 'ServiceMix' (

http://servicemix.org/).

Tim
---------------------------
"Your morbid fear of losing,
destroys the lives you're using." - Ozzy

 

[Milleniumlegend]

Tim

The Servicemix seems to be helpful but I guess that would involve learning the architecture for servicemix and then implementing servicemix.

Does that run as a windows service or is that another standard java application.

I am not sure how can I implement custom FTP and E-mail as well as event logging and Archiving of Files using Service Mix.

Thanks

 

[timw]

ServiceMix comes pre-packaged with JBI pluggable components wrapping ftp, email, logging and many more services.

All you should need to do is provide an XML configuration file which enlists the services you want and wires them together to satisfy your requirements.

The ServiceMix site (and the download itself) provides several sample use-cases of its use to get you started.

On the face of it, you should be able to get it do what you are wanting without having to write any code.

On the downside, this stuff is pretty new so documentation is sketchy. I ended up reading some of the JSR208 specification to get a grip on this.

It can run standalone or embedded within your own application (or application server). There are tools on the net which will make this (or any standalone java program) into a Windows Service (search this forum).

Tim
---------------------------
"Your morbid fear of losing,
destroys the lives you're using." - Ozzy

 

[Milleniumlegend]

Tim,

Could you please let me know how to poll a file and have it FTPed using this Service Mix. I did run the Sample File Binding application to transfer from one folder to the other but the files do not end up at the destination directory.

Could you please let me know if I need to do anything special or any tips. In addition to this If I need to do FTP the files to that are polled to a certain server across the domain what needs to be done. Can I create my Own customised FTP or just pass in the parameters ?

Let me know as this looks promising and may be I can dig some information in relation to this JBI.

Thanks

 

[timw]

I just need to point out that I'm fairly new to ServiceMix myself. My interest is integrating it with our company's legacy enterprise application to (hopefully) greatly enhance its interoperability with other applications.

I've had a look at the File Binding example and the current support within ServiceMix for tranferring files across the service bus. At present, only files already in XML format are supported, since these can be embedded within the intra service message (also in XML format) without any messing.

Files are imported into such a message using implementations of the org.servicemix.components.util.FileMarshaler interface. ServiceMix provides a DefaultFileMarshaler implementation for XML files. If the files you need to transfer are XML then you're laughing. Otherwise you'll probably need to write your own FileMarshaler implementation.

It's looking very likely that I'll have a similar requirement to yourself in this respect. I'll be handling EDI files, though. My initial thoughts were to write a FileMarshaler which encodes the file's URI as an XML snippet which is then included as the message content to be forwarded to whatever service wants the file. This latter would then need to use the same FileMarshaler to decode the message, thereby extracting the file URI and enabling it to get at the file's contents.

Alternatively, if I can encode the file's content so that it can be safely incorporated within an XML message (perhaps within a CDATA section), then I might write a FileMarshaler to do this instead. This is closer to the way the File Binding example works, with the file's actual content stuffed into the JBI intra-service message. (Though I am concerned about how this would perform for large files!!)

As for FTP, you've got a couple of choices. Assuming you have a FileMarshaler to handle whatever files your working with, you could use the FTP component (

http://www.servicemix.org/FTP)

or the VFS component (

http://www.servicemix.org/VFS),

both of which can handle FTP communication.

Both the above links give a detailed example. I'd be very interested to know how you get on with this if you attempt it. Good luck.

Tim
---------------------------
"Your morbid fear of losing,
destroys the lives you're using." - Ozzy

 

[timw]

I've just tried to modify the FileBinding example so it FTPs a file rather than copying it to an outbox directory. The following works nicely using the 'test-file.xml' file as provided in their example. Note that the destination name of the file is generated from data within the xml file and becomes 'sample_555-3482.xml'

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="[URL unfurl="true"]http://xbean.org/schemas/spring/1.0"[/URL]
	xmlns:spring="[URL unfurl="true"]http://xbean.org/schemas/spring/1.0"[/URL]
	xmlns:sm="[URL unfurl="true"]http://servicemix.org/config/1.0"[/URL]
	xmlns:xsi="[URL unfurl="true"]http://www.w3.org/2001/XMLSchema-instance"[/URL]
	xsi:schemaLocation="[URL unfurl="true"]http://xbean.org/schemas/spring/1.0[/URL] ../../conf/spring-beans.xsd
	                    [URL unfurl="true"]http://servicemix.org/config/1.0[/URL] ../../conf/servicemix.xsd"
	xmlns:foo="[URL unfurl="true"]http://servicemix.org/demo/">[/URL]

	<!-- the JBI container -->
	<sm:container id="jbi" useMBeanServer="true"
		createMBeanServer="true" dumpStats="true" statsInterval="10">

		<sm:activationSpecs>

			<!-- Send files via FTP protocol -->
			<sm:activationSpec componentName="ftpSender" service="foo:ftpSender">

				<sm:component>
					<bean class="org.servicemix.components.net.FTPSender">
						<property name="clientPool">
							<bean id="ftpClientPool" class="org.servicemix.components.net.FTPClientPool">
								<property name="host" value="[b][i]your host[/i][/b]"/>
							<property name="username" value="[b][i]your username[/i][/b]"/>
								<property name="password" value="[b][i]your password[/i][/b]"/>
							</bean>
						</property>
						<property name="marshaler">
						<bean class="org.servicemix.components.util.DefaultFileMarshaler">
								<property name="fileName">
									<bean class="org.servicemix.expression.JaxenStringXPathExpression">
									<constructor-arg value="concat('sample_', /sample/@id, '.xml')" />
									</bean>
								</property>
							</bean>
						</property>
					</bean>
				</sm:component>
			</sm:activationSpec>

			<!-- Look for files in the inbox directory -->
			<sm:activationSpec componentName="filePoller"
				destinationService="foo:ftpSender" service="foo:filePoller">
				<sm:component>
					<bean xmlns="[URL unfurl="true"]http://xbean.org/schemas/spring/1.0"[/URL]
						class="org.servicemix.components.file.FilePoller">
						<property name="workManager" ref="workManager" />
						<property name="file" value="inbox" />
						<property name="period" value="1000" />
					</bean>
				</sm:component>
			</sm:activationSpec>
		</sm:activationSpecs>
	</sm:container>

	<!-- the work manager (thread pool) for this container -->
	<bean id="workManager"
		class="org.jencks.factory.WorkManagerFactoryBean">
		<property name="threadPoolSize" value="30" />
	</bean>

</beans>

I've highlighted in bold/italics the bits where you provide your ftp login details.

Tim
---------------------------
"Your morbid fear of losing,
destroys the lives you're using." - Ozzy

 

[Milleniumlegend]

Tim,

This has been a great pointer and also thanks for the xml configuration that you sent. It worked perfectly for small files. Also I noticed that when sending a file that is non xml using FTP it would not work. The other thing is if the file exists with the same name then the service throws an error or creates an empty file. I am not sure if I am missing something here.

If I have a huge file then the service does get the file but does not post the file to the output area or the destination FTP host.

Also if I have more than 5000 directories to poll and very few files say 3 to 5 files under those directories then on my system it takes couple of seconds and the CPU that it uses is pretty high usually around 55%. I am not sure if this is normally the case.

Appreciate your help, it has been very valuable. But I think we should be able to play with the JBI components and tweak the servicemix to serve our purpose.

I guess with you doing the encoding of the file to send them as a interservice message would be good idea but I am not sure if that would work for binary files as well especially if they are large enough. I guess it would be ideal to split the file and then rejoin them but there would be this extra overhead to split the files and join them.


I would like to explore the filemarshaller and see what can be done to customise the messages and transfer of files. Since we get files from various places and they are all in different formats and sizes and based on the filename and location we need to do follow different process which is quite specific.

Could you please let me know where should I be looking to get info on the filemarshaller so that I can attempt to write the transfer for any file type and size.

Many Thanks

 

[timw]

The FileMarshaler interface (and it's only ServiceMix implementation - DefaultFileMarshaler) live in the org.servicemix.components.util package. The source for this can be found at the package 'path' under ServiceMix_Home/core/src/main/java.

I've tried the 'file-embedded-in-interservice-message' approach for plain text files, but I agree with you. Large files and binary files are going to make it unwieldy. I'm now tending towards putting the file URL in the message instead. One problem with this approach is that the file must remain in-situ until it is no longer needed. Perhaps the FileMarshaler could be instructed via a property in the config file to remove/leave the file as needed.

I expect issues like this will be documented and catered for in future releases. Hey, and have a read of

http://forums.logicblaze.com/posts/list/6.page;jsessionid=186861026CEC16D43649D0FE2EE60F6B.

We're not the only ones wondering about this aspect.

Tim
---------------------------
"Your morbid fear of losing,
destroys the lives you're using." - Ozzy