Looking - Sync Package 1

[YoungManRiver]

All,

Looking for a PHP driven package to synchronize files on machines, with DB (MySQL) backend and searchable/syncable by:

1. Topic/Content,
2. Date,
3. Project,
4. Latest,
5. Exclusion List Driven (for directories and/or files to ignore when syncing),
6. Thumb/Flash Drive or CD/DVD media, non networked sync transport,
7. Integratable with Backup/Recovery/ISO imaging program(s).

If you know one please reply. If not, willing to Open Source with someone to make this happen. Program with partial functionality considered.

Currently have the following machines that need syncing:

1. Window 2000 Pro Desktop 2ea.,
2. Linux Ubuntu Desktop 1ea.,
3. Linux Gentoo Desktop 1ea.,
4. Linux Gentoo Server 1ea.,
5. Window 2000 Pro Laptop 2ea.

Actually started coding this and found it too large to do quickly or alone.

Please leave msg through this board or reply.

Thanks!

YMR

 

[jpadie]

it sounds that you're looking at a level of complexity/flexibility that is going to pose problems.

have you considered using a central repository and have each computer act as a slave/contributor to that repository? for example an svn server?

otherwise something like sharepoint or groove might work. I don't know whether groove has a *nix client though.

lastly I use foldershare from MS to keep my client data synced between four computers (two Mac, 2 PC). I don't think that foldershare has a *nix client but there is an alternative in the market place caller PowerFolder that is rather more feature-rich than FolderShare. Both are free to use for a basic level of service.

but to answer your question more directly, i don't think that php is the right to tool for managing file synchronisation. you want something that can address directly at the filesystem level so that file receipts can be properly registered (because different file systems give receipts at different points in the process).

Of course, if you are planning to store the binaries themselves in the database then i'd be starting on a diatribe about not storing files in a database!

as a passing thought: MS synctoy might work too. you can create mapped smb shares and sync between all of those using one installation as a master.

 

[YoungManRiver]

JPadie,

Liked your suggestion on SyncToy and downloaded it. Also got some suggestions that resulted in the following response to my query on another site. Oh I'm OldManRiver on all other board, but this one, thus Sig=OMR.

All,

Got some information on products named "groove", "sharepoint" and "subversion". What do you know about these? My Google search shows "Groove" and "Sharepoint" as MicroTrash products and not Open Source. Subversion shows as Apache (can run on localhost, so solves security issue) enabled Open Source replacement for CVS.

Anyone tried this and how well does it work. I want your feedback

Thanks!

OMR

Thanks for the input, checking out the SyncToy. Will get back to you on what I like and/or need as enhancements. Might be able to work with the SyncToy developers on this.

Thanks!

YMR

 

[jpadie]

synctoy is, i believe, a microsoft product. your post above indicates you don't like MS products or that you require an open source alternative. if the latter then svn and powerfolder might be your answers. groove was originally written by non-microsoft company and is very good.

 

[YoungManRiver]

All,

Writing spec for this. Will pastebin and reply with URL.

YMR

 

[YoungManRiver]

All,

Working on a simple piece of this at:

http://tek-tips.com/viewthread.cfm?qid=1459169&page=1

Enjoy!

Thanks!

YMR

 

[YoungManRiver]

All,

More information!

The piece I'm currently working on is to sync to/from a .zip file.

Simple enough, but scripting is hung-up on a template variable refresh issue.

YMR

 

[YoungManRiver]

All,

Been away for awhile so just now reviewing my posts.

YMR

 

[YoungManRiver]

All,

I think I now know what I'm looking for. It would be like a combination of:

PCdocs or Worldox,
Subversion,
Duplicate file eliminator,
Data Sync Tools,
Collaboration Suite/Tool
Backup/Recovery Software.

But instead of the repository being the active directory (like subversion), the local machine become the active directory much like FoxMarks (Firefox), where the online database serves only to coordinate and make sure all users/machines contain the latest file/file revision and in the right path etc.

Ex:

User A on machine A, in the company network, is also a user on Laptop 1, which he takes home and to meetings. This user needs all the code and power point presentations needed for marketing demos.

On Machine A the user's files are in:

H:\UserA\My Docs

the code resides at:

P:\Programs\ProgramA\source

on the Laptop the files are stored at:

C:\Documents & Settings\UserA\My Doucuments

and code at:

C:\ProgramA\source

Additionally User B supports User A from Machine B on the network writing to the same code source directory, but has his/her notes and local development at:

C:\ProgramA\docs

User B is controlled by Subversion, but User A is not. But User A needs the notes for review before each presentation.

Additionally the marketing team makes entries to a database that shows to User A at:

M:\Marketing\DB\data

User A must have the lastest data when making the presentation. Part of the presentation shows data which must be relevent daily.

Additionally User A must have access to and update all docs on his/her laptop prior to departing the company for the next scheduled presentation.

Q:
How is this all done seamlessly either by Program with "Sync" button or by script in the "Shutdown" sequence on User A's computer, that syncs his/her latest files and data with all the latest in his/her world?

I found the following links to software that seems to have some of the answers:

http://www.sharewareconnection.com/ajc-directory-synchronizer.htm
http://allwaysync.com/?g=1&gclid=CIO9m9_pzpYCFRLoxgodm1AFyA
http://www.cardrecovery.com/?rid=google&kid=crct0802
http://www.sharewareconnection.com/comparedata.htm
http://www.sharewareconnection.com/comparatorpro.htm
http://www.sharewareconnection.com/compare-advance.htm
http://www.sharewareconnection.com/db-explorer.htm
http://www.sharewareconnection.com/diffnmerge.htm
http://www.goodsync.com/land/land.p...&utm_medium=ppc-content&utm_campaign=goodsync
http://www.sharewareconnection.com/idem.htm
http://www.sharewareconnection.com/kls-backup-2005-professional.htm
http://www.powerdatarecovery.com/?c...ata recovery&gclid=CNSkiovqzpYCFQNKxwodjWUzwQ
http://www.sharewareconnection.com/file-compare-folder-synchronization-software.htm
http://www.sharewareconnection.com/smart-suite-for-mysql.htm
http://www.file-utilities.com/duplicate_files.htm
http://www.sharewareconnection.com/smartsynchronize.htm
http://www.sharewareconnection.com/synchronize-folders.htm
http://www.tucows.com/preview/361109

Obviously I'm looking for an Open Source version of this combination or wish to collaborate with those who think this is worthwhile.

I've downloaded all the shareware software from the links and evaluating features to put into the spec I'm writing for this, but only get about 1 hour a week on this, so progress is slow.

Thanks all!!

OMR

 

[jpadie]

i've started researching this a bit more in the last month after a painful backup failure.

my view on these things it should be press and forget. anything that requires user interaction has a point of failure in it. this is true for sync and backup.

the truth is that there is not a lot out there. those products that are emerging are cloud based solutions, which are not acceptable to me from a security standpoint. this can be fixed, but none of these services are yet allowing the use of a userspecified public key.

examples include sugarsync and dropbox. I'm quite keen on the latter, but there are still limitations.

powerfolder also exists cross-platform and is solely p2p, which is good. but the interface sucks badly as it runs on java. when it's running it murders resources. i also noticed that a disk rename on one of my machines caused the other two machines connected via PF to delete all their files. ouch. Luckily PF does not permanently delete them, but it took more than 3 days to restore the 80000 files on a local machine. i eventually killed the process and wrote an rsync script. which took 5 minutes. something definitely wrong with PF's implementation there: but they are responsive and they say they are working on the issue.

i also use chronosync but mostly i am using now a combination of rsync and psync and manual intervention.

Foldershare still seems to be the best implementation for p2p file sync as of yet. i don't like that it changes the file mod and create times on sync. that seems wholly wrong to me. and i really don't like this artificial limitation of 10000 files and 2GB. both are unnecessary and unduly limiting.

In answer to your question though ...:

you're shooting yourself in the foot by allowing people to store code/documents in different places (viz the laptop: it should also point to the same share as the desktop). Users should store user-generated information in their home directory or a network share. If users are mobile then have them use offline shares. then windows will handle your syncing for you.
if you're developing real code then subversion or mercurial or some other without distributed code repository is the only way to go. you can't survive it. make the svn interface web accessible (webdav) so that users can re-commit whilst on the move without the need for a vpn or similar.

the second strand to your requirements seem to be that you can have multiple libraries pointing to different local directories. no probs for subversion etc. no real probs for windows offline shares either.

for the M drive: map it to the user A's laptop too. let windows handle the syncing for you.

finally do take a look at dropbox. it doesn't work for me because of the security limitations, but you could get around this by storing a truecrypt archive in the dropbox and putting files into that. files can be edited directly in dropbox too. sugarsync is a bit more flexible in terms of folder locations etc (it can watch folders for change, for example). Where it lets itself down is by not using delta transfers any change means the whole file gets resync'd). Both sugarsync and dropbox store each version of the file for you to peruse. and really lastly, i have been recommended nomadesk (by one of the writers). it does not work for me as no mac client. but you guys are operating on windows, so no probs. it does use deltasync .

beware treading this road on your own. solutions like this are best coded in C or a derivative language and need to understand the internal file system calls of each operating system. you need a central hub either to act as a cloud node or to intermediate peer to peer connections: which means you have to understand about those too. lastly, because you are dealing with data integrity you need a foolproof way to ensure that data sent has actually been received by the remote node and correctly committed. and a foolproof way to maintain integrity in the event of an edit conflict. These are all easy to discuss but, imo, really quite difficult to implement. your best bet is an overlay on rsync if you're dead set on a pure open source solution. Personally, i'd rather pay to have someone to rant out when the software ate my data!

 

[YoungManRiver]

J,

Can you post the links to the packages you wrote about?

YMR

 

[jpadie]

rsync
sugarsync
dropbox
Nomadesk
Chronosync
psync
powerFolder
FolderShare
Mercurial
subversion

i think that's all of them.

 

[jpadie]

i should add that live mesh looks interesting too.
live mesh

 

[lgarner]

I've used the closed-source iFolder quite a bit and it worked great. I don't know how good the open-source version is, or if this is the functionality that you're looking for.

http://en.wikipedia.org/wiki/IFolder

 

[jpadie]

i got really put off by the interface of iFolder, but I agree that it should be on the list too!