Good Diff, Merge and Version Control solution with FoxBin2Prg 1

[dantheinfoman]

Hi All,

I've got SourceTree and I'm trying to get FoxBin2Prg to merge 2 different versions of an .SC2 file. It butchered the form, rendering it inoperable. Maybe I don't know how to merge right, or maybe SourceTree isn't as good as Plastic SCM like in this video:

https://www.youtube.com/watch?v=sE4wQ50Itqg&feature=youtu.be

. My only problem with this is that Plastic SCM isn't free like SourceTree is and it seems like they should both work. I don't know.

But surely all you brilliant FoxPro teams are using some sort of version control, possibly being able to work on the same binary files (vcx, scx, etc) at the same time and then merge together, right? I'm not just imagining this.

I'm trying to get my team out of the dark ages (using a word doc to version control and 'checking out' files by typing them in there). Please help me understand the best solution for version control and merging binary files.

Thanks a million for any suggestions.

Dan

 

[TamarGranor]

I'm using Mercurial with Beyond Compare (including Frank Perez's VFP add-in) as the comparison agent. I'm not using integrated source control.

Tamar

 

[Olaf Doschke]

I never used merge operations within source control. The textual representations of form code have manyfold problems in Foxpros original scctext. For example not outputting methods in alphabetical order even two identical forms might end up in two differntly sorted prg code representations, that will be merged unusable.

If you have to merge the work of two developers on the same form on a regular basis, that's a bigger issue. You should rather avoid that, check out files with blocking of other developers.

Bye, Olaf.

 

[Mike Lewis]

I've worked in situations where a whole team of developers (six or more) use a version control system, and we never felt the need for text differencing or merging. In fact, one of the reasons to use version control is to avoid the need for merging files from multiple developers.

Dan, you say you are trying to get out of "dark ages". That's good. But you are also unwilling to pay for a proper system. That's likely to be a false economy.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[dantheinfoman]

Thanks, all!

Do you have suggestions as to paid versions that are really good, then? I'm sure the company has no problem with paying for a good solution, I just want to do my research first.

Mike and Olaf, you mentioned not needing to merge files. Perhaps you know of software that allows you to inform the others that you're working with a particular file and that they should steer clear of it for a time? I thought that's what branching and merging were all about, but I see that's probably more for html type developers based on your comments above.

Thanks!

Dan

 

[Olaf Doschke]

Well, any source control software allows to check out a ile in a mode blocking others from checking out a version for editing. So everybody might extract the most recent version (you need as basis for every modification), but only the one having the check out for editing is able to check-in his change. Then the next can do hi work.

Since we work for a bigger corporation they host a team foundation server. This comes as benefit of being silver partner at least, but also smaller systems allow check out of files in such a mode.

In the worst case scenario users not able to check out an item for working on it remove a read only flag and work anyway. Then after the last change of the owning user is committed they commit their change. You're back to a situation needing a merge, then. So this has to be done with discipline, always, no matter if using source control or just a convention using a word file.

The mere nature of a project only compiling with full sources means you can't avoid developers getting at sources and starting their work, though it should wait for the changes of the checked out version.

Bye, Olaf.

 

[Mike Lewis]

Perhaps you know of software that allows you to inform the others that you're working with a particular file and that they should steer clear of it for a time?

Any version control system should do that. In fact, it's arguably the main function of a version control system. Or, rather, not so much to warn others to steer clear of the file, but to physically prevent them form working on it.

The system that I have used most is NG3. I found it very effective, and it met the entire needs of the team. But it's the only product of its type that I know well, so it's possible that there are better options available.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[fdbozzo69]

Hi Dan:

As with any merge operation with code in any language, you can't always trust on the SCM/DVCS tool do it automatically, and the Developer doing the merge must know what is merging and how is doing it, because it's the same as trying to merge a simple PRG with classes without looking at the "DEFINE CLASS/ENDDEFINE" or "PROCEDURE/ENDPROC" statements.

About the fiability of using FoxBin2Prg to do merging of code, I can say that we are using it since early 2014 with two different teams of 6-8 developers each one that maintain 2 legacy systems born at late 90's with a codebase of more than 3 millon lines of code, doing concurrent modifications and merges for more than 100 forms and class libraries without problems.

Apart from this, many people is using it with various SCM/DVCS tools, like git, SourceSafe, PlasticSCM, etc. There are some projects on VFPx for git that uses FoxBin2Prg to do the conversion part, so I can say that it is reliable.

Said that, problems can happen, but almost every time the source of the problem is a bad merging because human error, and can be fixed. No conversion component (FoxBin2Prg, TwoFox, Frank's VFP2Text, SccText) will assure you the merge, because it is a manual operation on which the Dev join different code sources.

But then, what benefits can add using one conversion tool or another?

Main difference is at the level of what Developers know better to work with, and a VFP Developer knows well PRG code, so merging PRG is pretty easier than merging XML or another text representation.

- SccText/X can't (shouldn't!) be used anymore because it only allow text generation and does not allow merge
- TwoFox and VFP2Text generate XML-style code, and can be used to merge, if you like to fight with XML structures and code inside it
- FoxBin2Prg generate PRG-style code, so it's easier to understand, modify and merge too. Plus, it implements many good ideas taken from the other ones, and that's why it's Open Source.

There is something important to have at mind when merging:

- Respect the file structure: That is, DEFINE CLASS/ENDDEFINE, PROCEDURE/ENDPROC, <DefinedPropArrayMethod> section and and metadata sections like CLASSDATA, OBJECTDATA and the like

- If one form adds a new property, then you will see that property in the <DefinedPropArrayMethod> section

- If one form adds a new method, then you will see that method in the <DefinedPropArrayMethod> section with it's corresponding PROCEDURE/ENDPROC section with the code

- A typical merge confusion, sometimes induced by the merge tool algorithms, is when merging code from one class into another class because the tool shows both at the same visual distance, one near the other

- Because the merging is one of the most tipical operations when using Source Control with teams, then using a good SCM/DVCS tool is very relevant, because the merging algorithms that every tool use makes the difference between comparing aples to aples or aples to oranges.

- For easing class libraries merges, you can configure FoxBin2Prg to generate one "class per file", as TwoFox does. Look at the docs in VFPx, where it is explained how to configure it

About your problem, if you attach the text files (sc2) of the files merged (the original files before the merge) and the 3rd file of the merge result, I can try to help you find what was the error at merging.


Best Regards.-

 

[dantheinfoman]

Thanks All, I appreciate your valuable input once again!

After considering your comments and watching a Youtube video AND reading Mike Podjer's whitepaper on using FoxBin2Prg with SourceTree, I've been able to realize that some merges necessitate the opening of the sc2 file in Notepad++ or the like and simply resolving the issue between the HEAD and the lower tag. Once I did that and ran FoxBin2Prg on the sc2 file, it worked perfectly, with changes from both 'users'.

Hi fdbozzo,

Thanks for the confirmation that one must open merges in notepad sometimes to resolve and that a team can find success using this system! I appreciate the tips on form structure as well.

The only problem I'm having now is how to merge a DBF table? We keep our data dictionary files in a dbf and I was trying to see how this would merge. I converted the dbf into a db2 file, but then when I tried to convert the db2 back to dbf, it said that it doesn't support this filetype. . . What??

Thanks all!!

Dan

 

[fdbozzo69]

Hi Dan:

I'm working on the DB2 Data reimport, that right now is not supported.

Please, have in mind that data tables should not go into the SCM/DVCS tool, except small data tables like configs or business rules and the like, because data is very heavy to sync.

You can find more help about VFP in general and about FoxBin2Prg on Universal Thread an Foxite. I look at them more frequently.

I suggest you subscribe to VFPx news, so you know every time a project is updated.

Best Regards.-

Fernando D. Bozzo

 

[Olaf Doschke]

How to merge data? APPEND? INSERT-SQL? UPDATE-SQL?

This has nothing to do with source code control.
Data is data, it's not source code.

Bye, Olaf.

 

[fdbozzo69]

Hi Olaf:

You should understand that people and teams have different necessities and different use-cases that you may not have.

One case on which this is useful is with business data:
- You have a table with validations for 3 products X, Y, Z
- Developer 1 add validations (records) for a new product A
- Developer 2 add validations (records) for a new product B
- Developer 3 change validations for either X, Y or Z

If data records are properly ordered, you can merge those tables and consolidate the final one, and this can be made by a 4th Developer as the Integrator.

You surely can think on other solutions, but reality is that there are some cases on which this is a needed feature, as you can see for Dan comments.

Obviously this is not intended for merging Contability, Client or Big tables, but there are many use cases, like config data and the like.

You can see this kind of requests on other forums too.

Best Regards.-

 

[Olaf Doschke]

Fernando,

such data editings should be done in a database. Not in a project. Even if that is stem data, meta data, you can handle that totally outside of source control. Shared data access is no problem at all and so this can even be done without checkout. Once you have such a central database you may then copy it local, if you want to embed data into EXE. I'd not do that anyway. Even if so, the normal scctext bin2prg way doesn't work here as diff/merge tool.

[tt]* These are the extensions used for table backups
...
#define SCC_DBC_TABLE_BAK "DB1"
#define SCC_DBC_MEMO_BAK "DB2"
#define SCC_DBC_INDEX_BAK "DB3"[/tt]

These are just backups, no more no less.

Bye, Olaf.

 

[fdbozzo69]

Hi Dante:

The new "Preview2" version of FoxBin2Prg v1.19.47 in VFPx have initial support for importing data:

https://vfpx.codeplex.com/releases/view/116407

Best Regards.-

 

[vgulielmus]

I like Fernando's FoxBin2Prg.
Before making changes into the generated files (VC2 and so on), is better to understand them.

But the utility is very powerful.
I made some tests and I managed to create binary files directly from prgs. Of course, I had to follow the rules.

Anyway, great work!

Respectfully,
Vilhelm-Ion Praisach
Resita, Romania

http://praisachion.blogspot.com/

 

[Olaf Doschke]

Back to the broader question about managing who can edit sources:

TFS allows locking of checked out files in this manner:

That's possible similar with other SCC tools, but you never can prevent developers to take a project and work on the versions of files they have in their local working directory even without officially checking out.
So you can still end up in a case of a item edited from version N to version N+1 by developer 1 and from version N to version N+2 from developer 2 (independent of each other), without developer 2 incorporating changes of 1 to N+1 of developer 1 and therefore needing a merge of both changes of versions N+1 and N+2.

No source control system can check from a checked in file on which version the change is based on. A diff can detect the changes of N+1 being removed again, but that may have been one of the changes towards version N+2: dev2 could have reverted to version N and doing another change instead.

Though the version control system could log and know which developers checked out which versions and at least warn developer 2 at check in, he hasn't checked out version N+1.

It's one of the rules of thumb to first check out, then in, to see intermediate changes as conflicts. Then you know you have to merge something. Obviously it's also good to do that before editing any item, but within that time span you also don't necessarily have exclusive access.

To conclude, source control still means not relying on this system alone, but having a discipline of checking out for editing to signal working on something. In VS this can be automatically done when you edit within Visual Studio while connected to team server. Even VFP has scc integration but it's not that easy to have this working because of the nature of libraries and forms being binary files needing the additional step to convert to scc PRG code and back to binary.

Bye, Olaf.

 

[fdbozzo69]

Olaf, the way you propose to work with a SCM tool is the same used by SourceSafe the last century.

Things have changed and now teams work using DVCS tools, which allow teams working in parallel and in branches without blocking themselves, something imposible to do with SourceSafe and tools like this.

The adventadges of using concurrent/parallel developing are anormous to ignore, and any system that need to block a file to modify it, is a team blocker by definition.

When you work with separeted teams that modify the same application and sometimes the same components, you don't rely on the tool to "block" them, you rely in some good practices that teams must follow so when they merge the conflicts are minimal and resolvable.

My team, and many others out there, are working this way since early 2014,and the boost of productivity is notable.

Previously we worked with SourceSafe, and because of the "blocking" thing, many were using copies in local unbound of VSS, so later, when checked-out the files, replaced the necessary ones.

But there are another well known problems related this way of work: when someone leves something checked out and the go out the office (because vacations, another new job, etc) you are in trouble, because that code can't be easily cheched-out until checked-in by the author, or somebody at the same author's PC with the author's credentials.

This is something that nobody wants, so there you have another good reason why blocking is a bad idea.

If you didn't work with the "unblocking" way, you should try it. Almost everybody else is trying it, and for a reason.

Regards.-

Fernando D. Bozzo

 

[Olaf Doschke]

Well, VFP has made no progress in this. You can work unblocking in VS projects, but I haven't seen merging work in VFP, yet. Even though scctextx and now foxbin2prg promise such things.

Can you suggest a way to configure VFP to get to the current age of source control?

Last config I tried is still mostly configured:

I only have deactivated TFS as source control provider, as I couldn't get this to work properly.

Bye, Olaf.

 

[fdbozzo69]

Yes, sure, I can make some suggestions about it.

First of all, the hard part: A mind-change about how to work with Source Control is needed. The problem with this is that SourceSafe have accostumed as for many many years in a way of working with SCM, that is what almost every SourceSafe user knows, which is checking-in and checking-out in a file by file basis, blocking meanwhile every developer that needs the same file.
This working is bad, forget about it, because it's not meant to be used by a team or geaographically distributed teams, even teams in the same building.

Once you accept that there is a better way, then can accept the second thing in which we have accostumed to work, that is with the SCM tool integrated in VFP interface. I mean, it's not BAD having integration on the IDE, Eclipse have it and many Dev tools have it, but the problem with VFP is the implementation they did, that is focused on the SourceSafe way of doing things, and that in the 90's was using the MSSCCI Provider API 1.0, that later have evolved with TFS (MSSCCI Provider - What? Why? Where?)

Even newer tools, like PlasticSCM, have this API available, but the problem with it is that restrict too much what you can do with Source Control.

The best way to work with Source Control tools is using it separated of the IDE. It's the way that you can get the best of the SCM/DVCS tool, which is some difficult to achieve inside the IDE, because when working with SCM you need to think, not in "file" terms, but in "changesets". The changeset should contain all the modifications to all the components needed to make one feature or to resolve one bug, so it contains all files changed for doing this.

When you see the history of modifications in SourceSafe, you see all the files of the project with their updated timestamps, but you don't know on which part of the history you have implemented a feature and which files where changed (yes, I know the tags and used them, but it's not enough). When you see the history in a DVCS tool, you see a history of "changesets", every one containing only the files changed at a point in history.

In the first post Dan mentioned a Youtube video I've made some time ago (I've made some shortest ones showing how to work with a DVCS (PlasticSCM in this case) and VFP 9, simulating 2 Developers that modify the same form, which is one of the most difficult component to merge:

PlasticSCM - Merge con Visual FoxPro 9.0 - Video 2

In the VFPx Documentation page of FoxBin2Prg I've made a very small tutorial in which I describe one way in which a team can work with a DVCS with VFP 9. It's based in the tool our team use and describes part of the working philosophy we use:

FoxBin2Prg and use with SCM tools

I think that once you see the video and read what I've wrote at VFPx (in previous link and the 7 pages that follows it about using PlasticSCM), you can have a better idea of one way of doing SCM in a different way.

There are other ways of working, and there other tools that uses FoxBin2Prg as the BIN<->TXT conversor, like some git implementations mentioned on main page of VFPx made by Lutz and Mike Potjer.


Best Regards.-

 

[Olaf Doschke]

Before seeing your videos, I just made one observation about just looking at classes of a VCX: The VCX header is changed, so the library gets a new timestamp. See LUPDATE(), it's not limited to DBFs, but also the specific dbfs of VCX, SCX, FRX.

I have no choice but TFS, which is one part of MY problem.

I can download all sources, make a copy, change readonly status, work on this, check out for edit, copy back m edits to working dir, check-in. The changeset of really changed files is computed, but this only is done at the server, I upload a volume of 180MB, this is a really large project.

I'd like to only need to upload the changeset of files I really worked on, but I can't manually keep track of this. Team members are not even my part of the problem, as I am the only dev on this.

I tend to simply make full copies in backups and only commit from time to time, as the process of VFP/TFS is totally unideal also in conjunction with data only available while at a VM of that customer. Nobody listens to me to make this a better situation, so it became one reason to quit that job. One of the lesser reasons, still one of the things unsolvable satisfying conditions of staying legal with IDE licenses, TFS license, data protection and as I am in a coworking space also keeping bandwidth usage low.

So this s not only about SCM and I gave up to address any two vectors of problems.

Bye, Olaf.