SAS/EG managing high volume of data and SAS/EG metadata managing high

[juanvg1972]

Hi,
I have started to work with SAS/EG a few ago, I have worked a lot with SAS/BASE, but not SAS/EG.
I have a doubt about SAS/EG. I know the main features of the tool, but I don’t know how it works with high volume of data and a lot of number of projects.
I am starting a project with SAS/EG 4.3. We have to migrate JCL-Cobol process to SAS/EG. The process are typical process of reading files, sorting data, merging data, filtering data, aggregating data…, typical ETL steps. The process are typical process yo can do with SAS/EG.
The problem is the volume. The volume of process and the volume of data used in the process.
We have to migrate 500 processes. There are process that move files of 100 Gb.
If I make 500 .egp projects in guide, ¿can we have them in a unique metadata or server?, ¿do I have to share out between several servers/metadata?. ¿How many projects can manage a metadata with good performance?.
If you have to work with 100Gb files, would you use SAS/BASE or SAS/EG. I know, that I can modify code in my SAS/BASE programs to have a better performance, shorter processing times (objects like indexes, data in memory, etc..), but in SAS/EG I think I can modify less things….am I wrong?
Other question:
I will make the project in a windows server but I want to deploy it in a Unix server, the process will be scheduled in a Unix server and will be called from a .sh Unix shell script. Is it possible?, I think so…., but I prefer to make the question.
As you can see my english is not profiency. I hope my questions are clear…
Any advice will be greatly appreciated.
Thanks in advance

 

[sasbuddy]

Hi

As per as my knowledge is concerned; I can give you my opinions as folllows,

About handling large volume of data - this capability depends on the background infrastructure i.e. work space server where SAS EG runs the code. It depends on the capacity of CPU and memory provided on that machine.
Your system admin can give you more idea on this.

About making changes in code for optimization - In SAS EG, users can write there own SAS codes and its not mandatory to use only existing features. Hence you can write your own base SAS code and do whatever things needed to do optimization or for better performance.

About deploying code on UNIX - I am not sure if .EGP files can be deplyoed on unix but I would suggest you to develope the processes as BASE SAS codes only. Because as EG is windows tool, it might have some features which would not work properly on UNIX. hence its better to create .SAS files and then deploy them on UNIX by using cron tab inbuilt scheduler of UNIX.


sasbuddy

http://sites.google.com/site/sasbuddy/