Bulk extraction of all workspaces used in a legacy app
7 posts
• Page 1 of 1
Bulk extraction of all workspaces used in a legacy app
Hello,
I have to manage legacy applications written in APL for my company. The app is composed by more than 100 separate workspaces (mainly one for each input source, one that combine all the cleaned data and few other ones used to disseminate and build GUIs for end-users).
As I am a beginner in APL, I don't want to change the core engine for now but I would like at least to push everything in a git. I noticed that years after year, some element have been stored/duplicated in different places. So, I would like to do some clean-up but with the possibility to track and restore if something goes wrong. In this context, pushing everything into a git would be a big plus.
As I have a lot of workspaces involved, I cannot do by hand. So, I tried to due first with dyalogscript but I didn't succeed. So I have created a batch file as follows:
The unix script loops in a source folder and do a specific action for each workspace. It is working but –loadfn option fails and I didn't succeed to generate a load_ws.dyalog file with this command. Do you know how to fix that?
]SNAP is still the best method for extracting all info from a workspace? I tried CreateProject but it seems to be interactive and I need to do in bulk (the batch mode was not working in my case).
I have to manage legacy applications written in APL for my company. The app is composed by more than 100 separate workspaces (mainly one for each input source, one that combine all the cleaned data and few other ones used to disseminate and build GUIs for end-users).
As I am a beginner in APL, I don't want to change the core engine for now but I would like at least to push everything in a git. I noticed that years after year, some element have been stored/duplicated in different places. So, I would like to do some clean-up but with the possibility to track and restore if something goes wrong. In this context, pushing everything into a git would be a big plus.
As I have a lot of workspaces involved, I cannot do by hand. So, I tried to due first with dyalogscript but I didn't succeed. So I have created a batch file as follows:
- Code: Select all
#!/bin/bash
INDIR="$PWD/_SANDBOX_"
OUTDIR="$PWD/_SANDBOX_UNICODE_"
rm -r $OUTDIR
for ws in `find $INDIR -type f \( -name "*.dws" -o -name "*.DWS" \)`
do
echo "WS: $ws"
NEWDIR=$(dirname -- "$ws")
NEWDIR=${NEWDIR/$INDIR/$OUTDIR}
echo $NEWDIR
mkdir -p $NEWDIR
cd $NEWDIR
filename=$(basename -- "$ws")
extension="${filename##*.}"
WS_name="${filename%.*}"
echo $WS_name
/usr/bin/dyalog -tty << END_OF_FILE
)xload $ws
]SNAP $NEWDIR/ -makedir
⍝ OK but no file generated to reload the ws
⍝ ]SNAP $NEWDIR/ -makedir –loadfn
⍝* Command Execution Failed: Unable to create file /home/viguice/Documents/_SAND
⍝ BOX_UNICODE_/ -makedir –loadfn/=AMECOINIT=.dyalog: /home/viguice/Document
⍝ s/_SANDBOX_UNICODE_/ -makedir –loadfn/=AMECOINIT=.dyalog: Unable to creat
⍝ e file
⍝ Based on https://dyalog.tv/Dyalog18/?v=w4Wp01-d3Rw but not working even using batch mode
⍝]CreateProject $NEWDIR/$WS_name #
)OFF
END_OF_FILE
done
The unix script loops in a source folder and do a specific action for each workspace. It is working but –loadfn option fails and I didn't succeed to generate a load_ws.dyalog file with this command. Do you know how to fix that?
]SNAP is still the best method for extracting all info from a workspace? I tried CreateProject but it seems to be interactive and I need to do in bulk (the batch mode was not working in my case).
- viguice
- Posts: 2
- Joined: Wed Jun 26, 2024 11:26 am
Re: Bulk extraction of all workspaces used in a legacy app
Hi viguice,
I shall try this out and write an email to you from Dyalog Support.
Regards,
Vince
I shall try this out and write an email to you from Dyalog Support.
Regards,
Vince
- Vince|Dyalog
- Posts: 443
- Joined: Wed Oct 01, 2008 9:39 am
Re: Bulk extraction of all workspaces used in a legacy app
Hi,
Looks like an excellent plan to get the legacy app under source control. I assume you want to put it all into one workspace managed by one git project (that is what I would do).
Why not just copy all the workspaces into one workspace, and then use Link (or something link Dado) to write out the workspace as text files in a single folder? (Do not use ]SNAP, it is obsolete).
First, something like:
Then, just use Link to write out the folder.
I would also not put all the namespaces in the root. I would create a top level namespace for your application, and then put the 100 namespaces under that. If you start cleaning up and refactoring in a big way, or even enhancing the legacy app, you are likely to extract some code as a dependency, or use some other packages, and that will make it all neater and easier to manage. To do this, simply make new a namespace and copy the function above into the new namespace and run it from there.
Note ⎕IO is 0 and ⎕ML is 1 for above code.
Note also that the short file names must be valid namespace names.
Sounds like it could be a fun project. Good luck!
Looks like an excellent plan to get the legacy app under source control. I assume you want to put it all into one workspace managed by one git project (that is what I would do).
Why not just copy all the workspaces into one workspace, and then use Link (or something link Dado) to write out the workspace as text files in a single folder? (Do not use ]SNAP, it is obsolete).
First, something like:
- Code: Select all
CopyInAllWorkspaces←{
⍝ ⍵ ←→ Folder
f←(⎕NINFO⍠1)⍵,'\*.dws' ⍝ All workspaces
n←1⊃¨⎕NPARTS↑f ⍝ Short file names
s←⍎¨n ⎕NS¨⊂'' ⍝ Create namespaces
_←s.⎕CY¨f ⍝ Copy in code
0
}
Then, just use Link to write out the folder.
I would also not put all the namespaces in the root. I would create a top level namespace for your application, and then put the 100 namespaces under that. If you start cleaning up and refactoring in a big way, or even enhancing the legacy app, you are likely to extract some code as a dependency, or use some other packages, and that will make it all neater and easier to manage. To do this, simply make new a namespace and copy the function above into the new namespace and run it from there.
Note ⎕IO is 0 and ⎕ML is 1 for above code.
Note also that the short file names must be valid namespace names.
Sounds like it could be a fun project. Good luck!
- paulmansour
- Posts: 433
- Joined: Fri Oct 03, 2008 4:14 pm
Re: Bulk extraction of all workspaces used in a legacy app
Which version of Dyalog are you using ?
I am assuming that you're running on Linux ..
You say that you're dealing with "legacy applications written in APL" .. depending on how old some of your workspaces are, you may be better off not looking for .dws or .DWS files, but rather use something like
In the unlikely event that you're running on AIX, you'll need to run something like
I am assuming that you're running on Linux ..
You say that you're dealing with "legacy applications written in APL" .. depending on how old some of your workspaces are, you may be better off not looking for .dws or .DWS files, but rather use something like
- Code: Select all
find . -exec file {} \; | grep -i "dyalog.*workspace" | sed 's/:.*$//'
In the unlikely event that you're running on AIX, you'll need to run something like
- Code: Select all
find . -exec file -m /opt/mdyalog/19.0/64/unicode/p9/magic {} \; |\
grep -i "dyalog.*workspace" | sed 's/:.*$//'
-
AndyS|Dyalog - Posts: 263
- Joined: Tue May 12, 2009 6:06 pm
Re: Bulk extraction of all workspaces used in a legacy app
Thanks for your feedback to both of you.
Paul:
My first trial was based on your presentation of ]CreateProject during Dyalog'18 user meeting but it doesn't react as you did. Now, it is asking me confirmation for the creation and opens the cider_config file. Not suitable in my case for running in batch mode and I gave up with this option. Can you tell if ]Link is a better option than ]CreateProject?
At least, it is sure that I have better results using ]Link that I had with ]SNAP.
So, I have adapted my code as follows:
The shell script looks for all workspaces in the tree in input folder and creates a new WS_<migated> folder at the same location in the output folder with all the content saved at separate files.
Now, I tried to generate a workspace file linked to the folder but it works only on the command line. It seems that there is currently an issue with Create (https://github.com/Dyalog/link/issues/654). So I tried to use import instead. For backward compatibility with the current setup, the idea was to create an "empty" workspace with one unique function ∆WS_LX used when I open the workspace.
I have two issues with this:
- I would like to use relative paths. It works if I do interactively:
But I tried to concatenate for linking the ressource folder based on location of the current workspace and it doesn't seem to work. The expression
If I hardcode the path, it works. I can open the workspace and everything is reimported. This is mainly due to my inexperience in APL but I didn't manage to have something dynamic.
- My second issue is that the overwrite option seems not to work properly. As soon as save my workspace, it fails afterwards to open because the objects are already in the workspace. We can supposedly overcome this issue using overwrite option but it seems not be recognized as a valid option.
On the general strategy, this is indeed my goal to merge them into one workspace with separated namespaces if possible. But as they are interlinked somehow, I need to process step by step. Merging them from will break all the dependencies.
Andy:
The original project in production is running on Dyalog 17 Classic on Windows. I have created a sandbox on Ubuntu with Dyalog 19 to try to extract the content. I don't know what exactly you mean by not looking to wfs files but my files are wfs files. There is no other type of file and your command gives me the same output. But it helps me for giving you more details on the original files: They are all 32-bit classic little-endian version 17.8. I am attending to recreate them into 64-bit unicode little-endian version 19.9 with code managed in a git.
Paul:
My first trial was based on your presentation of ]CreateProject during Dyalog'18 user meeting but it doesn't react as you did. Now, it is asking me confirmation for the creation and opens the cider_config file. Not suitable in my case for running in batch mode and I gave up with this option. Can you tell if ]Link is a better option than ]CreateProject?
At least, it is sure that I have better results using ]Link that I had with ]SNAP.
So, I have adapted my code as follows:
- Code: Select all
#!/bin/bash
INDIR="$PWD/_SANDBOX_"
OUTDIR="$PWD/_SANDBOX_UNICODE_"
LOGFILE="$PWD/link_all.log"
TEMPFILE="$PWD/link_temp.log"
ERRFILE="$PWD/link_err.log"
rm -r $OUTDIR
rm $LOGFILE $ERRFILE
OIFS="$IFS"
IFS=$'\n'
for ws in `find $INDIR -type f \( -name "*.dws" -o -name "*.DWS" \)`
do
echo "WS: $ws"
echo "WS: $ws" >> $LOGFILE
NEWDIR=$(dirname -- "$ws")
NEWDIR=${NEWDIR/$INDIR/$OUTDIR}
echo $NEWDIR
mkdir $NEWDIR
cd $NEWDIR
filename=$(basename -- "$ws")
extension="${filename##*.}"
WS_name="${filename%.*}"
/usr/bin/dyalog -tty << END_OF_FILE > $TEMPFILE
⎕PW←256
)xload "$ws"
]Link.export # "./WS_$WS_name" -arrays
)OFF
END_OF_FILE
if grep -q "File name case clash" "$TEMPFILE"; then
echo "-> Force casecode" >> $LOGFILE
echo " Issue for $(grep "^#." $TEMPFILE | tr -d ' ' | tr '\n' ' ')" >> $LOGFILE
/usr/bin/dyalog -tty << END_OF_FILE2 > $TEMPFILE
⎕PW←256
)xload "$ws"
]Link.export # "./WS_$WS_name" -arrays -casecode
)OFF
END_OF_FILE2
fi
if grep -q "ERROR" "$TEMPFILE"; then
echo "WS: $ws" >> $ERRFILE
cat $TEMPFILE >> $ERRFILE
echo "-> Error for $(grep "^#." $TEMPFILE | tr '\n' ' ')" >> $LOGFILE
echo " Retry without these objects..." >> $LOGFILE
rm -r ./WS_$WS_name
/usr/bin/dyalog -tty << END_OF_FILE3 > $TEMPFILE
⎕PW←256
)xload "$ws"
)erase $(grep "^#." $TEMPFILE | tr '\n' ' ')
]Link.export # "./WS_$WS_name" -arrays -casecode
)OFF
END_OF_FILE3
if grep -q "ERROR" "$TEMPFILE"; then
echo "WS: $ws" >> $ERRFILE
cat $TEMPFILE >> $ERRFILE
echo " Still not working..." >> $LOGFILE
fi
else
cat $TEMPFILE >> $LOGFILE
fi
/usr/bin/dyalog -tty << END_OF_FILE4
⍝ ⎕SE.Link.Import # ''WS_$WS_name'' -overwrite not working
⍝ #.⎕FX'∆WS_LX' '⎕SE.Link.Import # ''WS_$WS_name'''
⍝ #.⎕FX'∆WS_LX' '⎕SE.Link.Import # (⎕WSID,WS_$WS_name)'
#.⎕FX'∆WS_LX' '⎕SE.Link.Import # ''$NEWDIR/WS_$WS_name'''
)SAVE $WS_name
)OFF
END_OF_FILE4
done
IFS="$OIFS"
rm $TEMPFILE
grep -v "Opening in existing browser session." $LOGFILE
grep -v "Opening in existing browser session." $ERRFILE
The shell script looks for all workspaces in the tree in input folder and creates a new WS_<migated> folder at the same location in the output folder with all the content saved at separate files.
Now, I tried to generate a workspace file linked to the folder but it works only on the command line. It seems that there is currently an issue with Create (https://github.com/Dyalog/link/issues/654). So I tried to use import instead. For backward compatibility with the current setup, the idea was to create an "empty" workspace with one unique function ∆WS_LX used when I open the workspace.
I have two issues with this:
- I would like to use relative paths. It works if I do interactively:
]CD 'my/path'
⎕SE.Link.Import # 'WS_<myworkspace>'
But I tried to concatenate for linking the ressource folder based on location of the current workspace and it doesn't seem to work. The expression
⎕SE.Link.Import #(⎕WSID,'/<myworkspace>')says that the path is not correct even it looks ok.
If I hardcode the path, it works. I can open the workspace and everything is reimported. This is mainly due to my inexperience in APL but I didn't manage to have something dynamic.
- My second issue is that the overwrite option seems not to work properly. As soon as save my workspace, it fails afterwards to open because the objects are already in the workspace. We can supposedly overcome this issue using overwrite option but it seems not be recognized as a valid option.
On the general strategy, this is indeed my goal to merge them into one workspace with separated namespaces if possible. But as they are interlinked somehow, I need to process step by step. Merging them from will break all the dependencies.
Andy:
The original project in production is running on Dyalog 17 Classic on Windows. I have created a sandbox on Ubuntu with Dyalog 19 to try to extract the content. I don't know what exactly you mean by not looking to wfs files but my files are wfs files. There is no other type of file and your command gives me the same output. But it helps me for giving you more details on the original files: They are all 32-bit classic little-endian version 17.8. I am attending to recreate them into 64-bit unicode little-endian version 19.9 with code managed in a git.
- viguice
- Posts: 2
- Joined: Wed Jun 26, 2024 11:26 am
Re: Bulk extraction of all workspaces used in a legacy app
Regarding ]CreateProject, it looks like that is a Cider command you are using, which is not what was demonstrated back in 2018, which was Acre. Cider is a project management application that uses Link. If it sounds confusing, that is because it is. I can't really offer any assistance with Cider or Link as I don't use them (I use Dado, a successor of Acre).
My guess is you might want to use Cider, and then not worry about Link.
I don't really understand the shell script, or why you using a shell script vs a simple APL function, or why you are saving a workspace at all, (though you might have good reason for all of that) so I'm not sure I can offer help more than the original function I wrote and the advice to use something like Link or Cider or Dado to write out the code as text files and dispense with the workspaces. But I'll try to answer any further questions I can.
My guess is you might want to use Cider, and then not worry about Link.
I don't really understand the shell script, or why you using a shell script vs a simple APL function, or why you are saving a workspace at all, (though you might have good reason for all of that) so I'm not sure I can offer help more than the original function I wrote and the advice to use something like Link or Cider or Dado to write out the code as text files and dispense with the workspaces. But I'll try to answer any further questions I can.
- paulmansour
- Posts: 433
- Joined: Fri Oct 03, 2008 4:14 pm
Re: Bulk extraction of all workspaces used in a legacy app
Hi viguice,
a) You can read more about Cider here, https://github.com/aplteam/Cider
b) You can trace into ⎕SE.Link.Import to see what is happening when you do
⎕SE.Link.Import # 'WS_<myworkspace>'
compared with:
⎕SE.Link.Import #(⎕WSID,'/<myworkspace>')
What is in ⎕WSID when you are doing it non-interactively?
For a user coming from MS Windows, tracing this will be easier if you use RIDE. https://github.com/Dyalog/ride
c) Here is an illustration of how to set the overwrite option for ⎕SE.Link.Import:
Regards,
Vince
a) You can read more about Cider here, https://github.com/aplteam/Cider
b) You can trace into ⎕SE.Link.Import to see what is happening when you do
⎕SE.Link.Import # 'WS_<myworkspace>'
compared with:
⎕SE.Link.Import #(⎕WSID,'/<myworkspace>')
What is in ⎕WSID when you are doing it non-interactively?
For a user coming from MS Windows, tracing this will be easier if you use RIDE. https://github.com/Dyalog/ride
c) Here is an illustration of how to set the overwrite option for ⎕SE.Link.Import:
⎕SE.Link.Import #'D:\temp\linktest'
⎕SE.Link.Import: Names already exist: use -overwrite flag to force overwriting the following names:
#.myns
#.ns
#.ns.foo
#.ns.subns
#.ns.subns.foo
#.opand
#.foo
⎕SE.Link.Import #'D:\temp\linktest'
∧
options←⎕NS ⍬ ⍝ create empty namespace
options.overwrite←1
options ⎕SE.Link.Import #'D:\temp\linktest'
Imported: # ← D:\temp\linktest
Regards,
Vince
- Vince|Dyalog
- Posts: 443
- Joined: Wed Oct 01, 2008 9:39 am
7 posts
• Page 1 of 1
Return to Source Code Management
Who is online
Users browsing this forum: No registered users and 0 guests
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group